加載TMDB資料集，進行資料預處理
TMDb電影資料庫，資料集中包含來自1960-2016年上映的近11000部電影的基本資訊，主要包括了電影型別、預算、票房、演職人員、時長、評分等資訊，用于練習資料分析，
參考文章https://blog.csdn.net/moyue1002/article/details/80332186
python 3.7
pandas 0.23
numpy 1.18
metplotlib 2.2
import pandas as pd credits = pd.read_csv('./tmdb_5000_credits.csv') movies = pd.read_csv('./tmdb_5000_movies.csv')
查看各個dataframe的一般資訊
# 這是movies表的資訊 movies.head(1) Out[3]: budget genres homepage id ... tagline title vote_average vote_count 0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 ... Enter the World of Pandora. Avatar 7.2 11800
這是credits表的資訊
print(credits.info()) credits.head(1) Out[4]: <class 'pandas.core.frame.DataFrame'> RangeIndex: 4803 entries, 0 to 4802 Data columns (total 4 columns): movie_id 4803 non-null int64 title 4803 non-null object cast 4803 non-null object crew 4803 non-null object dtypes: int64(1), object(3) memory usage: 150.2+ KB None movie_id ... crew 0 19995 ... [{"credit_id": "52fe48009251416c750aca23", "de...
credits表的cast列很奇怪，資料很多
進行具體查看
# 查看credists表的cast列索引0的值，發現是一長串東西 print('cast格式：', type(credits['cast'][0])) # 查看其型別，為`str`型別，無法處理 Out[5]: cast格式： <class 'str'>
json格式化資料處理 從表中看出，cast列其實是json格式化資料，應該用json包進行處理
json格式是[{},{}]
將json格式的字串轉換成Python物件用json.loads()json.load()針對的是檔案，從檔案中讀取json
import json type(json.loads(credits['cast'][0])) Out[6]: list
從上面可以看出json.loads()將json字串轉成了list,可以知道list里面又包裹多個dict
接下來批量處理
import json json_col = ['cast','crew'] for i in json_col: credits[i] = credits[i].apply(json.loads) >> credits['cast'][0][:3] Out[7]: [{'cast_id': 242, 'character': 'Jake Sully', 'credit_id': '5602a8a7c3a3685532001c9a', 'gender': 2, 'id': 65731, 'name': 'Sam Worthington', 'order': 0}, {'cast_id': 3, 'character': 'Neytiri', 'credit_id': '52fe48009251416c750ac9cb', 'gender': 1, 'id': 8691, 'name': 'Zoe Saldana', 'order': 1}, {'cast_id': 25, 'character': 'Dr. Grace Augustine', 'credit_id': '52fe48009251416c750aca39', 'gender': 1, 'id': 10205, 'name': 'Sigourney Weaver', 'order': 2}] print('再次查看cast型別是:',type(credits['cast'][0])) # 資料型別變成了list，可以用于回圈處理 Out[8]: 再次查看cast型別是: <class 'list'>
提取其中的名字
credits['cast'][0][:3] # credits第一行的cast，是個串列 Out[9]: [{'cast_id': 242, 'character': 'Jake Sully', 'credit_id': '5602a8a7c3a3685532001c9a', 'gender': 2, 'id': 65731, 'name': 'Sam Worthington', 'order': 0}, {'cast_id': 3, 'character': 'Neytiri', 'credit_id': '52fe48009251416c750ac9cb', 'gender': 1, 'id': 8691, 'name': 'Zoe Saldana', 'order': 1}, {'cast_id': 25, 'character': 'Dr. Grace Augustine', 'credit_id': '52fe48009251416c750aca39', 'gender': 1, 'id': 10205, 'name': 'Sigourney Weaver', 'order': 2}] credits['cast'][0][0]['name'] # 獲取第一行第一個字典的人名 Out[10]: 'Sam Worthington'
dict字典常用的函式 dict.get() 回傳指定鍵的值，如果值不在字典中回傳default值
dict.items() 以串列回傳可遍歷的(鍵, 值) 元組陣列
# 代碼測驗如下： i = credits['cast'][0][0] for x in i.items(): print(x) Out[11]: ('cast_id', 242) ('character', 'Jake Sully') ('credit_id', '5602a8a7c3a3685532001c9a') ('gender', 2) ('id', 65731) ('name', 'Sam Worthington') ('order', 0)
創建get_names()函式，進一步分割cast
def get_names(x): return ','.join(i['name'] for i in x) credits['cast'] = credits['cast'].apply(get_names) credits['cast'][:3] Out[12]: 0 Sam Worthington,Zoe Saldana,Sigourney Weaver,S... 1 Johnny Depp,Orlando Bloom,Keira Knightley,Stel... 2 Daniel Craig,Christoph Waltz,Léa Seydoux,Ralph... Name: cast, dtype: object
crew提取導演
credits['crew'][0][0] Out[13]: {'credit_id': '52fe48009251416c750aca23', 'department': 'Editing', 'gender': 0, 'id': 1721, 'job': 'Editor', 'name': 'Stephen E. Rivkin'} # 需要創建回圈，找到job是director的，然后讀取名字并回傳 def director(x): for i in x: if i['job'] == 'Director': return i['name'] credits['crew'] = credits['crew'].apply(director) print(credits[['crew']][:3]) credits.rename(columns = {'crew':'director'},inplace=True) #修改列名 credits[['director']][:3] Out[[14]: crew 0 James Cameron 1 Gore Verbinski 2 Sam Mendes
movies表進行json決議
>>> movies.head(1) Out[15]: budget genres homepage id ... tagline title vote_average vote_count 0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 ... Enter the World of Pandora. Avatar 7.2 11800
可以看出genres, keywords, spoken_languages, production_countries, producion_companies需要json決議的
# 方法同crew表 json_col = ['genres','keywords','spoken_languages','production_countries','production_companies'] for i in json_col: movies[i] = movies[i].apply(json.loads) movies[i] = movies[i].apply(get_names) >>> movies.head(1) Out[16]: budget genres homepage id ... tagline title vote_average vote_count 0 237000000 Action,Adventure,Fantasy,Science Fiction http://www.avatarmovie.com/ 19995 ... Enter the World of Pandora. Avatar 7.2 11800
這樣，就把資料預處理做完了，
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/112849.html
標籤：其他
上一篇：分享學習 PHP 原始碼的方法