本文的文字及圖片來源于網路,僅供學習、交流使用,不具有任何商業用途,如有問題請及時聯系我們以作處理，
以下文章來源于大鄧和他的Python ，作者：大鄧
Python爬取知乎資料案例講解視頻
https://www.bilibili.com/video/BV1fz4y1Z7zz/
前言
問題鏈接
https://www.zhihu.com/question/432119474/answer/1597194524
爬蟲設計流程
探尋網址規則嘗試對某些網頁訪問決議學術的資料存盤到csv整理匯總代碼
1.探尋網址規則
- 按F12鍵打開開發者工具，
- 進入網路面板，點擊查看全部6217個回答
- 準備觀察開發者工具中的監測到的網址
- 對每個網址經過下圖456操作
- 點擊預覽
- 查看內容與當前頁面的回答是否一致
- 最終發現網址如7中的紅色方框，請求方式為GET方法
- 依舊是7位于的頁面，滑動到最下方，可以看到offset和limit
發現的網址（注意最后一行的偏移）
https://www.zhihu.com/api/v4/questions/432119474/answers?include=data%5B*%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cattachment%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%2Cis_recognized%2Cpaid_info%2Cpaid_info_content%3Bdata%5B*%5D.mark_infos%5B*%5D.url%3Bdata%5B*%5D.author.follower_count%2Cbadge%5B*%5D.topics%3Bsettings.table_of_content.enabled%3B&offset=3&limit=5&sort_by=default&platform=desktop中也存在offset，該單詞的意思是偏移量，
- offset我猜測該值數值頁面頁面數
- limit每個網址能展現多少個回答，至少5個，
網址模板（注意模板內最后一行偏移）
https://www.zhihu.com/api/v4/questions/432119474/answers?include=data%5B*%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cattachment%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%2Cis_recognized%2Cpaid_info%2Cpaid_info_content%3Bdata%5B*%5D.mark_infos%5B*%5D.url%3Bdata%5B*%5D.author.follower_count%2Cbadge%5B*%5D.topics%3Bsettings.table_of_content.enabled%3B&offset={offset}&limit=5&sort_by=default&platform=desktop當前回答一共有6200多個，每頁5個，那么offset可以有1240頁，
2.嘗試對某些網頁訪問
import requests template = 'https://www.zhihu.com/api/v4/questions/432119474/answers?include=data%5B*%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cattachment%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%2Cis_recognized%2Cpaid_info%2Cpaid_info_content%3Bdata%5B*%5D.mark_infos%5B*%5D.url%3Bdata%5B*%5D.author.follower_count%2Cbadge%5B*%5D.topics%3Bsettings.table_of_content.enabled%3B&offset={offset}&limit=5&sort_by=default&platform=desktop' #for page in range(1, 1240): url = template.format(offset=1) headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36'} resp = requests.get(url, headers=headers) resp
<Response [200]>
我們注意到5種資料可以展開，大概率是json樣式的資料，
所以嘗試使用resp.json（）來拿到格式化的字典資料，
3.決議研究的資料
由于resp.json（）回傳的是字典，決議定位資料會很方便，
咱們就需求簡單點，只采集作者，id，摘錄三個變數，
其中作者內部含有很多更豐富的資訊，研究的可以把作者再整理下，此處對作者不做過多清洗，
for info in resp.json()['data']: author = info['author'] Id = info['id'] text = info['excerpt'] data = {'author': author, 'id': Id, 'text': text} print(data) {'author': {'id': '10779c1a8b8de0b7f14bb6c217100680', 'url_token': 'da-mo-wang-81-18', 'name': '緊衣衛', 'avatar_url': 'https://pic2.zhimg.com/v2-ebdfa810ec26e81720b8bc14fc6c1d76_l.jpg?source=1940ef5c', 'avatar_url_template': 'https://pic3.zhimg.com/v2-ebdfa810ec26e81720b8bc14fc6c1d76.jpg?source=1940ef5c', 'is_org': False, 'type': 'people', 'url': 'https://www.zhihu.com/api/v4/people/10779c1a8b8de0b7f14bb6c217100680', 'user_type': 'people', 'headline': '夢想太大 心太執著', 'badge': [{'type': 'identity', 'description': '江蘇省寶應縣公安局 警察', 'topics': []}], 'badge_v2': {'title': '江蘇省寶應縣公安局 警察', 'merged_badges': [{'type': 'identity', 'detail_type': 'identity', 'title': '認證', 'description': '江蘇省寶應縣公安局 警察', 'url': 'https://www.zhihu.com/account/verification/intro', 'sources': [], 'icon': '', 'night_icon': ''}], 'detail_badges': [{'type': 'identity', 'detail_type': 'identity_people', 'title': '已認證的個人', 'description': '江蘇省寶應縣公安局 警察', 'url': 'https://www.zhihu.com/account/verification/intro', 'sources': [], 'icon': 'https://pic1.zhimg.com/v2-235258cecb8a0f184c4d38483cd6f6b6_l.png', 'night_icon': 'https://pic3.zhimg.com/v2-45e870b8f0982bcd7537ea4627afbd00_l.png'}], 'icon': 'https://pic4.zhimg.com/v2-235258cecb8a0f184c4d38483cd6f6b6_l.png', 'night_icon': 'https://pic3.zhimg.com/v2-45e870b8f0982bcd7537ea4627afbd00_l.png'}, 'gender': 1, 'is_advertiser': False, 'follower_count': 3914, 'is_followed': False, 'is_privacy': False}, 'id': 1597225705, 'text': '我也不喜歡過洋節，但我知道最起碼的人情世故，俗話說出手不打笑臉人 阿姨是給你送巧克力，又不是給你送鶴頂紅 如果內心不愿意接受就婉拒好了 這個有什么好舉報的？即使阿姨真的是為了慶祝感恩節 你又想得到一個什么樣的舉報結果呢？據說舉報人是個博士生，我看也別研究別的了，轉方向先學習學習最基本的為人處事方法吧，'} {'author': {'id': '09c8a026f91155c87188ee6e45301f05', 'url_token': 'cao-ling-er-36', 'name': '曹小靈', 'avatar_url': 'https://pic1.zhimg.com/v2-942365af26ff3abd44f6c75c9899306e_l.jpg?source=1940ef5c', 'avatar_url_template': 'https://pic2.zhimg.com/v2-942365af26ff3abd44f6c75c9899306e.jpg?source=1940ef5c', 'is_org': False, 'type': 'people', 'url': 'https://www.zhihu.com/api/v4/people/09c8a026f91155c87188ee6e45301f05', 'user_type': 'people', 'headline': '偽商業分析*抽風式咨詢*個人成長', 'badge': [{'type': 'identity', 'description': 'Backbase 管理分析咨詢師', 'topics': []}], 'badge_v2': {'title': 'Backbase 管理分析咨詢師', 'merged_badges': [{'type': 'identity', 'detail_type': 'identity', 'title': '認證', 'description': 'Backbase 管理分析咨詢師', 'url': 'https://www.zhihu.com/account/verification/intro', 'sources': [], 'icon': '', 'night_icon': ''}], 'detail_badges': [{'type': 'identity', 'detail_type': 'identity_people', 'title': '已認證的個人', 'description': 'Backbase 管理分析咨詢師', 'url': 'https://www.zhihu.com/account/verification/intro', 'sources': [], 'icon': 'https://pic1.zhimg.com/v2-235258cecb8a0f184c4d38483cd6f6b6_l.png', 'night_icon': 'https://pic1.zhimg.com/v2-45e870b8f0982bcd7537ea4627afbd00_l.png'}], 'icon': 'https://pic2.zhimg.com/v2-235258cecb8a0f184c4d38483cd6f6b6_l.png', 'night_icon': 'https://pic2.zhimg.com/v2-45e870b8f0982bcd7537ea4627afbd00_l.png'}, 'gender': 1, 'is_advertiser': False, 'follower_count': 412637, 'is_followed': False, 'is_privacy': False}, 'id': 1597380398, 'text': '親人們，知道為什么很多機關辦事人員會僵化么？知道為什么很多機構不愿意創新么？知道為什么很多組織都是照章辦事缺乏靈活么？你看到的這個案例，就是原因，很多事情，你不做，就不會錯；做了，取得的回報很少，但是承擔的風險很高，你永遠不知道什么地方會突然飛過來一泡口水，把你的好意扭曲為惡意，把你的行為定性為「不妥」，如果是在影視行業，不斷在邊緣瘋狂的試探，還可以理解，這樣可以給觀眾帶來新的刺激，換來一句…'} {'author': {'id': 'a82cf4ad4be905dbc044297e70e3b382', 'url_token': 'alyssa-song', 'name': '貂絲女嬸教授', 'avatar_url': 'https://pic4.zhimg.com/v2-2b1a52a861aecec566a3c2c4f00a5e06_l.jpg?source=1940ef5c', 'avatar_url_template': 'https://pic2.zhimg.com/v2-2b1a52a861aecec566a3c2c4f00a5e06.jpg?source=1940ef5c', 'is_org': False, 'type': 'people', 'url': 'https://www.zhihu.com/api/v4/people/a82cf4ad4be905dbc044297e70e3b382', 'user_type': 'people', 'headline': '985副教授，少女型大媽，生化環材', 'badge': [], 'badge_v2': {'title': '', 'merged_badges': [], 'detail_badges': [], 'icon': '', 'night_icon': ''}, 'gender': 0, 'is_advertiser': False, 'follower_count': 10200, 'is_followed': False, 'is_privacy': False}, 'id': 1597237844, 'text': '我從前就住在哈工大的17公寓，宿管阿姨們真的非常好，非常溫暖，我當時有一段生病天天待在寢室，阿姨們白天巡寢的時候還經常問我好點了沒有，用不用幫我從樓下帶盒飯上來，對我這種從小就缺愛的人，阿姨真的就像自己的親人一樣，讓我從內心深處感覺到了家的溫暖，住在17公寓的日子真的超級美好，當年哈工大的后勤管理還有一點很好，就是每一屆畢業的學生把不穿也不帶走的舊大衣捐給學校，新一屆本科生入校時候，貧困生可以每人…'} {'author': {'id': 'e6bbeff00614e94be5225c9fb401f989', 'url_token': 'zai-yi-qi-61-23', 'name': '精分的曹小靈', 'avatar_url': 'https://pic4.zhimg.com/v2-d3325d2f1d7de8511a5e9a328c75a539_l.jpg?source=1940ef5c', 'avatar_url_template': 'https://pic2.zhimg.com/v2-d3325d2f1d7de8511a5e9a328c75a539.jpg?source=1940ef5c', 'is_org': False, 'type': 'people', 'url': 'https://www.zhihu.com/api/v4/people/e6bbeff00614e94be5225c9fb401f989', 'user_type': 'people', 'headline': '頭腦中存在兩種截然相反的思想，開始精分', 'badge': [], 'badge_v2': {'title': '', 'merged_badges': [], 'detail_badges': [], 'icon': '', 'night_icon': ''}, 'gender': 1, 'is_advertiser': False, 'follower_count': 4997, 'is_followed': False, 'is_privacy': False}, 'id': 1597348175, 'text': '啊…又來了，對這種道德高地站的穩的學生，張博洋曾經發出過靈魂質問：[圖片] [圖片] 宿管阿姨明確的說了：這是西方的節日，自己是借著這個節日的機會，感謝同學們對作業的支持，現在很多女生找各種借口過節，很多男生找各種借口送禮，怎么就不能讓宿管阿姨給你一個不想奮斗的機會呢？這種人，不配當個男人，至少不配當一個新時代的以阿姨包養為夢想的好男人，你背叛了你的階級！手動差評！要上綱上線，別急，咱們一層一層的來，先講歷…'} {'author': {'id': '0ba346aab1e8761d12e8d1cf28fce8f8', 'url_token': 'xia-xiao-qi-61', 'name': '瞄小七.sunshine', 'avatar_url': 'https://pic2.zhimg.com/v2-fc018e46c81d7828e65d34613f62425c_l.jpg?source=1940ef5c', 'avatar_url_template': 'https://pic4.zhimg.com/v2-fc018e46c81d7828e65d34613f62425c.jpg?source=1940ef5c', 'is_org': False, 'type': 'people', 'url': 'https://www.zhihu.com/api/v4/people/0ba346aab1e8761d12e8d1cf28fce8f8', 'user_type': 'people', 'headline': '三甲醫院手術室護師/護理學穿搭博主', 'badge': [], 'badge_v2': {'title': '', 'merged_badges': [], 'detail_badges': [], 'icon': '', 'night_icon': ''}, 'gender': 0, 'is_advertiser': False, 'follower_count': 9792, 'is_followed': False, 'is_privacy': False}, 'id': 1598254838, 'text': '[圖片] 舉報這個事，小題大做，本來是一件在寒冬天里溫暖人心的好事，也不知道是觸動了一個腦回路清奇之人的哪根敏感神經？這位舉報愛好者就這樣憤憤地向宿管阿姨發出了最后通牒，我心里還在想呢，這么有愛的宿管阿姨，果然是別人學校的宿管阿姨，沒想到，有這樣的宿舍阿姨，有些同學非但不知珍惜，還要上綱上線舉報她，那位舉報阿姨的同學，平時心里應該很苦嘴巴才這么毒吧？是自己生活過的太平淡無為才用這種方式在善良的宿管阿…'}
4.存盤到csv
設定三個細分，存盤到csv，
注意現在只是對某些頁面的資料進行的存盤，
import csv csvf = open('zhihu.csv', 'a+', encoding='utf-8', newline='') fieldnames = ['author', 'id', 'text'] writer = csv.DictWriter(csvf, fieldnames=fieldnames) writer.writeheader() for info in resp.json()['data']: author = info['author'] Id = info['id'] text = info['excerpt'] data = {'author': author, 'id': Id, 'text': text} writer.writerow(data) csvf.close()
5.整理匯總代碼
將1234代碼合并整理，就是一個對知乎問題回答的批量采集爬蟲，
注意復制標簽代碼，要注意層次關系，上下變數名要對應銜接接，
import requests import csv import time #新建csv，存盤資料 csvf = open('zhihu.csv', 'a+', encoding='utf-8', newline='') fieldnames = ['author', 'id', 'text'] writer = csv.DictWriter(csvf, fieldnames=fieldnames) writer.writeheader() #偽裝頭，用于反爬 headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36'} #網址模板 template = 'https://www.zhihu.com/api/v4/questions/432119474/answers?include=data%5B*%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cattachment%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%2Cis_recognized%2Cpaid_info%2Cpaid_info_content%3Bdata%5B*%5D.mark_infos%5B*%5D.url%3Bdata%5B*%5D.author.follower_count%2Cbadge%5B*%5D.topics%3Bsettings.table_of_content.enabled%3B&offset={offset}&limit=5&sort_by=default&platform=desktop' #批量采集， #目前可爬1240+頁資料 #這里僅抓取前100頁 for page in range(1, 100): #對第page頁進行訪問 url = template.format(offset=page) resp = requests.get(url, headers=headers) #決議定位第page頁的資料 for info in resp.json()['data']: author = info['author'] Id = info['id'] text = info['excerpt'] data = {'author': author, 'id': Id, 'text': text} #存入csv writer.writerow(data) #降低爬蟲對知乎的訪問速度 time.sleep(1) #關閉csvf csvf.close()
測驗
采集完畢后，我們檢查下zhihu.csv，看看是否采集到的資料，
import pandas as pd df = pd.read_csv('zhihu.csv') df.head()
詞云圖
找出所有回答的內容文本，制作詞云圖，可以看到知乎用戶對哈工大感恩節事件的看法，
import jieba import json from pyecharts import options as opts from pyecharts.charts import WordCloud text_contents = ''.join(df['text']) words = jieba.lcut(text_contents) words = [w for w in words if len(w)>1] wordfreqs = [(w, words.count(w)) for w in set(words)] c = ( WordCloud() .add("", wordfreqs, word_size_range=[10, 70]) .set_global_opts(title_opts=opts.TitleOpts(title="哈工大感恩節")) .render_notebook() )
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/238356.html
標籤：Python