上次發了一篇《熱榜標題詞云實時更新詞云上線，給標題起名提供參觀建議》，今天突然上熱榜一一段時間，還被范博at了，雖然對上熱榜本身已經無感，但是被范博at還非常讓我高興的，
這到晚上后，有人在評論區問到熱榜資料怎么爬，
熱心的我，現在就花幾分鐘時間教會大家，
首先，打開熱榜頁面并打開開發者工具，監控xhr型別的請求：
然后復制為curl(bash)型別的請求：
然后我們在jupyte中執行：
!curl2py
注意：該命令需要安裝filestools
安裝命令：pip install filestools
詳見：https://pypi.org/project/filestools/
然后就可以將當前請求的python代碼粘貼出來了，洗掉cookie部分后如下：
import requests
import json
headers = {
"authority": "blog.csdn.net",
"accept": "application/json, text/plain, */*",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "cors",
"sec-fetch-dest": "empty",
"referer": "https://blog.csdn.net/rank/list",
"accept-language": "zh-CN,zh;q=0.9"
}
params = {
"page": "0",
"pageSize": "25"
}
res = requests.get(
"https://blog.csdn.net/phoenix/web/blog/hot-rank",
params=params,
headers=headers,
)
print(res.text)
這樣我們就能獲取到熱榜第一頁的資料，
稍微改進一下代碼，回圈爬取四頁：
import pandas as pd
import requests
headers = {
"authority": "blog.csdn.net",
"accept": "application/json, text/plain, */*",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "cors",
"sec-fetch-dest": "empty",
"referer": "https://blog.csdn.net/rank/list",
"accept-language": "zh-CN,zh;q=0.9"
}
session = requests.session()
session.get("https://blog.csdn.net/rank/list", headers=headers)
params = {
"page": "",
"pageSize": "25"
}
result = []
for i in range(4):
params["page"] = str(i)
res = session.get(
"https://blog.csdn.net/phoenix/web/blog/hot-rank", params=params, headers=headers)
df = pd.DataFrame(res.json()['data'])[[
"userName", "nickName", "articleTitle", "viewCount",
"commentCount", "favorCount", "articleDetailUrl", "hotRankScore"]]
result.append(df)
hot_rank = pd.concat(result, ignore_index=True)
hot_rank.commentCount = hot_rank.commentCount.astype(int)
hot_rank.favorCount = hot_rank.favorCount.astype(int)
hot_rank.viewCount = hot_rank.viewCount.astype(int)
hot_rank.hotRankScore = hot_rank.hotRankScore.astype(int)
hot_rank.index += 1
hot_rank.reset_index(inplace=True)
hot_rank.columns = ["排名", "csdnid", "昵稱", "標題", "閱讀", "評論", "收藏", "鏈接", "熱度"]
hot_rank
這樣我們就獲取熱榜100條的所有資料了，
就是這么簡單，童鞋們都再也不用擔心不會采集熱榜了，
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/335561.html
標籤：python