前言
之前爬取美團，馬蜂窩等網站的資料都挺順利，大眾點評（這個反爬機制有點麻煩）在磕磕絆絆中也算成功（重點是網頁頁數的變化和關鍵字的隱藏替換）但攜程居然遇到了瓶頸，
主要是查看源代碼時發現關鍵商戶資訊根本就找不到，就很奇怪，對于關鍵資訊評論發現翻頁時網頁的url不變，網上查了一下說是使用是動態的網址進行建構的，Ajax頁面加載，那么通用的request.get()就不能用了，所以采取模擬瀏覽器進行資料爬取，
為什么選取龜峰呢？因為正好才從龜峰回來，這幾乎是去過的人最少的5A級景區，大膽猜測資料可能沒有那么多（相比去的其他地方而言）而且去的時候驚喜的給免了門票，所以用這個來測驗，
PS：如有需要Python學習資料的小伙伴可以加下方的群去找免費管理員領取
可以免費領取原始碼、專案實戰視頻、PDF檔案等
第一步：打開網頁
可能需要登錄后才能查看評價
查看網頁源代碼
可以看到無法加載出關鍵評論資訊
F12查看評論的具體代碼，如下圖找到評論串列，找到url和request payload
至此已經獲取了網址最重要的內容
直接上完整代碼
import requests
import json
import time
import csv
import re
c=open(r'D:\guifeng.csv','a+',newline='',encoding='utf-8')
fieldnames=['user','time','score','content']
writer=csv.DictWriter(c,fieldnames=fieldnames)
writer.writeheader()
head = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0'}
postUrl = "https://sec-m.ctrip.com/restapi/soa2/12530/json/viewCommentList"
data_1 = {
"pageid": "290510",
"viewid": "127481",
"tagid": "-11",
"pagenum": "1",
"pagesize": "10",
"contentType": "json",
"SortType": "1",
"head": {
"appid": "100013776",
"cid": "09031010311083118856",
"ctok": "",
"cver": "1.0",
"lang": "01",
"sid": "8888",
"syscode": "09",
"auth": "",
"extension": [
{
"name": "protocal",
"value": "https"
}
]
},
"ver": "7.10.3.0319180000"
}
html = requests.post(postUrl, data=https://www.cnblogs.com/hhh188764/p/json.dumps(data_1)).text
html = json.loads(html)
jingqu = '龜峰'
pages = html['data']['totalpage']
datas = []
for j in range(pages):
data1 = {
"pageid": "290510",
"viewid": "127481",
"tagid": "-11",
"pagenum": str(j + 1),
"pagesize": "10",
"contentType": "json",
"SortType": "1",
"head": {
"appid": "100013776",
"cid": "09031010311083118856",
"ctok": "",
"cver": "1.0",
"lang": "01",
"sid": "8888",
"syscode": "09",
"auth": "",
"extension": [
{
"name": "protocal",
"value": "https"
}
]
},
"ver": "7.10.3.0319180000"
}
datas.append(data1)
for k in datas[:50]:
print('正在抓取第' + k['pagenum'] + "頁")
time.sleep(3)
html1 = requests.post(postUrl, data=https://www.cnblogs.com/hhh188764/p/json.dumps(k)).text
html1 = json.loads(html1)
comments = html1['data']['comments']
for i in comments:
user = i['uid']
time1 = i['date']
score = i['score']
content = i['content']
content = re.sub(" ", "", content)
content = re.sub("
", "", content)
writer.writerow({'user': user,'time':time1,'score': score,'content':content})
c.close()即
按F5直接運行得出檔案（自動存盤在D盤里）
右鍵以記事本打開（如果使用excel需要先另存為否則初次打開會是亂碼）
關鍵步驟解讀
postUrl = "https://sec-m.ctrip.com/restapi/soa2/12530/json/viewCommentList"這個網址是評論的通用網址不需要改動
可以改動的地方為：
c=open(r'D:\guifeng.csv','a+',newline='',encoding='utf-8')存盤的路徑可以自定義，這里我直接用的是D盤
data_1 = {
"pageid": "290510",
"viewid": "127481",
"tagid": "-11",
"pagenum": "1",
"pagesize": "10",
"contentType": "json",
"SortType": "1",
"head": {
"appid": "100013776",
"cid": "09031010311083118856",
"ctok": "",
"cver": "1.0",
"lang": "01",
"sid": "8888",
"syscode": "09",
"auth": "",
"extension": [
{
"name": "protocal",
"value": "https"
}
]
},
"ver": "7.10.3.0319180000"
}具體資料就是如下圖需爬取網頁評論代碼里紅圈的部分
同樣的這個也是
data1 = {
"pageid": "290510",
"viewid": "127481",
"tagid": "-11",
"pagenum": str(j + 1),
"pagesize": "10",
"contentType": "json",
"SortType": "1",
"head": {
"appid": "100013776",
"cid": "09031010311083118856",
"ctok": "",
"cver": "1.0",
"lang": "01",
"sid": "8888",
"syscode": "09",
"auth": "",
"extension": [
{
"name": "protocal",
"value": "https"
}
]
},
"ver": "7.10.3.0319180000"
}
datas.append(data1)注意：
"pagenum": str(j + 1),這一步不可更改
最后
for k in datas[:50]:
print('正在抓取第' + k['pagenum'] + "頁")
time.sleep(3)這里的爬取的頁數50是可以更改的
時間間隔3也是可以更改的
至此可以成功爬取攜程上任意一家的評論，yes！！！
本文的文字及圖片來源于網路,僅供學習、交流使用,不具有任何商業用途,著作權歸原作者所有,如有問題請及時聯系我們以作處理，
作者：一千口水牛
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/36404.html
標籤：Python