在本篇博文當中，將會教會大家如何使用高性能爬蟲，快速爬取并決議頁面當中的資訊，一般情況下，如果我們請求網頁的次數太多，每次都要發出一次請求，進行串行執行的話，那么請求將會占用我們大量的時間，這樣得不償失，因此我們可以i使用高性能爬蟲，也就是采用多行程，異步的方式對資料進行爬取和決議，這樣就可以在更快的時間內得到我們想要的結果，本篇博文給出有關爬取豆瓣電影的例子，以此來教會大家如何使用高性能爬蟲，
一.網頁分析
首先我們來分析豆瓣電影的網頁代碼，在本次的案例當中，我們需要爬取豆瓣電影top250當中的標題title和星數star，
發現，豆瓣電影當中的所有有關電影的資訊全部都隱藏在< ol >這個標簽，當中，因此我們在撰寫xpath的時候，可以利用對它做一個回圈，然后又發現，對于電影的title而言，有兩個地方出現，一個地方是在圖片上，另一個地方是在span標簽下的class = title處，但是在span標簽下具有多個標題，為了以免引起混，因此我們使用圖片當中所暗含的標題title文字，使用xpath進行定位即可，
對于star而言，就更加簡單了，我們發現每次一個star的分數出現，就會有又一個<div >的標簽在前面，然后再出現了與span有關的標簽，因此我們撰寫xpath運算式為：//ol[@]//div[@]/span[@]/text()
這樣就可以得到一整個頁面的star的數值了，當然這樣我們只能獲取第一頁的我們想要得到的資料，怎么得到第二頁的資料呢？
二.翻頁處理
翻頁處理對于豆瓣電影這個網站還是比較簡單的，我們分別查看第一，二，三頁的url，就會驚奇的發現它的網址如下：
https://movie.douban.com/top250?start=0&filter= https://movie.douban.com/top250?start=25&filter= https://movie.douban.com/top250?start=50&filter=
十分明顯，這個網址后面有問號說明想要獲取頁面內容肯定需要發起get請求，都沒有做有關post請求的加密，這樣看來這也太簡單了吧！
同樣的我們發現里面的引數start在不斷的變化，而filter卻保持不變，因此我們只需要得到start引數的規律就知道該怎么撰寫爬蟲了，
對于start而言，每跳轉一頁，就會增加25的數值，因為每一個頁面里面均僅有25部電影，這樣我們就找到了start引數的規律，開始撰寫爬蟲，
三.爬蟲代碼的撰寫
在撰寫的代碼時候，我們匯入了多行程的庫，使用這個庫進行爬蟲，也就只需要在原本代碼的基礎之上多添加兩行代碼即可，如下所示：
pool=Pool(4)
pool.map(get_information,number_ls)
這兩行代碼當中，第一個引數的4表示了我們使用4個行程的行程池進行資料的抓取，數值越大，爬取的效率就越高，這取決于你CPU的數量，數值不能超過CPU核心數的數量，因為一個一個CPU核心同時只能夠運行單個行程，
第二行代碼使用了map函式，第一個引數填寫我們進行爬蟲的函式，第二個引數填寫爬蟲函式所需要的引數，把這兩個東西放到map函式里，就可以開始高性能爬蟲了，
Remark：
在進行行程池爬蟲的時候，我們放入的引數number_ls一定是一個串列，同時我們在get_imformation函式里使用得到的引數時，每次系統會呼叫這個引數串列當中的任意一個數值，而不是對整個串列進行呼叫，
由于整個原因，因此我們撰寫整個的代碼·如下所示：
import requests from lxml import etree from multiprocessing.dummy import Pool cookie='bid=N3Zqe_FFUKc; douban-fav-remind=1; viewed="27093751"; _vwo_uuid_v2=D401F17C96234AE149C4E04B78C3C8066|6fcc3cefe576bff2b89cdf28c4c5f597; __gads=ID=21cdec44606b00df-2250ba4d7ac4009b:T=1604034713:RT=1604034713:S=ALNI_Mb6iYJKYfbUjLxlisTQX5HCODTGKg; gr_user_id=fb6ac40c-94c3-400e-b170-47e126a9b78a; _gid=GA1.2.1520341169.1612004212; _ga=GA1.2.645228582.1602221486; ll="108288"; UM_distinctid=17752f076e4530-0b6eef25ebabba-f7b1332-1fa400-17752f076e57f0; Hm_lvt_19fc7b106453f97b6a84d64302f21a04=1612004228; Hm_lpvt_19fc7b106453f97b6a84d64302f21a04=1612004253; ap_v=0,6.0; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1612004299%2C%22https%3A%2F%2Fwww.google.com%2F%22%5D; _pk_ses.100001.4cf6=*; __utma=30149280.645228582.1602221486.1611225800.1612004300.9; __utmb=30149280.0.10.1612004300; __utmc=30149280; __utmz=30149280.1612004300.9.9.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __utma=223695111.645228582.1602221486.1612004300.1612004300.1; __utmb=223695111.0.10.1612004300; __utmc=223695111; __utmz=223695111.1612004300.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _pk_id.100001.4cf6=9a1bb1df4597b334.1612004299.1.1612005471.1612004299.' headers={ 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36', } url='https://movie.douban.com/top250' number_ls=[] for i in range(0,251,25): number_ls.append(i) print(number_ls) def get_information(number_ls): param={ 'start':number_ls, 'filter' :'' } page_content=requests.get(url=url,headers=headers,params=param).text with open('douban.html','w',encoding='utf-8') as fp: fp.write(page_content) tree=etree.HTML(page_content) vedio_title=tree.xpath('//ol[@]//div[@]//a/img/@alt') star=tree.xpath('//ol[@]//div[@]/span[@]/text()') vedio_title_ls=[] star_ls=[] for i in vedio_title: vedio_title_ls.append(i) for i in star: star_ls.append(i) j=0 while j<len(star_ls): print("the movie is ",vedio_title_ls[j]) print("the star is ",star_ls[j]) print() j+=1 pool=Pool(4) pool.map(get_information,number_ls)
四.輸出的結果
輸出的結果十分完美，一共有250份電影，如下圖所示：
[0, 25, 50, 75, 100, 125, 150, 175, 200, 225, 250] the movie is 搏擊俱樂部 the star is 9.0 the movie is 教父2 the star is 9.2 the movie is 獅子王 the star is 9.0 the movie is 指環王2：雙塔奇兵 the star is 9.1 the movie is 死亡詩社 the star is 9.1 the movie is 鋼琴家 the star is 9.2 the movie is 黑客帝國 the star is 9.0 the movie is 指環王1：魔戒再現 the star is 9.0 the movie is 飲食男女 the star is 9.1 the movie is 竊聽風暴 the star is 9.1 the movie is 美麗心靈 the star is 9.0 the movie is 讓子彈飛 the star is 8.8 the movie is 綠皮書 the star is 8.9 the movie is 兩桿大煙槍 the star is 9.1 the movie is 本杰明·巴頓奇事 the star is 8.9 the movie is 海蒂和爺爺 the star is 9.2 the movie is 飛越瘋人院 the star is 9.1 the movie is 看不見的客人 the star is 8.8 the movie is 西西里的美麗傳說 the star is 8.9 the movie is 拯救大兵瑞恩 the star is 9.0 the movie is 穿條紋睡衣的男孩 the star is 9.1 the movie is 小鞋子 the star is 9.2 the movie is 音樂之聲 the star is 9.0 the movie is 情書 the star is 8.9 the movie is 海豚灣 the star is 9.3 the movie is 美國往事 the star is 9.2 the movie is 致命魔術 the star is 8.9 the movie is 沉默的羔羊 the star is 8.9 the movie is 低俗小說 the star is 8.9 the movie is 禁閉島 the star is 8.8 the movie is 蝴蝶效應 the star is 8.8 the movie is 七宗罪 the star is 8.8 the movie is 心靈捕手 the star is 8.9 the movie is 布達佩斯大飯店 the star is 8.9 the movie is 春光乍泄 the star is 8.9 the movie is 摩登時代 the star is 9.3 the movie is 被嫌棄的松子的一生 the star is 8.9 the movie is 哈利·波特與死亡圣器(下) the star is 8.9 the movie is 阿凡達 the star is 8.7 the movie is 喜劇之王 the star is 8.8 the movie is 致命ID the star is 8.8 the movie is 剪刀手愛德華 the star is 8.7 the movie is 勇敢的心 the star is 8.9 the movie is 加勒比海盜 the star is 8.8 the movie is 殺人回憶 the star is 8.9 the movie is 狩獵 the star is 9.1 the movie is 請以你的名字呼喚我 the star is 8.9 the movie is 天使愛美麗 the star is 8.7 the movie is 斷背山 the star is 8.8 the movie is 紅辣椒 the star is 9.0 the movie is 觸不可及 the star is 9.2 the movie is 蝙蝠俠：黑暗騎士 the star is 9.2 the movie is 末代皇帝 the star is 9.3 the movie is 活著 the star is 9.3 the movie is 尋夢環游記 the star is 9.1 the movie is 亂世佳人 the star is 9.3 the movie is 何以為家 the star is 9.1 the movie is 指環王3：王者無敵 the star is 9.2 the movie is 飛屋環游記 the star is 9.0 the movie is 摔跤吧！爸爸 the star is 9.0 the movie is 哈利·波特與魔法石 the star is 9.1 the movie is 素媛 the star is 9.3 the movie is 少年派的奇幻漂流 the star is 9.1 the movie is 十二怒漢 the star is 9.4 the movie is 哈爾的移動城堡 the star is 9.1 the movie is 鬼子來了 the star is 9.3 the movie is 天空之城 the star is 9.1 the movie is 大話西游之月光寶盒 the star is 9.0 the movie is 我不是藥神 the star is 9.0 the movie is 聞香識女人 the star is 9.1 the movie is 羅馬假日 the star is 9.0 the movie is 天堂電影院 the star is 9.2 the movie is 辯護人 the star is 9.2 the movie is 貓鼠游戲 the star is 9.0 the movie is 大鬧天宮 the star is 9.4 the movie is 肖申克的救贖 the star is 9.7 the movie is 霸王別姬 the star is 9.6 the movie is 阿甘正傳 the star is 9.5 the movie is 這個殺手不太冷 the star is 9.4 the movie is 泰坦尼克號 the star is 9.4 the movie is 美麗人生 the star is 9.5 the movie is 千與千尋 the star is 9.4 the movie is 辛德勒的名單 the star is 9.5 the movie is 盜夢空間 the star is 9.3 the movie is 忠犬八公的故事 the star is 9.4 the movie is 星際穿越 the star is 9.3 the movie is 海上鋼琴師 the star is 9.3 the movie is 楚門的世界 the star is 9.3 the movie is 三傻大鬧寶萊塢 the star is 9.2 the movie is 機器人總動員 the star is 9.3 the movie is 放牛班的春天 the star is 9.3 the movie is 大話西游之大圣娶親 the star is 9.2 the movie is 瘋狂動物城 the star is 9.2 the movie is 無間道 the star is 9.2 the movie is 熔爐 the star is 9.3 the movie is 教父 the star is 9.3 the movie is 當幸福來敲門 the star is 9.1 the movie is 龍貓 the star is 9.2 the movie is 怦然心動 the star is 9.1 the movie is 控方證人 the star is 9.6 the movie is 7號房的禮物 the star is 8.9 the movie is 幽靈公主 the star is 8.9 the movie is 小森林 夏秋篇 the star is 9.0 the movie is 陽光燦爛的日子 the star is 8.8 the movie is 第六感 the star is 8.9 the movie is 重慶森林 the star is 8.8 the movie is 入殮師 the star is 8.9 the movie is 唐伯虎點秋香 the star is 8.7 the movie is 小森林 冬春篇 the star is 9.0 the movie is 愛在黎明破曉前 the star is 8.8 the movie is 超脫 the star is 8.9 the movie is 消失的愛人 the star is 8.7 the movie is 一一 the star is 9.0 the movie is 菊次郎的夏天 the star is 8.8 the movie is 蝙蝠俠：黑暗騎士崛起 the star is 8.8 the movie is 側耳傾聽 the star is 8.9 the movie is 倩女幽魂 the star is 8.7 the movie is 功夫 the star is 8.6 the movie is 超能陸戰隊 the star is 8.7 the movie is 無人知曉 the star is 9.1 the movie is 人生果實 the star is 9.5 the movie is 螢火之森 the star is 8.9 the movie is 甜蜜蜜 the star is 8.8 the movie is 借東西的小人阿莉埃蒂 the star is 8.8 the movie is 瑪麗和馬克思 the star is 8.9 the movie is 愛在日落黃昏時 the star is 8.8 the movie is 馴龍高手 the star is 8.7 the movie is 完美的世界 the star is 9.1 the movie is 幸福終點站 the star is 8.8 the movie is 告白 the star is 8.7 the movie is 大魚 the star is 8.8 the movie is 陽光姐妹淘 the star is 8.8 the movie is 射雕英雄傳之東成西就 the star is 8.7 the movie is 哈利·波特與阿茲卡班的囚徒 the star is 8.8 the movie is 恐怖直播 the star is 8.8 the movie is 天書奇譚 the star is 9.2 the movie is 怪獸電力公司 the star is 8.7 the movie is 神偷奶爸 the star is 8.6 the movie is 玩具總動員3 the star is 8.8 the movie is 傲慢與偏見 the star is 8.6 the movie is 時空戀旅人 the star is 8.8 the movie is 哈利·波特與密室 the star is 8.7 the movie is 教父3 the star is 8.9 the movie is 釜山行 the star is 8.6 the movie is 血戰鋼鋸嶺 the star is 8.7 the movie is 哪吒鬧海 the star is 9.1 the movie is 被解救的姜戈 the star is 8.7 the movie is 七武士 the star is 9.3 the movie is 喜宴 the star is 8.9 the movie is 電鋸驚魂 the star is 8.7 the movie is 爆裂鼓手 the star is 8.7 the movie is 貧民窟的百萬富翁 the star is 8.6 the movie is 螢火蟲之墓 the star is 8.7 the movie is 東邪西毒 the star is 8.6 the movie is 海街日記 the star is 8.8 the movie is 黑天鵝 the star is 8.6 the movie is 驚魂記 the star is 9.0 the movie is 無敵破壞王 the star is 8.7 the movie is 你看起來好像很好吃 the star is 8.9 the movie is 冰川時代 the star is 8.6 the movie is 雨人 the star is 8.7 the movie is 小偷家族 the star is 8.7 the movie is 綠里奇跡 the star is 8.9 the movie is 戀戀筆記本 the star is 8.5 the movie is 愛在午夜降臨前 the star is 8.8 the movie is 瘋狂的石頭 the star is 8.5 the movie is 哈利·波特與火焰杯 the star is 8.6 the movie is 寄生蟲 the star is 8.7 the movie is 恐怖游輪 the star is 8.5 the movie is 奇跡男孩 the star is 8.6 the movie is 雨中曲 the star is 9.0 the movie is 魔女宅急便 the star is 8.7 the movie is 二十二 the star is 8.7 the movie is 海邊的曼徹斯特 the star is 8.6 the movie is 房間 the star is 8.8 the movie is 風之谷 the star is 8.9 the movie is 一個叫歐維的男人決定去死 the star is 8.9 the movie is 我是山姆 the star is 8.9 the movie is 頭號玩家 the star is 8.7 the movie is 英雄本色 the star is 8.7 the movie is 上帝之城 the star is 9.0 the movie is 諜影重重3 the star is 8.8 the movie is 瘋狂原始人 the star is 8.7 the movie is 未麻的部屋 the star is 9.0 the movie is 歲月神偷 the star is 8.7 the movie is 盧旺達飯店 the star is 8.9 the movie is 縱橫四海 the star is 8.8 the movie is 三塊廣告牌 the star is 8.7 the movie is 達拉斯買家俱樂部 the star is 8.8 the movie is 花樣年華 the star is 8.7 the movie is 心迷宮 the star is 8.7 the movie is 記憶碎片 the star is 8.6 the movie is 模仿游戲 the star is 8.7 the movie is 黑客帝國3：矩陣革命 the star is 8.8 the movie is 新世界 the star is 8.8 the movie is 頭腦特工隊 the star is 8.7 the movie is 荒蠻故事 the star is 8.8 the movie is 你的名字， the star is 8.4 the movie is 真愛至上 the star is 8.6 the movie is 忠犬八公物語 the star is 9.2 the movie is 諜影重重2 the star is 8.7 the movie is 阿飛正傳 the star is 8.5 the movie is 地球上的星星 the star is 8.9 the movie is 彗星來的那一夜 the star is 8.5 the movie is 完美陌生人 the star is 8.5 the movie is 戰爭之王 the star is 8.7 the movie is 諜影重重 the star is 8.6 the movie is 香水 the star is 8.5 the movie is 東京教父 the star is 9.0 the movie is 東京物語 the star is 9.2 the movie is 朗讀者 the star is 8.6 the movie is 千鈞一發 the star is 8.8 the movie is 再次出發之紐約遇見你 the star is 8.6 the movie is 驢得水 the star is 8.3 the movie is 猜火車 the star is 8.5 the movie is 黑客帝國2：重裝上陣 the star is 8.6 the movie is 無間道2 the star is 8.6 the movie is 我愛你 the star is 9.1 the movie is 浪潮 the star is 8.7 the movie is 崖上的波妞 the star is 8.5 the movie is 聚焦 the star is 8.8 the movie is 小蘿莉的猴神大叔 the star is 8.4 the movie is 追隨 the star is 8.9 the movie is 黑鷹墜落 the star is 8.7 the movie is 網路謎蹤 the star is 8.6 the movie is 虎口脫險 the star is 8.9 the movie is 人工智能 the star is 8.7 the movie is 九品芝麻官 the star is 8.6 the movie is 2001太空漫游 the star is 8.8 the movie is 可可西里 the star is 8.8 the movie is 羅生門 the star is 8.8 the movie is 色，戒 the star is 8.5 the movie is 終結者2：審判日 the star is 8.7 the movie is 城市之光 the star is 9.3 the movie is 初戀這件小事 the star is 8.4 the movie is 魂斷藍橋 the star is 8.8 the movie is 牯嶺街少年殺人事件 the star is 8.9 the movie is 遺愿清單 the star is 8.7 the movie is 大佛普拉斯 the star is 8.7 the movie is 新龍門客堆疊 the star is 8.6 the movie is 波西米亞狂想曲 the star is 8.7 the movie is 源代碼 the star is 8.5 the movie is 青蛇 the star is 8.6 the movie is 海洋 the star is 9.1 the movie is 燃情歲月 the star is 8.8 the movie is 無恥混蛋 the star is 8.6 the movie is 瘋狂的麥克斯4：狂暴之路 the star is 8.6 the movie is 血鉆 the star is 8.7 the movie is 穿越時空的少女 the star is 8.6 the movie is 步履不停 the star is 8.8
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/255062.html
標籤：Python
下一篇：839相似字串