Python爬蟲篇（代理IP）--lizaza.cn-有解無憂

在做網路爬蟲的程序中經常會遇到請求次數過多無法訪問的現象，這種情況下就可以使用代理IP來解決，但是網上的代理IP要么收費，要么沒有API介面，秉著能省則省的原則，自己創建一個代理IP庫，

廢話不多說，直接上代碼：

 1 import requests
 2 from bs4 import BeautifulSoup
 3 
 4 
 5 # 發送請求
 6 def GetInfo(url):
 7     headers = {
 8         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
 9     }
10     proxies = {"http": "https://119.180.173.81:8060"}
11     response = requests.get(url=url, proxies=proxies, headers=headers)
12     response.encoding = "utf8"
13     return response.text
14 
15 
16 # 將資料寫入檔案
17 def WriteData():
18     for i in range(100):
19         url = "https://www.xicidaili.com/nn/" + str(i+1)
20         data =https://www.cnblogs.com/lizaza/p/ GetData(url)
21         file = open('Proxies.txt', 'a+')
22         file.write(str(data))
23         file.close()
24 
25 
26 # 驗證該代理能否使用
27 def verify(proxies):
28     req = requests.get("https://www.baidu.com", proxies=proxies)
29     return req.status_code
30 
31 
32 # 決議頁面
33 def GetData(url):
34     data =https://www.cnblogs.com/lizaza/p/ list()
35     html = GetInfo(url)
36     soup = BeautifulSoup(html, "lxml")
37     table = soup.find_all("table", id="ip_list")
38     soup = BeautifulSoup(str(table[0]), "lxml")
39     trs = soup.find_all("tr")
40     del trs[0]
41     for tr in trs:
42         ip = tr.select("td")[1].get_text()
43         port = tr.select("td")[2].get_text()
44         protocol = tr.select("td")[5].get_text()
45         address = protocol.lower()+"://"+ip+":"+port
46         proxies = {'http': address}
47         if verify(proxies) == 200:
48             data.append(address)
49     return data
50 
51 if __name__ == '__main__':
52     WriteData()

回傳資料：

['http://111.222.141.127:8118', 'https://117.88.177.101:3000', 'http://183.166.136.144:8888', 'http://27.208.231.100:8060', 'http://123.169.99.177:9999', 'http://119.84.84.185:12345', 'http://101.132.190.101:80', 'https://114.99.54.65:8118', 'https://119.4.13.26:1133', 'http://58.253.158.177:9999', 'http://114.223.208.165:8118', 'http://112.84.73.53:9999']

源站地址：https://www.lizaza.cn/page23.html

轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/152956.html

標籤：Python

上一篇：Flask基礎知識

下一篇：如何優化Python占用的記憶體，面試必學