爬蟲代理 IP 池及隧道代理
目錄- 爬蟲代理 IP 池及隧道代理
- 1. 代理 IP 池
- 1.1 簡介
- 1.2 實作
- 1.3 測驗
- 2. 隧道代理
- 2.1 簡介
- 2.2 實作
- 2.2.1 目錄結構
- 2.2.2 組態檔
- 2.2.3 openresty
- 2.3 測驗
- 1. 代理 IP 池
日常開發中，偶爾會遇到爬取網頁資料的需求，為了隱藏本機真實 IP，常常會用到代理 IP 池，本文將基于 openresty 與代理 IP 池搭建更為易用的隧道代理，
1. 代理 IP 池
1.1 簡介
代理 IP 池即在資料庫中維護一個可用的 IP 代理佇列，一般實作思路如下：
- 定時從免費或收費代理網站獲取代理 IP 串列；
- 將代理 IP 串列以 Hash 結構存入 Redis；
- 定時檢測代理 IP 的可用性，剔除不可用的代理 IP；
- 對外提供 API 介面用來管理代理 IP 池；
1.2 實作
此處筆者采用的開源專案jhao104/proxy_pool，具體實作方式參考其檔案，
1.3 測驗
import json
import requests
from retrying import retry
def get_proxy_ip() -> str:
resp = requests.get(url="http://192.168.0.121:5010/get")
assert resp.status_code == 200
return f"http://{json.loads(resp.text)['proxy']}"
@retry(stop_max_attempt_number=5)
def proxy_test() -> None:
resp = requests.get(url="http://httpbin.org/get", proxies={"http": get_proxy_ip()}, timeout=5)
assert resp.status_code == 200
print(f"origin: {json.loads(resp.text)['origin']}")
if __name__ == "__main__":
try:
proxy_test()
except Exception as e:
print(f"Error: {e}.")
2. 隧道代理
2.1 簡介
通過代理 IP 池實作了隱藏本機真實 IP，但每次需要通過 API 介面獲取新的代理 IP，不太方便，所以出現了隧道代理，隧道代理內部自動將請求通過不同的代理 IP 進行轉發，對外提供統一的代理地址，
2.2 實作
此處筆者通過 openresty 配合上文搭建的代理 IP 池實作隧道代理，
2.2.1 目錄結構
openresty
├── conf.d
│ └── tunnel-proxy.stream
├── docker.sh
└── nginx.conf
2.2.2 組態檔
-
nginx.conf檔案為 openresty 的主組態檔，主要修改為引入了 stream 相關的組態檔，具體內容如下：# nginx.conf -- docker-openresty # # This file is installed to: # `/usr/local/openresty/nginx/conf/nginx.conf` # and is the file loaded by nginx at startup, # unless the user specifies otherwise. # # It tracks the upstream OpenResty's `nginx.conf`, but removes the `server` # section and adds this directive: # `include /etc/nginx/conf.d/*.conf;` # # The `docker-openresty` file `nginx.vh.default.conf` is copied to # `/etc/nginx/conf.d/default.conf`. It contains the `server section # of the upstream `nginx.conf`. # # See https://github.com/openresty/docker-openresty/blob/master/README.md#nginx-config-files # #user nobody; #worker_processes 1; # Enables the use of JIT for regular expressions to speed-up their processing. pcre_jit on; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; # Enables or disables the use of underscores in client request header fields. # When the use of underscores is disabled, request header fields whose names contain underscores are marked as invalid and become subject to the ignore_invalid_headers directive. # underscores_in_headers off; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; # Log in JSON Format # log_format nginxlog_json escape=json '{ "timestamp": "$time_iso8601", ' # '"remote_addr": "$remote_addr", ' # '"body_bytes_sent": $body_bytes_sent, ' # '"request_time": $request_time, ' # '"response_status": $status, ' # '"request": "$request", ' # '"request_method": "$request_method", ' # '"host": "$host",' # '"upstream_addr": "$upstream_addr",' # '"http_x_forwarded_for": "$http_x_forwarded_for",' # '"http_referrer": "$http_referer", ' # '"http_user_agent": "$http_user_agent", ' # '"http_version": "$server_protocol", ' # '"nginx_access": true }'; # access_log /dev/stdout nginxlog_json; # See Move default writable paths to a dedicated directory (#119) # https://github.com/openresty/docker-openresty/issues/119 client_body_temp_path /var/run/openresty/nginx-client-body; proxy_temp_path /var/run/openresty/nginx-proxy; fastcgi_temp_path /var/run/openresty/nginx-fastcgi; uwsgi_temp_path /var/run/openresty/nginx-uwsgi; scgi_temp_path /var/run/openresty/nginx-scgi; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; include /etc/nginx/conf.d/*.conf; # Don't reveal OpenResty version to clients. # server_tokens off; } stream { log_format proxy '$remote_addr [$time_local] ' '$protocol $status $bytes_sent $bytes_received ' '$session_time "$upstream_addr" ' '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"'; access_log /usr/local/openresty/nginx/logs/access.log proxy; error_log /usr/local/openresty/nginx/logs/error.log notice; open_log_file_cache off; include /etc/nginx/conf.d/*.stream; } -
tunnel-proxy.stream為配置隧道代理的檔案，通過查詢 Redis 獲取代理 IP，并將請求通過代理 IP 轉發到指定目標地址，具體內容如下：# tunnel-proxy.stream upstream backend { server 0.0.0.0:9870; balancer_by_lua_block { local balancer = require "ngx.balancer" local host = ngx.ctx.proxy_host local port = ngx.ctx.proxy_port local success, msg = balancer.set_current_peer(host, port) if not success then ngx.log(ngx.ERR, "Failed to set the peer. Error: ", msg, ".") end } } server { # 對外代理監聽埠 listen 9870; listen [::]:9870; proxy_connect_timeout 10s; proxy_timeout 10s; proxy_pass backend; preread_by_lua_block { local redis = require("resty.redis") local redis_instance = redis:new() redis_instance:set_timeout(3000) # Redis 地址 local rhost = "192.168.0.121" # Redis 埠 local rport = 6379 # Redis 資料庫 local database = 0 # Redis Hash 鍵名 local rkey = "use_proxy" local success, msg = redis_instance:connect(rhost, rport) if not success then ngx.log(ngx.ERR, "Failed to connect to redis. Error: ", msg, ".") end redis_instance:select(database) local proxys, msg = redis_instance:hkeys(rkey) if not proxys then ngx.log(ngx.ERR, "Proxys num error. Error: ", msg, ".") return redis_instance:close() end math.randomseed(tostring(ngx.now()):reverse():sub(1, 6)) local proxy = proxys[math.random(#proxys)] local colon_index = string.find(proxy, ":") local proxy_ip = string.sub(proxy, 1, colon_index - 1) local proxy_port = string.sub(proxy, colon_index + 1) ngx.log(ngx.NOTICE, "Proxy: ", proxy, ", ip: ", proxy_ip, ", port: ", proxy_port, "."); ngx.ctx.proxy_host = proxy_ip ngx.ctx.proxy_port = proxy_port redis_instance:close() } }
2.2.3 openresty
通過 docker 啟動 openresty，此處筆者為了方便，將 docker 命令保存成了 shell 檔案，具體內容如下：
docker run --name openresty -itd --restart always \
-p 9870:9870 \
-v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf \
-v $PWD/conf.d:/etc/nginx/conf.d \
-e LANG=C.UTF-8 \
-e TZ=Asia/Shanghai \
--log-driver json-file \
--log-opt max-size=1g \
--log-opt max-file=3 \
openresty/openresty:alpine
執行 bash docker.sh 命名啟動 openresty，至此隧道代理搭建完成，
2.3 測驗
import json
import requests
from retrying import retry
proxies = {
"http": "http://192.168.0.121:9870"
}
@retry(stop_max_attempt_number=5)
def proxy_test() -> None:
resp = requests.get(
url="http://httpbin.org/get", proxies=proxies, timeout=5, )
assert resp.status_code == 200
print(f"origin: {json.loads(resp.text)['origin']}")
if __name__ == "__main__":
try:
proxy_test()
except Exception as e:
print(f"Error: {e}.")
參考鏈接：
- 只要5分鐘，創建一個隧道代理 - 知乎 (zhihu.com)
- openresty正向代理搭建 - 簡書 (jianshu.com)
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/481309.html
標籤：Python
下一篇：Django中間件