我正在撰寫一個腳本來每天從網站中提取財務資料，但是由于 cookie 過期，該腳本在幾個小時后停止作業。每次運行腳本（每天一次）時，我都需要生成新的標頭（我認為只是 x-xsrf-token 和 cookie），否則我會得到 401 狀態代碼。
這是我試圖從中提取資料的頁面：
https://www.barchart.com/futures/quotes/CLZ22/futures-prices?viewName=main&timeFrame=current
這是我試圖抓取的 XHR 回應網址：
https://www.barchart.com/proxies/core-api/v1/quotes/get?fields=symbol,contractSymbol,lastPrice,priceChange,openPrice,highPrice,lowPrice,previousPrice,volume,openInterest,tradeTime,symbolCode,symbolType,hasOptions&lists=futures.contractInRoot&root=CL&meta=field.shortName,field.type,field.description,lists.lastUpdate&hasOptions=true&page=1&limit=100&raw=1
我目前知道如何做到這一點的唯一方法是訪問網站并將 XHR 請求復制為 cURL (bash)，然后將其粘貼到 Postman 并手動將這些標頭粘貼到我現有的代碼中。
下面是 Postman 生成的代碼。我一直在試圖弄清楚如何從 URL 生成標題，這樣我就不必每天跑到 Postman 并手動重繪 標題。
import requests
url = "https://www.barchart.com/proxies/core-api/v1/quotes/get?fields=symbol,contractSymbol,lastPrice,priceChange,openPrice,highPrice,lowPrice,previousPrice,volume,openInterest,tradeTime,symbolCode,symbolType,hasOptions&lists=futures.contractInRoot&root=CL&meta=field.shortName,field.type,field.description,lists.lastUpdate&hasOptions=true&page=1&limit=100&raw=1"
payload={}
headers = {
'authority': 'www.barchart.com',
'accept': 'application/json',
'accept-language': 'en-US,en;q=0.9',
'cookie': 'webinar124WebinarClosed=true; market=eyJpdiI6IkovREZvUVlZMGFzM2x3b05wb3V4cGc9PSIsInZhbHVlIjoiSVJoT00rMTdWUFFYRlJiOG53OU12dTdjcUhEL3FKTW5XUy9FZFNjc1Z2VWkwdjV2RkNrWXpGZzYzMUNpK2IxbyIsIm1hYyI6IjkzNDYxOTg5OWQ5MzgxYjhlMGI4ODg4NDRlMDA1NWE3MjUxYTNmOTMzNzllYjBjYjhmNGM1ZGZiYjA0Yzk5ODEifQ==; bcFreeUserPageView=0; laravel_token=eyJpdiI6ImR3bGlHVTY3WEhGdkdEWlBleGtkMFE9PSIsInZhbHVlIjoiWnFJMml3dis3cVN0d2VIdDRCbFQvczRmVGZxcjFYYTF4YWpBd09NSVJBVXRQYVVLdWxnaUlTM3dXTElUaUJHK1VoQkxaQkdsRHNlTzZRU3c2R3NhZzVROUYvRHM4TTQ3V2srcHZLZG9Ra3BzOUZndXhxME4rSmtYODZHTWtmN3pmOENtRGZWQmdhUEZFc0FiZ0dSV1BEbC9acTVVQnBTOUl1Y2ZleW50WVAxSmYvMTdQQVZlN0lRQ25qR1BKQWZUMU1XbE5rcW14ZTYvTkpVbkpmcXc2RVRHUmtrUHlTNithNkJiY1ZTNG1rWkl1cHkxeVRWUU9zZUE2RFhoN2VYeGFnZytPN2RBZ3VPS0tJdVQxZUw4eHB1d2FZN3JKNlJ3QmllYWx2N21nUGlFb25OYXM0aFhjbFBCS0Q0ajJTSmMiLCJtYWMiOiJkODgwMjliMzM4MGI4M2E4Njk4MmE3ODYzMDY2ZmRkYjRmN2MzZGExYThhMTliMTE2YjNiZDQ1YzkzZWMzMWQwIn0=; XSRF-TOKEN=eyJpdiI6InQyUll2aHRCaXFlQkZIRXV0TjdaVGc9PSIsInZhbHVlIjoiSmtZaXlTbmVrTkJNVmEyUHQrUDFZN1RWNCt5cmFSanMxcnpTTW8vTjdrTU1RVlZQWktXNnhtakJjeVJ6Y0h3cFpkaWl4UnBvS28vTHNCUzNsM0ZRcXN2ZG9tWnFLTUVwdUZHY2VhNmxSRFg0ajhXU0lobFRZaFZRanhHZis4STkiLCJtYWMiOiI2MDc3NjIzNTAwMmY5MjlkNjRkMTVkYTZjYmNiM2RiNjg4ZDI1MmUzZWEzYjc1NWY0ZDNiZGNjNzY0ZGY2NGY5In0=; laravel_session=eyJpdiI6IlhPVGVDbTVURlpWRDcvNWVMWUgxclE9PSIsInZhbHVlIjoiVEJvTUVIVkRHOFlQUXNKcUJRaGtmZ2U4aVcrbE9JNDV3bG1adG1DLzVpSzI5Z0lqYlk2NU5TQkE5ZTAzMHZPL1VoVjJlZU9kSkYvT1VERFBsK1BnRUVzaGMzVlNiRFFTQzFPblEyMUFXSjM3dmdRQXhnTXFSaVYwSkNkZ3ZJS3UiLCJtYWMiOiJjMzUzMzAyMjEzYzYwZGZmM2M3OTMwMGE0OGM3NTJmM2M3MzhkNDUyYjE2OTI4Njg5ODQxNDM3NjcyMzM0ZWE5In0=',
'referer': 'https://www.barchart.com/futures/quotes/CLZ22/futures-prices?viewName=main&timeFrame=current',
'sec-ch-ua': '"Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
'x-xsrf-token': 'eyJpdiI6InQyUll2aHRCaXFlQkZIRXV0TjdaVGc9PSIsInZhbHVlIjoiSmtZaXlTbmVrTkJNVmEyUHQrUDFZN1RWNCt5cmFSanMxcnpTTW8vTjdrTU1RVlZQWktXNnhtakJjeVJ6Y0h3cFpkaWl4UnBvS28vTHNCUzNsM0ZRcXN2ZG9tWnFLTUVwdUZHY2VhNmxSRFg0ajhXU0lobFRZaFZRanhHZis4STkiLCJtYWMiOiI2MDc3NjIzNTAwMmY5MjlkNjRkMTVkYTZjYmNiM2RiNjg4ZDI1MmUzZWEzYjc1NWY0ZDNiZGNjNzY0ZGY2NGY5In0='
}
response = requests.request("GET", url, headers=headers, data=payload)
print(response.text)
有沒有辦法從 xhr 回應 url 在 Python 中生成這些標頭，然后我可以在發送GET請求時使用這些標頭？
uj5u.com熱心網友回復：
餅干是瓶頸。您首先必須獲取它們，然后將它們與請求一起傳遞：
import requests
from urllib.parse import unquote
ua_headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0"
}
params = {
'fields': 'symbol,contractSymbol,lastPrice,priceChange,openPrice,highPrice,lowPrice,previousPrice,volume,openInterest,tradeTime,symbolCode,symbolType,hasOptions',
'lists': 'futures.contractInRoot',
'root': 'CL',
'meta': 'field.shortName,field.type,field.description,lists.lastUpdate',
'hasOptions': 'true',
'page': '1',
'limit': '100',
'raw': '1',
}
with requests.Session() as s:
# get cookies
s.get("https://www.barchart.com/options/iv-rank-percentile/stocks", headers=ua_headers)
# use one cookie as HTTP header
headers["X-XSRF-TOKEN"] = unquote(s.cookies["XSRF-TOKEN"])
response = s.get('https://www.barchart.com/proxies/core-api/v1/quotes/get', params=params, headers=headers)
print(response.json())
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/522196.html