我想用 Beautiful Soup抓取這個網站： https ://cage.dla.mil/Home/UsageAgree。我在做什么：
import requests
url = "https://cage.dla.mil/Home/UsageAgree"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")
print(soup)
它從 cookie 協議頁面回傳 HTML。然后，我正在尋找的是在我們接受 cookie 后繞過它來抓取實際頁面的內容。
我關注了這篇文章：使用需要單擊“我同意 cookie”按鈕的 Python（美麗湯）抓取網頁？
并做了：
import requests
url = 'https://cage.dla.mil/'
s = requests.Session()
s.cookies.update({'agree': 'True'})
s.get(url)
soup = BeautifulSoup(r.content, "html.parser")
print(soup)
但我仍然得到協議頁面。
似乎其中一個 cookie 總是提供獨特的價值。我不確定如何處理這個問題。
uj5u.com熱心網友回復：
嗯，這應該作業。
import requests
from bs4 import BeautifulSoup
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"
}
with requests.Session() as s:
token = (
BeautifulSoup(
s.get(
"https://cage.dla.mil/Home/UsageAgree",
headers=headers,
).text,
"lxml",
).select_one("form input")["value"]
)
payload = {
"__RequestVerificationToken": token,
"returningURL": "",
}
_ = s.post(
"https://cage.dla.mil/Home/UsageAgree",
data=payload,
headers=headers
)
soup = (
BeautifulSoup(
s.get("https://cage.dla.mil/", headers=headers).text,
"lxml",
).select("#briefnewslist > div > p > em")
)
print("\n".join(p.getText(strip=True) for p in soup))
輸出：
Scheduled Maintenance
SAM Validation: Unable To Find A Matching Entity When Asked To Enter Or Validate My Entity Information
SAM Validation: Continue A Registration Update Or Renewal If Validation Fails
SAM.gov Registration for Financial Assistance
Financial Assistance Update
CAGE Expiration Date
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/490648.html