我正在使用以下代碼嘗試繼續單擊“加載更多”按鈕，直到所有頁面結果都顯示在網站上：
from selenium import webdriver
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def startWebDriver():
global driver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(options = chrome_options)
startWebDriver()
driver.get("https://together.bunq.com/all")
time.sleep(4)
while True:
try:
wait = WebDriverWait(driver, 10,10)
element = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[title='Load More']")))
element.click()
print("Loading more page results")
except:
print("All page results displayed")
break;
但是，由于單擊按鈕不會更改 URL，因此不會將新資料加載到 chromedriver 中，while 回圈將在第二次迭代時中斷。
uj5u.com熱心網友回復：
Selenium 在這方面太過分了。你只需要requests. 記錄一個人的網路流量顯示，在某些時候，JavaScript 會向 REST API 端點發出 XHR HTTP GET 請求，其回應是 JSON 并包含您可能想要抓取的所有資訊。
該端點 URL 的查詢字串引數之一是page[offset]，用于偏移分頁的查詢結果（在本例中為“加載更多按鈕”）。值0對應于沒有偏移，或“從頭??開始”。增加此值以滿足您的需要 - 在回圈中可能是執行此操作的好地方。
只需模仿 XHR HTTP GET 請求 - 復制 API 端點 URL 和查詢字串引數以及請求標頭，然后決議 JSON 回應：
def get_discussions():
import requests
url = "https://together.bunq.com/api/discussions"
params = {
"include": "user,lastPostedUser,tags,firstPost",
"page[offset]": 0
}
headers = {
"user-agent": "Mozilla/5.0"
}
response = requests.get(url, params=params, headers=headers)
response.raise_for_status()
yield from response.json()["data"]
def main():
for discussion in get_discussions():
print(discussion["attributes"]["title"])
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
輸出：
??What’s new in App Update 18.8.0
Local Currencies Accounts Fees
Local Currencies ??
Spanish IBANs Service Coverage ????
bunq Update 18 FAQ ??
Phishing and Spoofing - The new ways of scamming, explained ??
Easily rent a car with your bunq credit card ??
Giveaway - Hallo Deutschland! ????
Giveaway - Hello Germany! ????
True Name: Feedback ??
True Name ??
What plans are available?
Everything about refunds ??
Identity verification explained! ??
When will I receive my payment?
Together Community Guidelines ??
What is my Tax Identification Number (TIN)?
How do I export a bank statement?
How do I change my contact info?
Large cash withdrael
If this is a new concept for you, I would suggest you look up tutorials on how to use your browser's developer tools (Google Chrome's Devtools, for example), how to log your browser's network traffic, REST APIs, HTTP, etc.
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/344497.html