所以我使用 Selenium 按下“加載更多”按鈕，一切都正確加載。然后我想獲取所有已加載產品的資訊，但我只獲取第一個“加載更多”按鈕之前的前 36 個專案的資訊。
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import json
import time
import requests
allinfo=[]
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_options.add_experimental_option('excludeSwitches', ['enable-logging'])
url="https://zadaa.co/de-en/products/women/clothes-dresses/"
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),chrome_options=chrome_options)
driver.get(url)
r=requests.get(url)
soup=BeautifulSoup(r.content,"html.parser")
wait = WebDriverWait(driver, 10)
closebutton=wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="content"]/div[5]/button')))
closebutton.click()
for x in range(9):
button = wait.until(EC.element_to_be_clickable((By.ID, "load-more-products")))
button.click()
content=soup.find_all('a',class_='product-list-item')
for properties in content:
brand=properties.find("p",class_='product-list-item-title').text
info={
'name':brand,
}
allinfo.append(info)
df=pd.DataFrame(allinfo)
print(df.head())
df.to_csv('zadaa.csv')
這是我要抓取的網頁 - https://zadaa.co/de-en/products/women/clothes-dresses/
抱歉有些奇怪的英語用法。
uj5u.com熱心網友回復：
您可以使用模塊模擬 Ajax 呼叫以requests直接獲取資料，而無需selenium（注意，有 12k 產品）：
import requests
from bs4 import BeautifulSoup
url = "https://zadaa.co/de-en/products/women/clothes-dresses/"
api_url = "https://zadaa.co/wp-admin/admin-ajax.php"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
payload = {
"action": "get_more_products",
"lang": "de-en",
"security": "05ef973f4c",
"query_id": soup.select_one("[data-query-id]")["data-query-id"],
"offset": 0,
}
while True:
data = requests.post(api_url, data=payload).json()
if not data["success"]:
break
soup = BeautifulSoup(data["data"], "html.parser")
for i in soup.select(".product-list-item"):
print(i.select_one(".product-list-item-title").text)
print(i["href"])
print("-" * 80)
payload["offset"] = 36
印刷：
...
CITY GIRL PARIS
https://zadaa.co/de-en/products/women/clothes-dresses/city-girl-paris/3735824/
--------------------------------------------------------------------------------
ZAFUL
https://zadaa.co/de-en/products/women/clothes-dresses/zaful/3735781/
--------------------------------------------------------------------------------
NKD
https://zadaa.co/de-en/products/women/clothes-dresses/nkd/3735768/
--------------------------------------------------------------------------------
GREAT RUMORS
https://zadaa.co/de-en/products/women/clothes-dresses/great-rumors/3735762/
--------------------------------------------------------------------------------
...and so on.
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/489823.html