我正在嘗試為以下頁面上顯示的每個專案抓取一些資訊： https ://www.finewineandgoodspirits.com/webapp/wcs/stores/servlet/CatalogSearchResultView?storeId=10051&catalogId=10051&langId=-1&categoryId=1351370&variety=New Spirits&categoryType=Spirits&top_category=25208&sortBy=0&searchSource=E&pageView=&beginIndex=0#facet:&productBeginIndex:0&orderBy:&pageView:&minPrice:&maxPrice:&pageSize:&
但是，我似乎無法訪問專案資訊。我需要的資訊是每個產品的名稱和鏈接，例如第一項包含在：
<a aria-hidden="true" tabindex="-1" id="WC_CatalogEntryDBThumbnailDisplayJSPF_3074457345616901168_link_9b" href="/webapp/wcs/stores/servlet/ProductDisplay?catalogId=10051&storeId=10051&productId=3074457345616901168&langId=-1&partNumber=000086630prod&errorViewName=ProductDisplayErrorView&categoryId=1351370&top_category=25208&parent_category_rn=25208&urlLangId=&variety=New Spirits&categoryType=Spirits&fromURL=/webapp/wcs/stores/servlet/CatalogSearchResultView?storeId=10051&catalogId=10051&langId=-1&categoryId=1351370&variety=New+Spirits&categoryType=Spirits&top_category=25208&parent_category_rn=&sortBy=0&searchSource=E&pageView=&beginIndex=0">Woodford Reserve Master Collection Five Malt Stouted Mash</a>
所以我試圖抓取的資訊是：
Woodford Reserve Master Collection Five Malt Stouted Mash
和
/webapp/wcs/stores/servlet/ProductDisplay?catalogId=10051&storeId=10051&productId=3074457345616901168&langId=-1&partNumber=000086630prod&errorViewName=ProductDisplayErrorView&categoryId=1351370&top_category=25208&parent_category_rn=25208&urlLangId=&variety=New Spirits&categoryType=Spirits&fromURL=/webapp/wcs/stores/servlet/CatalogSearchResultView?storeId=10051&catalogId=10051&langId=-1&categoryId=1351370&variety=New+Spirits&categoryType=Spirits&top_category=25208&parent_category_rn=&sortBy=0&searchSource=E&pageView=&beginIndex=0
我正在嘗試對頁面上的每個專案進行迭代。我肯定會連接到該頁面，但由于某種原因，我無法使用for product in soup.select以下是我的腳本的簡化版本，我一直在嘗試從上面的 catalog_item_name 收集資訊
import requests
import sys
import time
import smtplib
from email.message import EmailMessage
import hashlib
from urllib.request import urlopen
from datetime import datetime
import json
import random
import requests
from itertools import cycle
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from urllib3.exceptions import InsecureRequestWarning
from requests_html import HTMLSession
session = HTMLSession()
user_agent_list = [
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
]
for i in range(1,4):
#Pick a random user agent
user_agent = random.choice(user_agent_list)
url = []
url = 'https://www.finewineandgoodspirits.com/webapp/wcs/stores/servlet/CatalogSearchResultView?storeId=10051&catalogId=10051&langId=-1&categoryId=1351370&variety=New Spirits&categoryType=Spirits&top_category=25208&sortBy=0&searchSource=E&pageView=&beginIndex=0#facet:&productBeginIndex:0&orderBy:&pageView:&minPrice:&maxPrice:&pageSize:&'
response = requests.get(url,headers=headers)
soup = BeautifulSoup(response.text,features="html.parser")
link = []
for product in soup.select('a.catalog_item_name'):
link.append(product)
print(link)
任何幫助將不勝感激！
編輯：用另外兩個網站測驗了腳本，它作業得很好。網站上一定有什么東西讓你失望了？
uj5u.com熱心網友回復：
我想這里最好的方法是檢查網路流量并直接查詢 API。例如，對于上面的 url，有一些POST針對 API 的請求https://www.finewineandgoodspirits.com/webapp/wcs/stores/servlet/CategoryProductsListingView。
我可以用它來獲取產品串列，即：
from bs4 import BeautifulSoup
import requests
import urllib
base_url = 'https://www.finewineandgoodspirits.com'
path = '/webapp/wcs/stores/servlet/CategoryProductsListingView?sType=SimpleSearch&resultsPerPage=15&sortBy=0&disableProductCompare=false&ajaxStoreImageDir=/wcsstore/WineandSpirits/&variety=New Spirits&categoryType=Spirits&ddkey=ProductListingView'
params = {
'storeId': '10051',
'categoryId': '1351370',
'searchType': '1002'
}
headers = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Content-Type': 'application/x-www-form-urlencoded',
'User-Agent': 'some super fancy browser',
}
request_url = base_url path '&' urllib.parse.urlencode(params)
response = requests.post(request_url, headers=headers)
soup = BeautifulSoup(response.text)
# now, extract the content form the soup, eg like you did above
product_links: list[str] = [base_url a['href'] for a in soup.select('a.catalog_item_name')]
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/478740.html