該網站有 9 頁，我的代碼只是將最后一頁元素添加到串列中。我想在串列中一起添加所有頁面的所有元素。
alltitles = []
allnames = []
alllinks = []
allpeices = []
allstocks = []
for n in range(pagenum):
pages_url = f"https://www.ispsupplies.com/manufacturers/TP~Link?order=relevance:asc&page=
{n 1}&keywords=tp-link"
driver.get(pages_url)
html = driver.page_source
soup = Soup(html)
title = soup.find_all("span", itemprop="name")
titleloop = [titles.text for titles in title]
alltitles.append(titleloop)
name = soup.find_all("div", class_="item-details-sku-container")
nameloop = [names.text for names in name]
allnames.append(nameloop)
link = soup.find_all("a", class_="facets-item-cell-grid-title")
linkloop = [links.text for links in link]
alllinks.append(linkloop)
price = soup.find_all("span", class_="item-views-price-lead")
priceloop = [prices.text for prices in price]
allpeices.append(priceloop)
stock = soup.find_all("div", class_="item-details-stock")
stockloop = [stocks.text for stocks in stock]
allstocks.append(stockloop)
uj5u.com熱心網友回復：
有什么理由不只是通過api嗎？效率更高，您將獲得更多資料。您始終可以過濾掉不需要的列。
import requests
import pandas as pd
items = []
page = 0
while True:
url = 'https://www.ispsupplies.com/api/items'
payload = {
'_t': '1641815468877',
'c': '393682',
'country': 'US',
'currency': 'USD',
'custitem_disable_from_main_website': '0',
'custitem_is_international': '0',
'fieldset': 'search',
'include': 'facets',
'language': 'en',
'limit': '100',
'manufacturers': 'TP~Link',
'n': '2',
'nocache': 'T',
'offset': str(page*100),
'sort': 'quantityavailable:desc'}
jsonData = requests.get(url, params=payload).json()
items = jsonData['items']
print('Page: %s' %(page 1))
if len(jsonData['items']) < 100:
break
page = 1
df = pd.DataFrame(items)
輸出：
完整輸出（僅 199 個產品的前 5 行）：
print(df.head(5).to_string())
custitem88 custitem89 custitem83 custitem_is_international custitem_open_box_ids custitem_ns_pr_item_attributes custitemnew ispurchasable custitem_ns_pr_attributes_rating stockdescription custitemclearance itemimages_detail custitem_commercecategory_brand custitemwarehousemessage custitem_incanada onlinecustomerprice_detail custitem71 weight custitem_ns_pr_rating_by_rate internalid itemoptions_detail outofstockmessage custitemextralargeimage2 custitem_availableus storedescription pricelevel1_formatted isinstock custitem67 custitem20 custitem21 onlinecustomerprice dontshowprice custitemrefurbished custitemonsale custitem68 manufacturer custitem69 custitemfree_shipping itemid custitemondiscount offersupport onlinecustomerprice_formatted nopricemessage custitem_disable_from_main_website pricelevel66_formatted isbackorderable custitemtariff_item custitemfree_shipping_cw custitem93 custitem94 custitem19 custitem18 custitem_st7 custitem_st6 showoutofstockmessage outofstockbehavior custitem_st8 itemtype quantityavailable custitem_st3 custitem_st2 custitem_st5 displayname storedisplayname2 custitem_st4 custitem_availableca pricelevel1 custitem_st1 custitem_gpon urlcomponent pricelevel66 custitem_commerce_category_1 custitem_commerce_category_3 custitem_commerce_category_2
0 0 False False True False {'5366': {'urls': [{'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-LINK-Gigabit-PCI-Express-Network-Adapter-TG-3468.5366-2.jpg'}]}} TP-Link 11/8/2021 False {'onlinecustomerprice_formatted': 'US$?14.99', 'onlinecustomerprice': 14.99} 0.50 5366 {'fields': [{'internalid': 'custcol19', 'label': 'Item Length', 'type': 'float'}, {'internalid': 'custcol20', 'label': 'Item Width', 'type': 'float'}, {'internalid': 'custcol21', 'label': 'Item Height', 'type': 'float'}, {'internalid': 'custcol_tariff_fee_option', 'label': 'Tariff Fee', 'type': 'currency'}, {'internalid': 'custcol_tariff_fee', 'label': 'Tariff Fee Custom', 'type': 'currency'}, {'internalid': 'custcol_is_tariff', 'label': 'Is Tariff', 'type': 'checkbox'}, {'internalid': 'custcol26', 'label': 'Purchase Price', 'type': 'currency'}, {'internalid': 'custcol36', 'label': 'Not Kit Component', 'type': 'checkbox'}, {'internalid': 'custcol67', 'label': 'Is Tariff (Webstore)', 'type': 'text'}, {'internalid': 'custcol_shiphawk_proposed_shipment_id', 'label': 'ShipHawk Proposed Shipment ID', 'type': 'text'}, {'internalid': 'custcol_shiphawk_source_system_line_n', 'label': 'ShipHawk Source System Line Number', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier', 'label': 'Carrier Name', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier_service', 'label': 'Carrier Service', 'type': 'text'}]} /core/media/media.nl?id=922920&c=393682&h=1qP1ijidIPW2P4DK3Fi_jlV_N3UT-StJuJYKXsZSuMSrOrIn 109 32-bit Gigabit PCIe Network Adapter US$?14.99 True 2.25 False 14.99 False False False TP-Link False True TG-3468 False False US$?14.99 False US$?14.99 True False False <div class="stock-detail-in ">In stock at College Station</div> 5.50 6.25 False - Default - InvtPart 109.0 TP-LINK 32-bit Gigabit PCIe Network Adapter 14.99 False TP-LINK-Gigabit-PCI-Express-Network-Adapter-TG-3468 14.99 PCI Adapters NaN NaN
1 0 False False True False {'urls': [{'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-LINK-TL-PA4010-KIT.01.jpg'}]} TP-Link 11/8/2021 False {'onlinecustomerprice_formatted': 'US$?39.99', 'onlinecustomerprice': 39.99} 1.00 5406 {'fields': [{'internalid': 'custcol19', 'label': 'Item Length', 'type': 'float'}, {'internalid': 'custcol20', 'label': 'Item Width', 'type': 'float'}, {'internalid': 'custcol21', 'label': 'Item Height', 'type': 'float'}, {'internalid': 'custcol_tariff_fee_option', 'label': 'Tariff Fee', 'type': 'currency'}, {'internalid': 'custcol_tariff_fee', 'label': 'Tariff Fee Custom', 'type': 'currency'}, {'internalid': 'custcol_is_tariff', 'label': 'Is Tariff', 'type': 'checkbox'}, {'internalid': 'custcol26', 'label': 'Purchase Price', 'type': 'currency'}, {'internalid': 'custcol36', 'label': 'Not Kit Component', 'type': 'checkbox'}, {'internalid': 'custcol67', 'label': 'Is Tariff (Webstore)', 'type': 'text'}, {'internalid': 'custcol_shiphawk_proposed_shipment_id', 'label': 'ShipHawk Proposed Shipment ID', 'type': 'text'}, {'internalid': 'custcol_shiphawk_source_system_line_n', 'label': 'ShipHawk Source System Line Number', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier', 'label': 'Carrier Name', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier_service', 'label': 'Carrier Service', 'type': 'text'}]} /core/media/media.nl?id=875835&c=393682&h=blNs8_wT0YD2isH8-8LHyXuDVz82k4V5VxMsQVeVrrUeVsAE 94 AV500 Nano Powerline Ethernet Adapter Starter Kit, Twin Pack US$?39.99 True 4.00 False 39.99 False False False TP-Link False True TL-PA4010 KIT False False US$?39.99 False US$?39.99 True False False <div class="stock-detail-in ">In stock at College Station</div> 6.00 8.00 False - Default - InvtPart 94.0 TP-LINK AV600 Powerline Starter Kit 39.99 False TP-LINK-TL-PA4010-KIT 39.99 Powerline Systems NaN NaN
2 0 False False True False {'urls': [{'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-UE300.01.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-UE300.02.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-UE300.03.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-UE300.04.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-UE300.05.jpg'}]} TP-Link 9/17/2021 False {'onlinecustomerprice_formatted': 'US$?12.99', 'onlinecustomerprice': 12.99} 0.25 20996 {'fields': [{'internalid': 'custcol19', 'label': 'Item Length', 'type': 'float'}, {'internalid': 'custcol20', 'label': 'Item Width', 'type': 'float'}, {'internalid': 'custcol21', 'label': 'Item Height', 'type': 'float'}, {'internalid': 'custcol_tariff_fee_option', 'label': 'Tariff Fee', 'type': 'currency'}, {'internalid': 'custcol_tariff_fee', 'label': 'Tariff Fee Custom', 'type': 'currency'}, {'internalid': 'custcol_is_tariff', 'label': 'Is Tariff', 'type': 'checkbox'}, {'internalid': 'custcol26', 'label': 'Purchase Price', 'type': 'currency'}, {'internalid': 'custcol36', 'label': 'Not Kit Component', 'type': 'checkbox'}, {'internalid': 'custcol67', 'label': 'Is Tariff (Webstore)', 'type': 'text'}, {'internalid': 'custcol_shiphawk_proposed_shipment_id', 'label': 'ShipHawk Proposed Shipment ID', 'type': 'text'}, {'internalid': 'custcol_shiphawk_source_system_line_n', 'label': 'ShipHawk Source System Line Number', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier', 'label': 'Carrier Name', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier_service', 'label': 'Carrier Service', 'type': 'text'}]} /core/media/media.nl?id=7189171&c=393682&h=qYPfPWXvWc_Udet9IChlyz96qbiA25Y-jMsjg8svIFm-WHxm 79 US$?12.99 True 0.67 False 12.99 False False False TP-Link False False UE300 False False US$?12.99 False US$?12.99 True False False <div class="stock-detail-in ">In stock at College Station</div> 3.35 6.10 False - Default - InvtPart 79.0 TP-Link USB 3.0 to Gigabit Ethernet Network Adapter 12.99 False TP-Link-UE300 12.99 USB Converters NaN NaN
3 0 False False True False {'urls': [{'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-LINK-2-4GHz-300Mbps-9dBi-Outdoor-CPE-CPE210.001.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-LINK-2-4GHz-300Mbps-9dBi-Outdoor-CPE-CPE210.002.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-LINK-2-4GHz-300Mbps-9dBi-Outdoor-CPE-CPE210.003.jpg'}]} TP-Link 9/22/2021 False {'onlinecustomerprice_formatted': 'US$?39.99', 'onlinecustomerprice': 39.99} 1.65 5319 {'fields': [{'internalid': 'custcol19', 'label': 'Item Length', 'type': 'float'}, {'internalid': 'custcol20', 'label': 'Item Width', 'type': 'float'}, {'internalid': 'custcol21', 'label': 'Item Height', 'type': 'float'}, {'internalid': 'custcol_tariff_fee_option', 'label': 'Tariff Fee', 'type': 'currency'}, {'internalid': 'custcol_tariff_fee', 'label': 'Tariff Fee Custom', 'type': 'currency'}, {'internalid': 'custcol_is_tariff', 'label': 'Is Tariff', 'type': 'checkbox'}, {'internalid': 'custcol26', 'label': 'Purchase Price', 'type': 'currency'}, {'internalid': 'custcol36', 'label': 'Not Kit Component', 'type': 'checkbox'}, {'internalid': 'custcol67', 'label': 'Is Tariff (Webstore)', 'type': 'text'}, {'internalid': 'custcol_shiphawk_proposed_shipment_id', 'label': 'ShipHawk Proposed Shipment ID', 'type': 'text'}, {'internalid': 'custcol_shiphawk_source_system_line_n', 'label': 'ShipHawk Source System Line Number', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier', 'label': 'Carrier Name', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier_service', 'label': 'Carrier Service', 'type': 'text'}]} /core/media/media.nl?id=875579&c=393682&h=skaSM39aCBHsxoAkbixkUtedRt2h7qw6xp6EXKWbFg9QUAGA 71 Outdoor 2.4GHz 300Mbps High power Wireless Access Point US$?39.99 True 4.10 False 39.99 False False False TP-Link False True CPE210 False False US$?39.99 False US$?39.99 True False False <div class="stock-detail-in ">In stock at College Station</div> 5.25 10.62 False - Default - InvtPart 71.0 TP-LINK 2.4GHz 300Mbps 9dBi Outdoor CPE CPE210 39.99 False TP-LINK-2-4GHz-300Mbps-9dBi-Outdoor-CPE-CPE210 39.99 2GHz PTP/PTMP NaN NaN
4 0 False False True False {'urls': [{'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-TL-WR902AC.011.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-TL-WR902AC.012.jpg'}, {'altimagetext': '', 'url': 'https://www.ispsupplies.com/SSP Applications/NetSuite Inc. - SCA Vinson/Development/product_images/images/TP-Link-TL-WR902AC.013.jpg'}]} TP-Link 11/29/2021 False {'onlinecustomerprice_formatted': 'US$?39.99', 'onlinecustomerprice': 39.99} 0.60 5512 {'fields': [{'internalid': 'custcol19', 'label': 'Item Length', 'type': 'float'}, {'internalid': 'custcol20', 'label': 'Item Width', 'type': 'float'}, {'internalid': 'custcol21', 'label': 'Item Height', 'type': 'float'}, {'internalid': 'custcol_tariff_fee_option', 'label': 'Tariff Fee', 'type': 'currency'}, {'internalid': 'custcol_tariff_fee', 'label': 'Tariff Fee Custom', 'type': 'currency'}, {'internalid': 'custcol_is_tariff', 'label': 'Is Tariff', 'type': 'checkbox'}, {'internalid': 'custcol26', 'label': 'Purchase Price', 'type': 'currency'}, {'internalid': 'custcol36', 'label': 'Not Kit Component', 'type': 'checkbox'}, {'internalid': 'custcol67', 'label': 'Is Tariff (Webstore)', 'type': 'text'}, {'internalid': 'custcol_shiphawk_proposed_shipment_id', 'label': 'ShipHawk Proposed Shipment ID', 'type': 'text'}, {'internalid': 'custcol_shiphawk_source_system_line_n', 'label': 'ShipHawk Source System Line Number', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier', 'label': 'Carrier Name', 'type': 'text'}, {'internalid': 'custcol_shiphawk_carrier_service', 'label': 'Carrier Service', 'type': 'text'}]} /core/media/media.nl?id=1056731&c=393682&h=che7-nic7o8Sln8Cl1UJWkH_DVUv7VRlcJi9_va_9WP4bFwv 60 AC750 Portable Wi-Fi Travel Router, 2.4/5GHz US$?39.99 True 3.00 False 39.99 False False False TP-Link False True TL-WR902AC False False US$?39.99 False US$?39.99 True False False <div class="stock-detail-in ">In stock at College Station</div> 4.50 4.50 False - Default - InvtPart 60.0 TP-Link AC750 Wireless Travel Router 2.4/5GHz 39.99 False TP-Link-TL-WR902AC 39.99 Wireless Routers NaN
或者只是在網站上看到的：
print(df[['storedisplayname2',
'itemid',
'urlcomponent',
'onlinecustomerprice_formatted',
'quantityavailable']].head(5).to_string())
storedisplayname2 itemid urlcomponent onlinecustomerprice_formatted quantityavailable
0 TP-LINK 32-bit Gigabit PCIe Network Adapter TG-3468 TP-LINK-Gigabit-PCI-Express-Network-Adapter-TG-3468 US$?14.99 109.0
1 TP-LINK AV600 Powerline Starter Kit TL-PA4010 KIT TP-LINK-TL-PA4010-KIT US$?39.99 94.0
2 TP-Link USB 3.0 to Gigabit Ethernet Network Adapter UE300 TP-Link-UE300 US$?12.99 79.0
3 TP-LINK 2.4GHz 300Mbps 9dBi Outdoor CPE CPE210 CPE210 TP-LINK-2-4GHz-300Mbps-9dBi-Outdoor-CPE-CPE210 US$?39.99 71.0
4 TP-Link AC750 Wireless Travel Router 2.4/5GHz TL-WR902AC TP-Link-TL-WR902AC US$?39.99 60.0
uj5u.com熱心網友回復：
怎么了？
代碼運行良好，但迭代速度很快，并且在您嘗試找到它們的那一刻，您正在尋找的元素并不存在。
怎么修？
使用selenium waits檢查元素存在于DOM：
...
driver.get(pages_url)
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '[data-type="item"]')))
html = driver.page_source
...
注意： 您必須進行額外的匯入
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
例子
不知道為什么決定使用這些串列，這個例子處理一個字典串列：
data = []
for n in range(2):
pages_url = f"https://www.ispsupplies.com/manufacturers/TP~Link?order=relevance:asc&page={n 1}&keywords=tp-link"
driver.get(pages_url)
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '[data-type="item"]')))
html = driver.page_source
soup = Soup(html)
for item in soup.select('[data-type="item"]'):
data.append({
'title' : item.find("span", itemprop="name").text,
'name' : item.find("div", class_="item-details-sku-container").text,
'link' : item.find("a", class_="facets-item-cell-grid-title")['href'],
'price' : item.find("span", class_="item-views-price-lead").text,
'stock' : item.find("div", class_="item-details-stock").text.strip()
})
pd.DataFrame(data)
輸出
| 標題 | 名稱 | 關聯 | 價格 | 股票 |
|---|---|---|---|---|
| TP-Link AC750 無線雙頻路由器 | SKU： 弓箭手 C20 | /TP-Link-Archer-C20 | US$ 34.99 | 直接發貨專案 專案通常直接從制造商發貨 |
| TP-Link 16 埠千兆非網管 Pro 交換機 | 貨號： TL-SG116E | /TP-Link-TL-SG116E | US$ 79.99 | 3 有貨 |
| TP-Link AC1200無線MU-MIMO千兆路由器Archer A6 | SKU： 弓箭手 A6_V3 | /TP-Link-Archer-A6 | US$ 49.99 | 直接發貨專案 專案通常直接從制造商發貨 |
| TP-Link AC4000 MU-MIMO 三頻 Wi-Fi 路由器 Archer A20 | SKU： 弓箭手 A20 | /TP-Link-Archer-A20 | US$ 189.99 | 直接發貨專案 專案通常直接從制造商發貨 |
| TP-Link AC5400 MU-MIMO 三頻游戲路由器 | SKU： 弓箭手 C5400X | /TP-Link-Archer-C5400X | US$ 279.99 | 直接發貨專案 專案通常直接從制造商發貨 |
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/409296.html
標籤：
下一篇：電報獲取訊息歷史