使用selenium抓取資料

該程式運行良好，但他們只會刮one TITLE我希望他們刮掉頁面中的所有標題這些是頁面鏈接https://www.eurobike.com/en/index-exhibitors/exhibitors/？

import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager

options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1920x1080")
options.add_argument("--disable-extensions")

chrome_driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=options
)

def supplyvan_scraper():
    with chrome_driver as driver:
        driver.implicitly_wait(15)
        URL = 'https://www.eurobike.com/en/index-exhibitors/exhibitors/?'
        driver.get(URL)
        time.sleep(3)

        # opt #1 visit first link, print the title uncomment to see
        # click the single link
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.card-exhibitor"))).click()
        time.sleep(2)
        
        # parse the h1 tag text
        title = driver.find_element(By.CSS_SELECTOR, 'h1.underlined').text
        print(title)

        driver.quit()


supplyvan_scraper()

uj5u.com熱心網友回復：

該網站完全由復雜的 JavaScript 填充。首先，要顯示來自此 url 的串列，必須接受 cookie，但接受并單擊 cookie 按鈕并非易事，因為 cookie 按鈕位于shadow root (open)selenium 和 webdriverWait 下對影子根不做任何事情，因此要執行影子根，您需要應用JavaScript querySelector.

完整的作業代碼：

import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup


options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
#chrome to stay open to see what's happening in the real word or make it comment to close
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)   
URL ='https://www.eurobike.com/en/index-exhibitors/exhibitors/?'
driver.get(URL)
time.sleep(5)
#To execute shadow root and accept cookies
driver.execute_script('''return document.querySelector('div#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')''').click()
#Grabbing all listing url and iterate,append and new deriver request
links=[]
for card in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.list__results > div > div > a'))):

    link=card.get_attribute('href')
    links.append(link)
for u in links:
    driver.get(u)
    time.sleep(5)
    #extracting desired data using bs4 to  avoid much uses  of selenium because of it's complexity and time killing
    soup = BeautifulSoup(driver.page_source,'lxml')
    title=soup.select_one('h1.underlined').get_text(strip=True)
    print(title)

輸出：

 ANGLE is
A&C Solutions
A&J International Co.,Ltd
                        (Taiwan Branch)
A-Pro Tech Co., LTD
A-Rim Ent. Co., Ltd.
Abbey Bike Tools
ABIMOTA
                        Associacao Nacional das Industrias 
                        de Duas Rodas, Ferragens, Mobiliári
ABIMOTA
                        Associacao Nacional das Industrias 
                        de Duas Rodas, Ferragens, Mobiliári
ABUS |August Bremicker S?hne KG
ABUS |August Bremicker S?hne KG
Accelerated Systems Inc. (ASI)
ACCORD ENTERPRISE CORP.
Acer Gadget Inc.
Acetrikes Industrial Co., Ltd.
ACT LAB LLC
ACTIA
Action Sports SRL
Activent 365 s.r.o.
ADAC e.V.
ADD-ONE
AddBike
AddRE-Mo
                        (Electric Bike Solutions GmbH)
ADFC e. V.
Adhestick Innovations Ltd. (Joe's No Flats)
ADViTEX GMBH
?ike
AER Electric Company Ltd
                        King Edward House
Aero Sensor Ltd
Aeroe Limited
Aforge Enterprise Co., Ltd
Agentura REPRO spol. s r.o.

uj5u.com熱心網友回復：

試試這個，將“find_element”更改為“find_elements”，這樣它就會得到所有串列：

titles = driver.find_elements(By.CSS_SELECTOR, 'h1.underlined')
for title in titles:
    print(title.text)

標籤：Python 硒 网页抓取

上一篇：以這些格式從json中提取資料

下一篇：使用R進行Web-Scraping-我想從網站中提取一些類似資料的表格