在Python中使用Selenium進行網頁抓取-有解無憂

我正在嘗試從 masari.io 抓取 DAO 串列，但我遇到了問題，因為我收到以下錯誤：

DeprecationWarning: executable_path has been deprecated, please pass in a Service object


driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)

DevTools listening on ws://127.0.0.1:56691/devtools/browser/b4609671-5e6e-4d25-b09e-4116b3dde4bf
[0525/100030.252:INFO:CONSOLE(1)] "enabling sentry error tracker", source: https://messari.io/static/js/main.977a4794.chunk.js (1)
[0525/100030.951:INFO:CONSOLE(2)] "Unable to refresh token: Login required", source: https://messari.io/static/js/23.778d04d0.chunk.js (2)
[0525/100031.065:INFO:CONSOLE(2)] "


88b           d88                                                            88
888b         d888                                                            ""
88'8b       d8'88
88 '8b     d8' 88   ,adPPYba,  ,adPPYba,  ,adPPYba,  ,adPPYYba,  8b,dPPYba,  88
88  '8b   d8'  88  a8P_____88  I8[    ""  I8[    ""  ""     'Y8  88P'   "Y8  88
88   '8b d8'   88  8PP"""""""   '"Y8ba,    '"Y8ba,   ,adPPPPP88  88          88
88    '888'    88  "8b,   ,aa  aa    ]8I  aa    ]8I  88,    ,88  88          88
88     '8'     88   '"Ybbd8"'  '"YbbdP"'  '"YbbdP"'  '"8bbdP"Y8  88          88


", source: https://messari.io/static/js/23.778d04d0.chunk.js (2)
[0525/100031.069:INFO:CONSOLE(2)] "Interested in a CHALLENGE? Check out: https://messari.io/quiz", source: https://messari.io/static/js/23.778d04d0.chunk.js (2)
Traceback (most recent call last):
  File "c:/Users/Student/webScrape/scraper.py", line 21, in <module>
    matches = WebDriverWait(driver, 10).until(
  File "C:\Users\Student\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\support\wait.py", line 89, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
Backtrace:
        Ordinal0 [0x0096B8F3 2406643]
        Ordinal0 [0x008FAF31 1945393]
        Ordinal0 [0x007EC748 837448]
        Ordinal0 [0x008192E0 1020640]
        Ordinal0 [0x0081957B 1021307]
        Ordinal0 [0x00846372 1205106]
        Ordinal0 [0x008342C4 1131204]
        Ordinal0 [0x00844682 1197698]
        Ordinal0 [0x00834096 1130646]
        Ordinal0 [0x0080E636 976438]
        Ordinal0 [0x0080F546 980294]
        GetHandleVerifier [0x00BD9612 2498066]
        GetHandleVerifier [0x00BCC920 2445600]
        GetHandleVerifier [0x00A04F2A 579370]
        GetHandleVerifier [0x00A03D36 574774]
        Ordinal0 [0x00901C0B 1973259]
        Ordinal0 [0x00906688 1992328]
        Ordinal0 [0x00906775 1992565]
        Ordinal0 [0x0090F8D1 2029777]
        BaseThreadInitThunk [0x777BFA29 25]
        RtlGetAppContainerNamedObjectPath [0x77B77A7E 286]
        RtlGetAppContainerNamedObjectPath [0x77B77A4E 238]

我知道 messari.io 有一個 API，但我幾乎可以肯定它只適用于他們的資產，而不是他們的 DAO 串列。我嘗試使用 Selenium，因為它是一個動態頁面，但我仍然遇到問題。這是我的代碼：

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import requests

url = 'https://messari.io/governor/daos'

DRIVER_PATH = 'PATH_TO_DRIVER_ON_MY_PC'
options = Options()
options.headless = True
options.add_argument("--window-size=1920, 1200")

# s = Service('PATH_TO_DRIVER_ON_MY_PC')
driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
driver.get('https://messari.io/governor/daos')

try:
    matches = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "td")))
    # for match in matches:
    #     print(match.text)

finally:
    driver.quit()

更新我修復了 executable_path 警告，但我仍然收到相同的 TimeoutException 錯誤。當我在沒有無頭的情況下運行它時，我還會收到以下訊息：

DevTools listening on ws://127.0.0.1:57773/devtools/browser/4450b78d-3a9f-401a-b39c-2c716ecad924
[9628:20616:0525/102300.840:ERROR:device_event_log_impl.cc(214)] [10:23:00.840] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[9628:20616:0525/102300.841:ERROR:device_event_log_impl.cc(214)] [10:23:00.841] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

我認為這部分更像是一條硬體訊息，基于類似的問題，我不應該擔心，因為當我拔下滑鼠時，它洗掉了其中一個。

uj5u.com熱心網友回復：

此頁面不<td>用于顯示 DAO 串列。
它使用<div>(with CSS) 來顯示它，類似于表格。

它保留了 DAO 的名稱<h4>

至少它在我的帶有 Linux 的筆記本電腦上的 Firefox 中使用。

完整的作業代碼（在 Linux Mint、Python 3.8、Selenium 4.x、Chrome 101.x 上測驗）

我使用了模塊webdriver_manager，因此當 Linux 安裝較新版本的 Chrome 時，它??會自動下載新的驅動程式

我必須使用find_elements()(with sin word elements) 或presence_of_all_elements_located()獲取所有<h4>.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from webdriver_manager.chrome import ChromeDriverManager

url = 'https://messari.io/governor/daos'

options = Options()
options.headless = True
options.add_argument("--window-size=1920, 1200")

driver = webdriver.Chrome(options=options, service=Service(ChromeDriverManager().install()))

driver.get('https://messari.io/governor/daos')

try:
    matches = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.TAG_NAME, "h4")))
    
    #matches = driver.find_elements(By.TAG_NAME, "h4")
    
    for match in matches:
        if match.text:
            print(match.text)
finally:
    driver.quit()

結果：

Fei
Rook
Cosmos
Stargate Finance
Aave
Treasure DAO
DODO
Radicle
Goldfinch
Merit Circle
EPNS
Perpetual Protocol
Gitcoin
SuperRare
Indexed
Doodles
Rome DAO
Badger
Paraswap
Unlock
Terra
Shapeshift
Lobis
Pool Together
The Graph
Yearn Finance
Ampleforth
Alpaca Finance
Balancer
Gro Protocol
Sismo DAO
BeethovenX
ENS
Lido
Alchemist

編輯：

要獲取所有值，您可能需要滾動頁面 - JavaScript 將添加新專案。

有使用while-loop的答案，execute_script()使用 JavaScript 代碼滾動到底部并獲取當前高度。如果高度與滾動前不同，則您必須再次滾動，但如果高度相同，則您有頁尾，現在您可以獲取所有專案。

uj5u.com熱心網友回復：

由于不推薦使用selenium4密鑰executable_path，您必須使用Service()該類的實體以及ChromeDriverManager().install()命令，如下所述

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get("https://www.google.com")

轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/482294.html

標籤：Python html 硒网页抓取动态的

上一篇：無法將“OpenQA.Selenium.Chrome.ChromeDriver”型別的物件轉換為“OpenQA.Selenium.Remote.RemoteWebDriver”型別

下一篇：如何在已打開的瀏覽器上執行selenium腳本