使用SeleniumWebDriver報廢動態表等待回傳截斷的資料幀

我嘗試從https://www.ishares.com/us/products/268752/ishares-global-reit-etf中洗掉一個名為“holding”的動態表

起初我使用 selenium，但我得到了空的 DataFrame。然后這里的社區幫助建議我誘導“WebDriverWait”在提取資料之前完全加載資料。它可以作業，但我得到的資料被從 400 行截斷到只有 10 行。我怎樣才能得到我需要的所有資料。任何人都可以幫助我。謝謝你。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

# Instantiate options
options = webdriver.ChromeOptions()
options.headless = True

# Instantiate a webdriver
site = 'https://www.ishares.com/us/products/268752/ishares-global-reit-etf'
wd = webdriver.Chrome('chromedriver', options=options)
wd.get(site)

# Induce WebDriver Wait
WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
wd.execute_script("arguments[0].scrollIntoView();", WebDriverWait(wd, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-componentname]/h2[normalize-space()='Holdings']"))))
data = WebDriverWait(wd, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@aria-describedby='allHoldingsTable_info']"))).get_attribute("outerHTML")
data2  = pd.read_html(data)
holding = data2[0]

uj5u.com熱心網友回復：

您撰寫的代碼還可以，但是您錯過了一點。默認情況下，該表采用分頁設計，每頁僅顯示 10 條記錄，因此您只檢索了這些記錄。您必須添加一個額外的操作步驟（單擊“顯示更多”按鈕），它將顯示所有記錄，因此您的 df 將擁有所有記錄。這是重構后的代碼：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
import pandas as pd
import time

# Instantiate options
opt = Options()
opt.add_argument("headless")
opt.add_argument("disable-gpu")
opt.add_argument("window-size=1920,1080")

# Instantiate a webdriver
site = 'https://www.ishares.com/us/products/268752/ishares-global-reit-etf'
wd = webdriver.Chrome('chromedriver', options=opt)
wd.maximize_window()
wd.get(site)

# Induce WebDriver Wait
WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
wd.execute_script("arguments[0].scrollIntoView();", WebDriverWait(wd, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-componentname]/h2[normalize-space()='Holdings']"))))
WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, "(//*[@class='datatables-utilities ui-helper-clearfix']//*[text()='Show More'])[2]"))).click()
data = WebDriverWait(wd, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@aria-describedby='allHoldingsTable_info']"))).get_attribute("outerHTML")
data2  = pd.read_html(data)
holding = data2[0]
print(holding)

輸出：

    Ticker                           Name  ...    SEDOL Accrual Date
0      PLD              PROLOGIS REIT INC  ...  B44WZD7            -
1     EQIX               EQUINIX REIT INC  ...  BVLZX12            -
2      PSA            PUBLIC STORAGE REIT  ...  2852533            -
3      SPG  SIMON PROPERTY GROUP REIT INC  ...  2812452            -
4      DLR  DIGITAL REALTY TRUST REIT INC  ...  B03GQS4            -
..     ...                            ...  ...      ...          ...
379    MYR                        MYR/USD  ...        -            -
380    MYR                        MYR/USD  ...        -            -
381    MYR                        MYR/USD  ...        -            -
382    MYR                        MYR/USD  ...        -            -
383    MYR                        MYR/USD  ...        -            -

[384 rows x 12 columns]

Process finished with exit code 0

標籤：

上一篇：Selenium看不到可點擊的按鈕

下一篇：當使用python的標頭中缺少內容處置時，如何從URL獲取檔案名？