我正在嘗試在此網站https://www.ercot.com/mp/data-products/data-product-details?id=NP3-233-CD上抓取所有表格資料 reportTable （該網站僅在美國有效）使用美麗的湯，它只回傳表頭和空身。
我嘗試了以下代碼：
url ='https://www.ercot.com/mp/data-products/data-product-details?id=NP3-233-CD'
usock = Request(url,headers={'User-Agent': 'Mozilla/5.0'})
data = urlopen(usock).read()
soup = BS(data,"html.parser")
report_table=soup.find_all(id='reportTable')
我也嘗試使用 selenium 并像這樣在 chrome 中打開 url 但仍然是一個空的主體
chrome.get(url)
data = chrome.page_source
soup = BS(data,"html.parser")
report_table=soup.find_all(id='reportTable')
有沒有人有解決方法？
謝謝，R
uj5u.com熱心網友回復：
該表中的資料正在通過對 API 的 XHR 呼叫進行補充。您可以直接抓取該 API：
import requests
import pandas as pd
r = requests.get('https://www.ercot.com/misapp/servlets/IceDocListJsonWS?reportTypeId=13103&_=1665672400202')
df = pd.json_normalize(r.json(), record_path=['ListDocsByRptTypeRes', 'DocumentList'])
df['DocUrl'] = df.apply(lambda row: 'https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=' str(row['Document.DocID']), axis=1)
display(df)
另外，如果您愿意，我還提供了下載實際檔案的 url。
結果在終端：
Document.ExpiredDate Document.ILMStatus Document.SecurityStatus Document.ContentSize Document.Extension Document.ReportTypeID Document.Prefix Document.FriendlyName Document.ConstructedName Document.DocID Document.PublishDate Document.ReportName Document.DUNS Document.DocCount DocUrl
0 2022-11-13T23:59:59-06:00 EXT P 1335 zip 13103 cdr HRLYRESOUTCAPNP3233_csv cdr.00013103.0000000000000000.20221013.090214639.HRLYRESOUTCAPNP3233_csv.zip 870085714 2022-10-13T09:02:14-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=870085714
1 2022-11-13T23:59:59-06:00 EXT P 2444 zip 13103 cdr HRLYRESOUTCAPNP3233_xml cdr.00013103.0000000000000000.20221013.090214585.HRLYRESOUTCAPNP3233_xml.zip 870085713 2022-10-13T09:02:14-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=870085713
2 2022-11-13T23:59:59-06:00 EXT P 1337 zip 13103 cdr HRLYRESOUTCAPNP3233_csv cdr.00013103.0000000000000000.20221013.080215293.HRLYRESOUTCAPNP3233_csv.zip 870074851 2022-10-13T08:02:15-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=870074851
3 2022-11-13T23:59:59-06:00 EXT P 2455 zip 13103 cdr HRLYRESOUTCAPNP3233_xml cdr.00013103.0000000000000000.20221013.080215212.HRLYRESOUTCAPNP3233_xml.zip 870074731 2022-10-13T08:02:15-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=870074731
4 2022-11-13T23:59:59-06:00 EXT P 1331 zip 13103 cdr HRLYRESOUTCAPNP3233_csv cdr.00013103.0000000000000000.20221013.070239337.HRLYRESOUTCAPNP3233_csv.zip 870066327 2022-10-13T07:02:39-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=870066327
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
1503 2022-10-13T23:59:59-05:00 EXT P 2334 zip 13103 cdr HRLYRESOUTCAPNP3233_xml cdr.00013103.0000000000000000.20220912.020252183.HRLYRESOUTCAPNP3233_xml.zip 863677980 2022-09-12T02:02:52-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=863677980
1504 2022-10-13T23:59:59-05:00 EXT P 1187 zip 13103 cdr HRLYRESOUTCAPNP3233_csv cdr.00013103.0000000000000000.20220912.010206931.HRLYRESOUTCAPNP3233_csv.zip 863670249 2022-09-12T01:02:06-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=863670249
1505 2022-10-13T23:59:59-05:00 EXT P 2334 zip 13103 cdr HRLYRESOUTCAPNP3233_xml cdr.00013103.0000000000000000.20220912.010206880.HRLYRESOUTCAPNP3233_xml.zip 863670162 2022-09-12T01:02:06-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=863670162
1506 2022-10-13T23:59:59-05:00 EXT P 1181 zip 13103 cdr HRLYRESOUTCAPNP3233_csv cdr.00013103.0000000000000000.20220912.000148740.HRLYRESOUTCAPNP3233_csv.zip 863662192 2022-09-12T00:01:48-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=863662192
1507 2022-10-13T23:59:59-05:00 EXT P 2332 zip 13103 cdr HRLYRESOUTCAPNP3233_xml cdr.00013103.0000000000000000.20220912.000148691.HRLYRESOUTCAPNP3233_xml.zip 863662191 2022-09-12T00:01:48-05:00 Hourly Resource Outage Capacity 0000000000000000 0 https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=863662191
1508 rows × 15 columns
uj5u.com熱心網友回復：
問題是表是從java腳本動態加載的。這是帶有硒的版本：
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
chrome = webdriver.Chrome()
chrome.get(url)
time.sleep(5) # wait some time to load, you can also use WebDriverWait().until or something
# in the selenium you don't have to use bs to parse data, it's just
report_table = chrome.find_element(By.ID, "reportTable")
print(report_table)
沒有測驗過這個，但它應該可以作業。
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/515487.html