我正在嘗試使用 Python 從網站上抓取一張表格，但由于某種原因，我所有已知的方法都失敗了。https://www.nbc4i.com/news/state-news/535-new-cases-of-covid-19-reported-in-ohio-schools-in-past-week/有一張表格， 有 45 頁。我嘗試使用：requests、requests-html（渲染它）、BeautifulSoup 和 selenium 來抓取它。這是我的代碼之一，我不會在這里復制我嘗試過的所有代碼，方法相似，只是使用不同的 Python 庫：
from requests_html import HTMLSession
from bs4 import BeautifulSoup
session = HTMLSession()
page = session.get('https://www.nbc4i.com/news/state-news/535-new-cases-of-covid-19-reported-in-ohio-schools-in-past-week/')
page.html.render(timeout=120)
soup = BeautifulSoup(page.content, 'lxml') #also tried with page.text and 'html.parser' and all permutations
table = soup.find_all(id='table')
我的表變數在這里是一個空串列，它不應該是。我試圖用 selenium 在表格中找到任何其他 web 元素，我也試圖通過類、xpath 找到，但所有這些都未能找到表格或其任何部分。我用這些方法抓取了相當多的類似網站，在此之前我從未遇到過問題。有什么想法嗎？
uj5u.com熱心網友回復：
您會看到結果表位于 iframe 中。您可以直接從 iframe 的源中提取資訊：
https://flo.uri.sh/visualisation/3894531/embed?auto=1
這里的代碼應該將結果保存到 .csv 檔案中：
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
def get_rows(driver):
"""
returns rows from a page
Returns:
Dict
"""
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//div[@class='tr body-row']")))
rows = driver.find_elements(By.XPATH, "//div[@class='tr body-row']")
table_info= {
'Rank': [],
'County':[],
'School/District':[],
'Type':[],
'Total cases':[],
'Student cases':[],
'Staff cases':[]
}
for row in rows:
cols = row.find_elements(By.CLASS_NAME, 'td')
for col, index in enumerate(table_info):
table_info[index].append(cols[col].text)
return table_info
# path to chrome driver
driver = webdriver.Chrome("D:\chromedriver\94\chromedriver.exe")
driver.get("https://flo.uri.sh/visualisation/3894531/embed?auto=1")
df = pd.DataFrame.from_dict(get_rows(driver))
for _ in range(44):
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//button[@]'))).click()
df = pd.concat([df, pd.DataFrame.from_dict(get_rows(driver))])
print(df)
df.to_csv('COVID-19_cases_reported_in_Ohio_schools.csv', index=False)
uj5u.com熱心網友回復：
表格內容在iframe中，需要切換到iframe頁面。請參閱API 檔案。
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
url = 'https://www.nbc4i.com/news/state-news/535-new-cases-of-covid-19-reported-in-ohio-schools-in-past-week/'
s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)
try:
driver.get(url)
driver.implicitly_wait(5)
driver.switch_to.frame(driver.find_element(By.XPATH,
'//div[@]/div/iframe'))
# table content is now in the driver context
while True:
table = driver.find_element(By.ID, "table")
for elt in table.find_elements(By.CLASS_NAME, "body-row"):
items = [td.text for td in elt.find_elements(By.CLASS_NAME, "td")]
# add code to append each of row of data to CSV file, database, etc.
print(items)
next_btn = driver.find_element(By.CLASS_NAME, 'next')
if 'disabled' in next_btn.get_attribute('class'):
# no more > done with pagination
break
next_btn.click() # click next button for next set of items
finally:
driver.quit()
輸出：
['1', 'Delaware', 'Olentangy Local', 'Public District', '38', '31', '7']
...
['446', 'Muskingum', 'West Muskingum Local', 'Public District', '1', '1', '0']
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/351357.html
標籤：Python 硒 网页抓取 美汤 python-请求-html