僅下載和提取來自BBC的新聞

我需要使用 Python 中的 Selenium 從 BBC 主頁獲取所有文章。在瀏覽了網站 HTML 之后，我能夠提取整個頁面的部分。問題是我試圖過濾不相關的部分，例如語言更改并跳過以連接型別的 url。

我的目標是輸入每篇文章并僅使用 Selenium 獲取其內容。

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.common.by import By

PATH = "C:/Program Files (x86)/chromedriver.exe"
driver = webdriver.Chrome(PATH)
url = 'https://www.bbc.com/'

driver.get(url)


hrefs = []

# Getting all sections from BBC
media_list = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.TAG_NAME, "section"))
)

for item in media_list:
    print(item.find_element_by_css_selector('trying to figure what to write here'))

driver.close()
driver.quit()

uj5u.com熱心網友回復：

如果您想輸入每篇文章，抓取資料，回傳并轉到下一篇文章，您的代碼將不像您在此處介紹的那么簡單。
每次回傳主頁時，您都需要再次獲取所有文章元素。
在單擊之前，您需要將所需的文章元素滾動到視圖中。
BTWpresence_of_all_elements_located將等待至少 1 個元素的出現，而不是您想象的與傳遞的定位器匹配的所有元素。所以最好使用visibility_of_element_located- 這將等待元素更成熟的狀態，然后添加一些額外的延遲以使所有其他元素加載。1秒在這里綽綽有余。
此外，您的定位器是錯誤的。
這應該有效：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains

PATH = "C:/Program Files (x86)/chromedriver.exe"
driver = webdriver.Chrome(PATH)
url = 'https://www.bbc.com/'

driver.get(url)
actions = ActionChains(driver)

hrefs = []

# Getting all sections from BBC
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.block-link__overlay-link")))
time.sleep(1)
media_list = driver.find_elements_by_css_selector("a.block-link__overlay-link")
for idx, val in enumerate(media_list):
    WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.block-link__overlay-link")))
    time.sleep(1)
    media_list = driver.find_elements_by_css_selector("a.block-link__overlay-link")
    item = media_list[idx]
    actions.move_to_element(item).perform()
    time.sleep(0.5)
    item.click()
    #scrape your data
    driver.execute_script("window.history.go(-1)")

uj5u.com熱心網友回復：

我提供了沒有 Selenium 的示例，但我希望你能得到它。第一個目標 - 從主頁獲取新聞的每個鏈接

def get_links():
    LINKS = []
    url = 'https://www.bbc.com'
    response = requests.get(url)
    soup = BeautifulSoup(response.text, features='lxml')
    for html in soup.find_all('li', class_='media-list__item'):
        link = html.find('a').get('href')
        if "https://" in link:
            LINKS.append(link)
        else:
            link = url   link
            LINKS.append(link)
    return LINKS

輸出：

    https://www.bbc.com/news/world-asia-60034170
    https://www.bbc.com/news/world-australia-60027360
    https://www.bbc.com/news/world-europe-60030615
    https://www.bbc.com/worklife/article/20220114-can-sleep-leadership-help-banish-burnout
    https://www.bbc.com/culture/article/20220117-what-happens-to-fascist-architecture-after-fascism
    https://www.bbc.com/news/world-latin-america-59944126
    https://www.bbc.com/news/business-60036831
    https://www.bbc.com/news/uk-60033012
    https://www.bbc.com/sport/live/tennis/58856643
    https://www.bbc.com/sport/live/football/60036185
    ...
    https://www.bbc.com/news/world-africa-59703123

現在我們有了主頁上的每一個href，讓我們把文章的頭部和正文去掉：

def get_page(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, features='lxml')
    heading = soup.find('h1', {"id": "main-heading"}).getText()
    print(heading) #or write it to file
    for block in soup.find_all('div', {"data-component": "text-block"}):
        print(block.getText()) #or write it to file

最后一個目標我們只需要新聞鏈接，就這么簡單：

for link in get_links():
    if "/news/" in link:
        get_page(link)        

完整代碼：

import requests
from bs4 import BeautifulSoup


def get_links():
    LINKS = []
    url = 'https://www.bbc.com'
    response = requests.get(url)
    soup = BeautifulSoup(response.text, features='lxml')
    for html in soup.find_all('li', class_='media-list__item'):
        link = html.find('a').get('href')
        if "https://" in link:
            LINKS.append(link)
        else:
            link = url   link
            LINKS.append(link)
    return LINKS


def get_page(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, features='lxml')
    heading = soup.find('h1', {"id": "main-heading"}).getText()
    print(heading)
    for block in soup.find_all('div', {"data-component": "text-block"}):
        print(block.getText())

for link in get_links():
    if "/news/" in link:
        get_page(link)

標籤：

上一篇：Seleniumelement.text()不被視為字串

下一篇：找不到表硒python