我正在從網站https://www.scmp.com/上抓取新聞文章 雖然我可以從每篇文章中獲取標題或作者姓名，但我無法獲取文章的正文或主要內容。我遵循了兩種方法，但都沒有用。
第一種方法
options = webdriver.ChromeOptions()
lists = ['disable-popup-blocking']
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "normal"
driver.get('https://www.scmp.com/news/asia/east-asia/article/3199400/japan-asean-hold-summit-tokyo-around-december-2023-japanese-official')
driver.implicitly_wait(5)
bsObj = BeautifulSoup(driver.page_source, 'html.parser')
text_res = bsObj.select('div[]')
text = ""
for item in text_res:
if item.get_text() == "":
continue
text = text item.get_text().strip() "\n"
第二種方法
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(executable_path= r"E:\chromedriver\chromedriver.exe", options=options) #add your chrome path
driver.get('https://www.scmp.com/news/asia/east-asia/article/3199400/japan-asean-hold-summit-tokyo-around-december-2023-japanese-official')
driver.implicitly_wait(5)
a = driver.find_element_by_class_name("details__body body").text
print(a)
請幫我解決一下這個。謝謝你。
uj5u.com熱心網友回復：
您無法從南華早報上的文章中獲取文本的原因有多種。
首先，當您使用文章的 URL 打開 Chrome 時selenium，會顯示 GDRP 通知。
必須通過單擊按鈕來接受 GDRP。
其次，該頁面還顯示一個彈出視窗來設定您的新聞偏好。
新聞首選項彈出視窗必須X退出。
第三次嘗試提取文本使用selenium將需要一些資料清理。我建議使用BeautifulSoup從頁面上的腳本標記中提取干凈的文章文本。
這是一些粗略的代碼，可以單擊 GDRP 按鈕，X彈出新聞首選項并提取文章文本。
可以改進此代碼以滿足您的需要。
import json
from time import sleep
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
capabilities = DesiredCapabilities().CHROME
chrome_options = Options()
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
chrome_options.add_argument("--ignore-certificate-errors")
# disable the banner "Chrome is being controlled by automated test software"
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)
url_main = 'https://www.scmp.com/news/asia/east-asia/article/3199400/japan-asean-hold-summit-tokyo-around-december-2023-japanese-official'
driver.get(url_main)
driver.implicitly_wait(20)
element_has_bottom_message = WebDriverWait(driver, 120).until(EC.presence_of_element_located((By.CLASS_NAME, "has-bottom-messaging")))
if element_has_bottom_message:
element_gdpr = WebDriverWait(driver, 120).until(
EC.presence_of_element_located((By.CLASS_NAME, "gdpr-banner__accept")))
if element_gdpr:
gdrp_button = driver.find_element_by_xpath("//*[@class='gdpr-banner__accept']")
driver.implicitly_wait(20)
ActionChains(driver).move_to_element(gdrp_button).click(gdrp_button).perform()
element_my_news_popup = WebDriverWait(driver, 120).until(
EC.presence_of_element_located((By.CLASS_NAME, "my-news-landing-popup__icon-close")))
if element_my_news_popup:
my_news_popup = driver.find_element_by_xpath("//*[@class='my-news-landing-popup__icon-close']")
ActionChains(driver).move_to_element(my_news_popup).click(my_news_popup).perform()
driver.implicitly_wait(20)
raw_soup = BeautifulSoup(driver.page_source, 'lxml')
json_dictionaries = raw_soup.find_all(name='script', attrs={'type': 'application/ld json'})
if len(json_dictionaries) != 0:
for json_dictionary in json_dictionaries:
dictionary = json.loads("".join(json_dictionary.contents), strict=False)
article_bool = bool([value for (key, value) in dictionary.items() if key == 'articleBody'])
if article_bool:
for key, value in dictionary.items():
if key == 'articleBody':
print(value)
sleep(30)
driver.close()
driver.quit()
輸出
The leaders of Japan and 10-member Asean on Saturday agreed to hold a summit in Tokyo
in or around December next year to commemorate the 50th anniversary of their relationship,
a Japanese official said. Japanese Prime Minister Fumio Kishida and his counterparts from
the Association of Southeast Asian Nations also pledged to deepen their cooperative ties
when they met in Phnom Penh, according to the official. Japan has been trying to boost
relations with Asean at a time when some of its members are increasingly vigilant against
China ’s assertive territorial claims in the East and South China seas . Why is Japan
losing ground in Asean despite being a bigger investor than China? “Although concerns are
growing over opaque and unfair development support, Japan will continue to back sustainable
growth” of Southeast Asia , Kishida said at the outset of the meeting, which was open to
the media, in a veiled reference to Beijing’s trade and economic practices. Leaders of
several nations mentioned the importance of freedom of navigation and overflight in the
South China Sea, and of the necessity of adhering to international law, the official said
after the meeting. The agreement on the special summit in Tokyo came as the US and China
have been intensifying their competition for influence in Southeast Asia. In November last
year, China and Asean agreed to upgrade their ties to a “comprehensive strategic
partnership” when the two sides held a special online summit commemorating the 30th
anniversary of their dialogue, with Chinese President Xi Jinping making a rare appearance.
China has stepped up efforts to expand its clout in the region as security tensions
with the US escalate in nearby waters. After China’s move, the US in May declared with
Asean that they had decided to elevate their relationship to a “comprehensive strategic
partnership” as well. At the Asean-Japan gathering, Kishida also reiterated his support
for the “Asean Outlook on the Indo-Pacific”, an initiative aimed at maintaining peace,
freedom and prosperity in the region, the official said.
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/535481.html