為什么在瀏覽器開發工具與BeautifulSoap/Postman中獲取不同的資料？

我想從這個網頁上洗掉資料

但是使用 Beautifulsoap 我得到

<div class="results">
</div>

在郵遞員中得到同樣的東西..

這就是我正在做的方式..

topicuri = "https://search.donanimhaber.com/portal?q=vodafone&p=3&devicetype=browsermobile&order=date_desc&in=all&contenttype=all&wordtype=both&range=all"
r = s.get(topicuri)
soup = BeautifulSoup(r.text, 'html.parser')
pages = soup.find('div', {'class': 'results'})
print(pages)

uj5u.com熱心網友回復：

該網站使用 Javascript 來顯示這些片段。BeautifulSoup 不執行 Javascript，而瀏覽器執行。您可能希望在 Python 中使用 Chromium 引擎來抓取基于 Javascript 的內容。

uj5u.com熱心網友回復：

您還可以從 api 呼叫 json 回應中獲取資料

import requests
import json 
body= "vodafone"
headers= {
    'content-type': 'application/json'
   }

api_url = "https://search.donanimhaber.com/api/search/portal/?q=vodafone&p=3&devicetype=browsermobile&order=date_desc&in=all&contenttype=all&wordtype=both&daterange=all"

jsonData = requests.post(api_url, data=json.dumps(body), headers=headers).json()

for item in jsonData['contents']:
    categoryName=item['categoryName']
    print(categoryName)

輸出：

Operat?rler - Kurumsal Haberler
Operat?rler - Kurumsal Haberler
Operat?rler - Kurumsal Haberler
Mobil Aksesuarlar
Operat?rler - Kurumsal Haberler
Kripto Para
Sinema ve Dizi
Mobil Oyunlar
Operat?rler - Kurumsal Haberler
Operat?rler - Kurumsal Haberler

uj5u.com熱心網友回復：

如前所述requests，無法渲染JavaScript，但有兩種選擇：

• 在您的網址上使用requests和執行發布請求
• 用于selenium獲得page_source您所期望的渲染。

例子

from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

url = 'https://search.donanimhaber.com/portal?q=vodafone&p=3&devicetype=browsermobile&order=date_desc&in=all&contenttype=all&wordtype=both&range=all'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
wait = WebDriverWait(driver, 10)
driver.get(url)

wait.until(EC.presence_of_all_elements_located((By.XPATH, './/div[@]/div[@]')))

content = driver.page_source
soup = BeautifulSoup(content,"html.parser")

pages = soup.find_all('div', {'class': 'snippet'})

for p in pages:
    print(p.h2.text.strip())

輸出

Vodafone'dan dijital sa?l?k projelerine ücretsiz 5G deste?i
Vodafone'un son 15 y?lda Türkiye ekonomisine katk?s? a??kland?
"Yar?n? Kodlayanlar" projesinde gen?ler afet sorunlar?na ??zümler üretti
Küresel ak?ll? saat pazar? y?l?n ilk ?eyre?inde yüzde 35 büyüdü
Vodafone Türkiye'nin ilk ?eyrek sonu?lar? a??kland?: Servis gelirlerinde yüzde 19 art??
Netflix'e yeni eklenen dizi ve filmleri takip edebilece?iniz site
Sony ve SinemaTV anla?t?! Spider-Man, Venom 2 ve daha fazlas? TV'de ilk kez SinemaTV'de yay?nlanacak
Vodafone ve Riot Games, Türkiye'nin ilk 5G Wild Rift turnuvas?n? duyurdu
Türkiye'de ka? ki?i numara ta??ma ile operat?r de?i?tirdi?
Turkcell'in Ramazan'a ?zel Salla Kazan kampanyas? ba?lad?

標籤：Python html 网页抓取 美丽的汤 邮差

上一篇：如何使用seleniumpython下載電子郵件附件？

下一篇：如何使用沒有值的屬性查找頁面上的每個元素？