我需要對 googlenews 進行網路抓取，以獲取來自不同報紙的不同文章的鏈接，并且我有一個適用于今日新聞（來自 googlenews）的代碼。但是它不適用于舊文章。例如，這段代碼可以從谷歌新聞中獲取不同的文章鏈接：
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import requests
import time
from newspaper import Article
import random
import pandas as pd
root = 'https://www.google.com/'
time.sleep(random.randint(0, 3)) #----------stop---------#
link = 'https://www.google.com/search?q=revuelta la tercera&rlz=1C1UEAD_esCL995CL995&biw=1536&bih=714&tbm=nws&ei=qEWUYorfOuiy5OUP-aGLgA4&ved=0ahUKEwiK07Wfr4b4AhVoGbkGHfnQAuAQ4dUDCA0&uact=5&oq=revuelta la tercera&gs_lcp=Cgxnd3Mtd2l6LW5ld3MQAzIFCCEQoAEyBQghEKABOgsIABCABBCxAxCDAToFCAAQgAQ6CAgAEIAEELEDOggIABCxAxCDAToKCAAQsQMQgwEQQzoECAAQQzoECAAQCjoGCAAQHhAWOggIABAeEA8QFlDIEliUnwFg1aABaAVwAHgAgAGSAYgBuw-SAQQyMS4ymAEAoAEBsAEAwAEB&sclient=gws-wiz-news'
time.sleep(random.randint(0, 6)) #----------stop---------#
req = Request(link, headers = {'User-Agent': 'Mozilla/5.0'})
time.sleep(random.randint(0, 3)) #----------stop---------#
requests.get(link, headers = {'User-agent': 'your bot 0.1'})
time.sleep(random.randint(0, 6)) #----------stop---------#
webpage = urlopen(req).read()
time.sleep(random.randint(0, 6)) #----------stop---------#
with requests.Session() as c:
soup = BeautifulSoup(webpage, 'html5lib')
for item in soup.find_all('div', attrs= {'class': 'ZINbbc luh4tb xpd O9g5cc uUPGi'}):
raw_link = item.find('a', href=True)['href']
link = raw_link.split('/url?q=')[1].split('&sa=U&')[0]
article = Article(link, language = "es")
article.download()
article.parse()
title = article.title
descript = article.text
date = article.publish_date
print(title)
print(descript)
print(link)
現在我需要更改相同搜索的日期，所以我只需使用自定義間隔更改鏈接：
root = 'https://www.google.com/'
time.sleep(random.randint(0, 3)) #----------stop---------#
link = 'https://www.google.com/search?q=revuelta la tercera&rlz=1C1UEAD_esCL995CL995&biw=1536&bih=714&source=lnt&tbs=cdr:1,cd_min:1/1/2018,cd_max:1/6/2018&tbm=nws'
time.sleep(random.randint(0, 6)) #----------stop---------#
req = Request(link, headers = {'User-Agent': 'Mozilla/5.0'})
time.sleep(random.randint(0, 3)) #----------stop---------#
requests.get(link, headers = {'User-agent': 'your bot 0.1'})
time.sleep(random.randint(0, 6)) #----------stop---------#
webpage = urlopen(req).read()
time.sleep(random.randint(0, 6)) #----------stop---------#
with requests.Session() as c:
soup = BeautifulSoup(webpage, 'html5lib')
for item in soup.find_all('div', attrs= {'class': 'ZINbbc luh4tb xpd O9g5cc uUPGi'}):
raw_link = item.find('a', href=True)['href']
link = raw_link.split('/url?q=')[1].split('&sa=U&')[0]
article = Article(link, language = "es")
article.download()
article.parse()
title = article.title
descript = article.text
date = article.publish_date
print(title)
print(descript)
print(link)
鏈接應該是不同的（由于搜索日期的變化），但它們都給了我相同的結果，我不明白為什么。請大家幫忙，我不知道如何解決這個問題。
uj5u.com熱心網友回復：
通過您提供的網址是
https://www.google.com/search?q=revuelta la tercera&rlz=1C1UEAD_esCL995CL995&biw=1536&bih=714&source=lnt&tbs=cdr:1,cd_min:1/1/2018,cd_max:1/6/2018&tbm=nws
如果您仔細閱讀cd_min和cd_max似乎包含日期時間資料。
cd_min:1/1/2018,cd_max:1/6/2018
所以讓我們從 URL 中分割它們。上面的字串是從 URL 中分割出來的，并且是 URL 編碼的。如果你解碼它，你會看到..
cd_min:1/1/2018,cd_max:1/6/2018
因此，如果您想更改查詢的日期，則應更改 URL。
from urllib import parse
# URL ENCODE
# DON'T FORGET : and ,
start_date = parse.quote_plus(":1/1/2018,")
end_date = parse.quote_plus(":1/6/2018")
# CREATE QUERY
link = f"https://www.google.com/search?q=revuelta la tercera&rlz=1C1UEAD_esCL995CL995&biw=1536&bih=714&source=lnt&tbs=cdr:1,cd_min{start_date}cd_max{end_date}&tbm=nws"
隨機化日期的代碼是你的作業:)
轉載請註明出處，本文鏈接：https://www.uj5u.com/qukuanlian/485041.html
上一篇：pdf檔案不會下載