如何在回圈中輸入可以添加到url以獲取結果的單詞串列

我將 url 作為輸入： url = "https://www.amazon.in/s?k=headphones&page=1" 這作業正常，但停在第 19 頁而不是我們在第 19 頁中斷，我想給出下一個輸入為“https://www.amazon.in/s?k=”

1. “演講者&頁面=1”
2. "earbuds&page=1" 等回圈運行

from bs4 import BeautifulSoup as soup
import pandas as pd
import requests

data =[]

def getdata (url):
    header = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' } 
    req = urllib.request.Request(url, headers=header)
    amazon_html = urllib.request.urlopen(req).read()
    a_soup = soup(amazon_html,'html.parser')
    
    for e in a_soup.select('div[data-component-type="s-search-result"]'):
        try:
            title = e.find('h2').text
        except:
            title = None
            
        data.append({
            'title':title
        })
        
    return a_soup

def getnextpage(a_soup):
  page= a_soup.find('a',attrs={"class": 's-pagination-item s-pagination-next s-pagination-button s-pagination-separator'})
  page = page['href']
  url =  'http://www.amazon.in'  str(page)
  return url
            
while True:
  geturl = getdata(url)
  url = getnextpage(geturl)
    
  if not url:
    break
  print(url)```


```output = pd.DataFrame(data)
output

此代碼回傳正確的結果，但不是每次我希望它輸入一個可以在 url 末尾添加的專案串列時都提供一個新的 url 以獲取可以添加到的結果DataFrame 注意：搜索結果在第 19 頁停止

uj5u.com熱心網友回復：

為您的關鍵字制作一個串列，對其進行迭代并將 while 回圈包含在每次迭代中。

keywords = ['speakers','earbuds']

for k in keywords:
    url = 'https://www.amazon.in/s?k=' k
    while True:
        geturl = getdata(url)
        url = getnextpage(geturl)

        if not url:
            break
        print(url)

請注意，亞馬遜不喜歡對其頁面進行這種自動訪問，并且可以很快識別訪問模式。為了稍微降低請求的頻率，您至少應該包含一些 delay time.sleep()。當然，使用官方的api會更好。

例子

from bs4 import BeautifulSoup as soup
import pandas as pd
import requests
import urllib

data =[]

def getdata (url):
    header = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' } 
    req = urllib.request.Request(url, headers=header)
    amazon_html = urllib.request.urlopen(req).read()
    a_soup = soup(amazon_html,'html.parser')
    
    for e in a_soup.select('div[data-component-type="s-search-result"]'):
        try:
            title = e.find('h2').text
        except:
            title = None
            
        data.append({
            'title':title,
            'url':'http://www.amazon.in'   e.h2.a['href']
        })
        
    return a_soup

def getnextpage(a_soup):
    try:
        page = a_soup.find('a',attrs={"class": 's-pagination-item s-pagination-next s-pagination-button s-pagination-separator'})['href']
        url =  'http://www.amazon.in'  str(page)
    except:
        url = None
    return url


keywords = ['speakers','earbuds']

for k in keywords:
    url = 'https://www.amazon.in/s?k=' k
    while True:
        geturl = getdata(url)
        url = getnextpage(geturl)

        if not url:
            break
        print(url)

輸出（列印）

http://www.amazon.in/s?k=speakers&page=2&qid=1649420352&ref=sr_pg_1
...
http://www.amazon.in/s?k=speakers&page=20&qid=1649420373&ref=sr_pg_19
http://www.amazon.in/s?k=earbuds&page=2&qid=1649420375&ref=sr_pg_1
...
http://www.amazon.in/s?k=earbuds&page=20&qid=1649420394&ref=sr_pg_19

輸出 ( pd.DataFrame(data))

標籤：Python html 循环 网页抓取 美丽的汤

上一篇：Python中用于網頁抓取的回圈函式

下一篇：如何抓取網站上的特定資訊