如何簡化網頁抓取代碼中的if陳述句？

我正在嘗試在此頁面https://myanimelist.net/topanime.php?limit=0 ( instead of page=1, page=2 and on, it goes like limit=0,limit=50, limit=100...) 中從 page1,2,3... 獲取資訊。如何簡化 de IF 函式中的代碼？

data = []

def main(number):

    driver = webdriver.Chrome()
    url = 'https://myanimelist.net/topanime.php?limit={}'

    
    if number <= 1:
        driver.get(url.format(0))
        soup = BeautifulSoup(driver.page_source, 'html.parser')
        results = soup.find_all('tr', class_= 'ranking-list')

        for result in results:
            Anime = result.find('h3', class_='hoverinfo_trigger fl-l fs14 fw-b anime_ranking_h3').text.replace('\n','')
            Date = result.find('div', class_='information di-ib mt4').text.replace('\n','')
            No_eps = result.find('div', class_='information di-ib mt4').text.replace('\n','')
            Ranking = result.find('td', class_='rank ac').text.replace('\n','')
            Score = result.find('div', class_='js-top-ranking-score-col di-ib al').text.replace('\n','')
            data.append({
                'Anime':Anime,
                'Date':Date,
                'No_eps':No_eps,
                'Ranking':Ranking,
                'Score':Score
            })
    elif number >= 2:
        for limit in range(0,(int(number)*50), 50):
            driver.get(url.format(limit))

            soup = BeautifulSoup(driver.page_source, 'html.parser')
            results = soup.find_all('tr', class_= 'ranking-list')

            for result in results:
                Anime = result.find('h3', class_='hoverinfo_trigger fl-l fs14 fw-b anime_ranking_h3').text.replace('\n','')
                Date = result.find('div', class_='information di-ib mt4').text.replace('\n','')
                No_eps = result.find('div', class_='information di-ib mt4').text.replace('\n','')
                Ranking = result.find('td', class_='rank ac').text.replace('\n','')
                Score = result.find('div', class_='js-top-ranking-score-col di-ib al').text.replace('\n','')
                data.append({
                    'Anime':Anime,
                    'Date':Date,
                    'No_eps':No_eps,
                    'Ranking':Ranking,
                    'Score':Score
                })

uj5u.com熱心網友回復：

如果存在，我會嘗試從按鈕中提取特定的 url / 引數，而不是猜測可能的下一頁編號：

if soup.select_one('a.link-blue-box.next'):
    url = 'https://myanimelist.net/topanime.php' soup.select_one('a.link-blue-box.next').get('href')
else:
    break

它也不需要大炮selenium，而是簡化使用requests并得到剝離的文本可以使用.get_text(strip=True)。

例子

出于演示目的，限制從 22100 開始，只需將其設定為 0 即可獲得所有結果。

import requests
from bs4 import BeautifulSoup


url = 'https://myanimelist.net/topanime.php?limit=22100'
data = []
while True:
    
    r = requests.get(url)
    soup = BeautifulSoup(r.text)
    
    for e in soup.find_all('tr', class_= 'ranking-list'):
        Anime = e.h3.get_text(strip=True),
        Date = e.find('div', class_='information di-ib mt4').get_text(strip=True)
        No_eps = e.find('div', class_='information di-ib mt4').get_text(strip=True)
        Ranking = e.find('td', class_='rank ac').get_text(strip=True)
        Score = e.find('div', class_='js-top-ranking-score-col di-ib al').get_text(strip=True)
        data.append({
            'Anime':Anime,
            'Date':Date,
            'No_eps':No_eps,
            'Ranking':Ranking,
            'Score':Score
        })
    if soup.select_one('a.link-blue-box.next'):
        url = 'https://myanimelist.net/topanime.php' soup.select_one('a.link-blue-box.next').get('href')
    else:
        break
data

標籤：Python熊猫if 语句网页抓取美丽的汤

上一篇：如何在case_when中將dplyr的if_all()與R中的數字資料一起使用？

下一篇：如何使用foreach和if過濾串列中的數字？