從多個URL中提取p個標簽

我已經為此苦苦掙扎了好幾天，不確定問題出在哪里——基本上，我正在嘗試提取每個鏈接的組態檔框資料（下圖）——通過檢查員，我想我可以拉出 p 標簽并這樣做。

我對此很陌生并試圖理解，但這是我迄今為止所擁有的：

- 一段代碼（在某種程度上）成功提取了一個鏈接的資訊：

import requests
from bs4 import BeautifulSoup


# getting html

url = 'https://basketball.realgm.com/player/Darius-Adams/Summary/28720'
req = requests.get(url)

soup = BeautifulSoup(req.text, 'html.parser')

container = soup.find('div', attrs={'class', 'main-container'})
playerinfo = container.find_all('p')


print(playerinfo)

然后，我還有一個代碼可以從多個鏈接中提取所有 HREF 標記：

    from bs4 import BeautifulSoup
import requests


def get_links(url):

    links = []

    website = requests.get(url)
    website_text = website.text
    soup = BeautifulSoup(website_text)

    for link in soup.find_all('a'):

        links.append(link.get('href'))

    for link in links:
        print(link)

        print(len(links))


get_links('https://basketball.realgm.com/dleague/players/2022')
get_links('https://basketball.realgm.com/dleague/players/2021')
get_links('https://basketball.realgm.com/dleague/players/2020')

所以基本上，我的目標是將這兩者結合起來，得到一個代碼，可以從多個 URL 中提取所有 P 標簽。我一直在嘗試這樣做，但我真的不確定為什么這在這里不起作用：

    from bs4 import BeautifulSoup
import requests


def get_profile(url):

    profiles = []

    req = requests.get(url)
    soup = BeautifulSoup(req.text, 'html.parser')
    container = soup.find('div', attrs={'class', 'main-container'})

    for profile in container.find_all('a'):

        profiles.append(profile.get('p'))

    for profile in profiles:
        print(profile)


get_profile('https://basketball.realgm.com/player/Darius-Adams/Summary/28720')
get_profile('https://basketball.realgm.com/player/Marial-Shayok/Summary/26697')

同樣，我對使用 Python 進行網路抓取真的很陌生，但任何建議都將不勝感激。最終，我的最終目標是擁有一個可以一次以干凈的方式抓取這些資料的工具。

（球員姓名，當前球隊，出生，出生地等）..也許我做錯了，但歡迎任何指導！

uj5u.com熱心網友回復：

您需要將兩個腳本組合在一起并對每個播放器發出請求。嘗試以下方法。這將搜索<td>具有以下data-td=Player屬性的標簽：

import requests
from bs4 import BeautifulSoup

def get_links(url):
    data = []
    req_url = requests.get(url)
    soup = BeautifulSoup(req_url.content, "html.parser")

    for td in soup.find_all('td', {'data-th' : 'Player'}):
        a_tag = td.a
        name = a_tag.text
        player_url = a_tag['href']
        print(f"Getting {name}")

        req_player_url = requests.get(f"https://basketball.realgm.com{player_url}")
        soup_player = BeautifulSoup(req_player_url.content, "html.parser")
        div_profile_box = soup_player.find("div", class_="profile-box")
        row = {"Name" : name, "URL" : player_url}
        
        for p in div_profile_box.find_all("p"):
            try:
                key, value = p.get_text(strip=True).split(':', 1)
                row[key.strip()] = value.strip()
            except:     # not all entries have values
                pass

        data.append(row)

    return data

urls = [
    'https://basketball.realgm.com/dleague/players/2022',
    'https://basketball.realgm.com/dleague/players/2021',
    'https://basketball.realgm.com/dleague/players/2020',
]


for url in urls:
    print(f"Getting: {url}")
    data = get_links(url)
    
    for entry in data:
        print(entry)

標籤：Python 网页抓取 美丽的汤

上一篇：使用Python網頁抓取和下載excel檔案

下一篇：為什么決議只發生在每個表的第一項上