我正在嘗試從百老匯演出頁面中提取統計資料表。這是我到目前為止：
import requests
from bs4 import BeautifulSoup
springsteen_url = 'https://www.ibdb.com/broadway-production/springsteen-on-broadway-515480#Statistics'
springsteen_response = requests.get(springsteen_url)
springsteen_soup = BeautifulSoup(springsteen_response.text, 'html.parser')
我找到了表格標簽，有 2 個名為“striped”的我選擇了第一個：
statistics = soup.find_all("table", attrs={"class": "striped"})
table1 = statistics[0]
我能夠創建一個 df 并填充標題，但不會填充行/單元格：
# Defining of the dataframe
df = pd.DataFrame(columns=['Week Ending', 'Gross', '% Gross Pot.', 'Attendance', '% Capacity'])
# Collecting Ddata
for row in table.tbody.find_all('tr'):
# Find all data for each column
columns = row.find_all('td')
if(columns != []):
week = columns[0].text.strip()
gross = columns[1].text.strip()
grosspot = columns[2].span.contents[0].strip('&0.')
attendance = columns[3].span.contents[0].strip('&0.')
capacity = columns[4].span.contents[0].strip('&0.')
df = df.append({'Week Ending': week, 'Gross': gross, '% Gross Pot.': grosspot, 'Attendance': attendance, '% Capacity': capacity}, ignore_index=True)
df.head()
uj5u.com熱心網友回復：
首先，我發現了一個錯誤在你的代碼，您使用springsteen_response.text的BeautifulSoup(springsteen_response.text, 'html.parser')，而不是springsteen_response.content因此要盡量避免它沒有與字符編碼的問題。該.content屬性保存原始位元組，可以更好地解碼。
我還使用Ali De?er Ozbakir's 提示撰寫了一個解決您的問題的方法，以使用正則運算式提取資料。我在代碼注釋中解釋了一些事情，但簡而言之，您需要使用Python 正則運算式來提取<script>標簽內的資料，然后從該正則運算式中提取一個組，最后您必須使用它ast.literal_eval來決議資料字串，因為該字串不匹配JSON格式，因此您無法決議它。
代碼：
import re
import ast
import requests
import pandas as pd
from bs4 import BeautifulSoup
springsteen_url = 'https://www.ibdb.com/broadway-production/springsteen-on-broadway-515480#Statistics'
springsteen_response = requests.get(springsteen_url)
springsteen_soup = BeautifulSoup(springsteen_response.content, 'html.parser')
# Find a <script> tag which have "grossdata" text inside
scriptdata = springsteen_soup.find("script", text=re.compile("grossdata")).string
# this is a regular expression
# ? after space means that space may or may not occur in string
# . (dot) means we are matching any character
# sign when used with previous dot means that any character need to occur at least one time
# at the end ";" may or may not occur
grossdata_pattern = re.compile(r"grossdata ?= ?({. });?")
grossdata_search = grossdata_pattern.search(scriptdata)
if grossdata_search:
# We are extracting here a second group (first is the whole string from "grossdata" to ";")
# the second group is simply between parentheses "()"
grossdata = grossdata_search.group(1)
else:
print("Failed to locate string to match regex.")
# Because of a data didn't match JSON format we couldn't parse to it
# so we need to use safe evaluation of a python code
# to make it a python dictionary
grossdata = ast.literal_eval(grossdata)
grossdata_2018 = grossdata[0]
df = pd.DataFrame(grossdata_2018)
# Remove unwanted columns
df.drop([5, 6, 7, 8, 9, 10, 11], axis=1, inplace=True)
df.columns = ['Week Ending', 'Gross', '% Gross Pot.', 'Attendance', '% Capacity']
print(df)
輸出：
Week Ending Gross % Gross Pot. Attendance % Capacity
0 Jun 3, 2018 $1,929,448 101% 3,792 100%
1 Jun 10, 2018 $2,402,103 101% 4,740 100%
2 Jun 17, 2018 $1,930,270 101% 3,792 100%
3 Jun 24, 2018 $2,411,075 101% 4,740 100%
4 Jul 1, 2018 $1,929,003 101% 3,792 100%
5 Jul 15, 2018 $2,410,195 101% 4,740 100%
6 Jul 22, 2018 $964,163 101% 1,895 100%
7 Jul 29, 2018 $1,931,618 101% 3,792 100%
8 Aug 12, 2018 $1,931,445 101% 3,792 100%
9 Aug 19, 2018 $2,406,003 101% 4,729 100%
10 Aug 26, 2018 $1,928,960 101% 3,792 100%
11 Sep 2, 2018 $1,930,020 101% 3,792 100%
12 Sep 9, 2018 $1,932,670 101% 3,792 100%
13 Sep 30, 2018 $1,927,620 101% 3,792 100%
14 Oct 7, 2018 $2,410,528 101% 4,740 100%
15 Oct 14, 2018 $1,929,795 101% 3,792 100%
16 Oct 21, 2018 $2,414,700 101% 4,740 100%
17 Oct 28, 2018 $1,937,995 102% 3,792 100%
18 Nov 4, 2018 $2,424,880 102% 4,740 100%
19 Nov 11, 2018 $1,935,170 101% 3,792 100%
20 Nov 18, 2018 $2,423,700 102% 4,740 100%
21 Dec 2, 2018 $1,929,320 101% 3,792 100%
22 Dec 9, 2018 $2,415,700 101% 4,740 100%
23 Dec 16, 2018 $1,895,695 100% 3,792 100%
轉載請註明出處，本文鏈接：https://www.uj5u.com/gongcheng/354334.html
標籤：Python 网页抓取 html-table
上一篇：如何使用R從網站下載檔案