我對美麗的湯不熟悉。我正在努力從源中抓取一些 excel 檔案。來源網址：https ://droughtmonitor.unl.edu/DmData/GISData.aspx/?mode=table&aoi=county&date= 原始資料來源：https ://droughtmonitor.unl.edu/DmData/GISData.aspx/
我的主要目標是從這個 URL 中抓取資料并將其轉換為一個資料框，包括原始資料源 URL 中的所有檔案，以及是否可以自動下載添加的一些新檔案并將其添加到源中。
from bs4 import BeautifulSoup
import requests
import json
import pandas as pd
url2 = 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/?mode=table&aoi=county&date=2022-06-21'
r2 = requests.get(url2)
soup = BeautifulSoup(r2.text,'html.parser')
raw_data = [data.text for data in soup]
上面的代碼給了我一個輸出： -
["MapDate,FIPS,County,State,Nothing,D0,D1,D2,D3,D4,ValidStart,ValidEnd\r\n20220621,01001,Autauga County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01003,Baldwin County,AL,81.22,18.78,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01005,Barbour County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01007,Bibb County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01009,Blount County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01011,Bullock County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01013,Butler County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01015,Calhoun County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01017,Chambers County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01019,Cherokee County,AL,69.27,30.73,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n20220621,01021,Chilton County,AL,100.00,0.00,0.00,0.00,0.00,0.00,2022-06-21,2022-06-27\r\n
我想讓最初的 12 個值：MapDate,FIPS,County,State,Nothing,D0,D1,D2,D3,D4,ValidStart,ValidEnd 成為我的列名，并保留與相同的值相關聯的值。
此外，原始資料源 URL 的值從 2000 年到 2022 年。我需要相同格式和單個 CSV 格式的所有資料。
另外，我需要讓代碼自動提取添加到源中的任何新資料。
有人可以指導我嗎？
uj5u.com熱心網友回復：
它發送檔案csv，所以你不需要BeautifulSoup
你可以io使用pandas.read_csv()
import requests
import pandas as pd
import io
url = 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/?mode=table&aoi=county&date=2022-06-21'
response = requests.get(url)
fh = io.StringIO(response.text) # create file in memory
df = pd.read_csv(fh)
print(df)
或者你可以使用io模塊csv
import requests
import csv
import io
url = 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/?mode=table&aoi=county&date=2022-06-21'
response = requests.get(url)
fh = io.StringIO(response.text) # create file in memory
data = list(csv.reader(fh))
print(data)
編輯：
你甚至可以url直接使用pandas
import pandas as pd
url = 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/?mode=table&aoi=county&date=2022-06-21'
df = pd.read_csv(url)
print(df)
編輯：
現在您只需要列出日期并運行for-loop 即可讀取所有 csv 并保留在串列中。稍后您可以使用pandas.concat()將此串列轉換為 single DataFrame。
Pandas doc：合并、連接、連接和比較
最小的作業示例：
import pandas as pd
# --- before loop ---
all_dates = ["2022-06-21", "2022-06-14", "2022-06-07"]
all_dfs = []
# url without `2022-06-21` at the end
url = 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/?mode=table&aoi=county&date='
# --- loop ---
for date in all_dates:
print('date:', date)
df = pd.read_csv( url date )
all_dfs.append( df )
# --- after loop ---
full_df = pd.concat(all_dfs)
print(full_df)
要獲取日期串列，您可以從網頁上抓取它們，但它可能需要Selenium而不是 beautifulsoup因為頁面使用 JavaScript 在頁面上添加日期。
或者您應該使用DevTools(tab: Network, filter: XHR) 來查看 JavaScript 使用哪個 url 來獲取日期并用于獲取日期requests。
import requests
# without header `Content-Type` it sends `HTML` instead of `JSON`
headers = {
# 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0',
# 'X-Requested-With': 'XMLHttpRequest',
# 'Referer': 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/',
'Content-Type': 'application/json; charset=utf-8',
}
url = 'https://droughtmonitor.unl.edu/DmData/GISData.aspx/ReturnDMWeeks'
response = requests.get(url, headers=headers)
#print(response.text)
data = response.json()
all_dates = data['d']
all_dates = [f"{d[:4]}-{d[4:6]}-{d[6:]}" for d in all_dates]
print(all_dates)
結果
['2022-06-21', '2022-06-14', '2022-06-07', ..., '2000-01-18', '2000-01-11', '2000-01-04']
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/497250.html