我正在嘗試從 FT 網站上抓取每日基金價格。URL 的型別如下：https://markets.ft.com/data/funds/tearsheet/historical?s=LU0526609390:EUR，其中 LU0526609390 是基金的 ISIN。問題是有一個過濾器可以訪問超過最近 30 個每日基金價格。
但是，資料是從允許設定日期范圍的 API 加載的，例如對于基金 LU052660390，我必須使用以下 URL：https ://markets.ft.com/data/equities/ajax/get-historical-prices ? startDate=2020/10/01&endDate=2021/10/01&symbol=535700333。這使得跳過過濾器問題成為可能。
我想從原始基金 URL 中提取符號 ( xid = 535700333)，然后請求所有可用的每日價格，最后將資訊匯出到ISIN.csv檔案中。
我有以下代碼，以 4 個基金 URL 為例：
from bs4 import BeautifulSoup
import requests
import json
import time
import pandas as pd
from datetime import datetime
#Create url list
urls = ['https://markets.ft.com/data/funds/tearsheet/historical?s=LU0526609390:EUR', 'https://markets.ft.com/data/funds/tearsheet/historical?s=IE00BHBX0Z19:EUR',
'https://markets.ft.com/data/funds/tearsheet/historical?s=LU1076093779:EUR', 'https://markets.ft.com/data/funds/tearsheet/historical?s=LU1116896363:EUR']
#create list of annual dates for the past 100 years starting from today
datelist = pd.date_range(end=datetime.now(),periods=100,freq=pd.DateOffset(years=1))[::-1].strftime('%Y/%m/%d')
#Build Dataframe
df = pd.DataFrame(None, columns=['Date','Open','High','Low','Close','Volume'])
# Change date format as there appears to be two versions of the date on the FT website for different sized browsers
def format_date(date):
date = date.split(',')[-2][1:] date.split(',')[-1]
return pd.Series({'Date': date})
# Build the scrapping loop
for url in urls:
ISIN = url.split('=')[-1].replace(':', '_')
ISIN = ISIN[:-4]
# Extract HTML element (symbol) from original fund url
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
elemList = soup.find_all('section', {'class':'mod-tearsheet-add-to-watchlist'})
for elem in elemList:
elemID = elem.get('class')
elemName = elem.get('data-mod-config')
if elemName is None:
pass
elif 'xid' in elemName:
data = json.loads(elemName)
val1 = data['xid']
# We have now extracted the fund xid so we can start the request loop using the API.
while True:
for end, start in zip(datelist, datelist[1:]):
try:
r = requests.get(f'https://markets.ft.com/data/equities/ajax/get-historical-prices?startDate={start}&endDate={end}&symbol={val1}').json()
df_temp = pd.read_html('<table>' r['html'] '</table>')[0]
df_temp.columns=['Date','Open','High','Low','Close','Volume']
df['Date'] = df['Date'].apply(format_date)
df.to_csv(r'/Users//' ISIN '.csv', index=False)
except:
break
break
但是，我不斷收到相同的ValueError: No tables found matching pattern '. '錯誤訊息。我是 Python 的新手，因此對于我為什么會收到此錯誤的任何幫助，將不勝感激！謝謝
uj5u.com熱心網友回復：
我沒有收到錯誤，在我這邊運行得很好。我唯一建議的是使用用戶代理添加 headers 引數。您可能不會得到 200 回應。在您的流程中放入一些列印陳述句也可能會有所幫助，以便您可以除錯并查看代碼在哪里跳閘。試試這個，看看它是否有幫助：
from bs4 import BeautifulSoup
import requests
import json
import time
import pandas as pd
from datetime import datetime
#Create url list
urls = ['https://markets.ft.com/data/funds/tearsheet/historical?s=LU0526609390:EUR', 'https://markets.ft.com/data/funds/tearsheet/historical?s=IE00BHBX0Z19:EUR',
'https://markets.ft.com/data/funds/tearsheet/historical?s=LU1076093779:EUR', 'https://markets.ft.com/data/funds/tearsheet/historical?s=LU1116896363:EUR']
#create list of annual dates for the past 100 years starting from today
datelist = pd.date_range(end=datetime.now(),periods=100,freq=pd.DateOffset(years=1))[::-1].strftime('%Y/%m/%d')
#Build Dataframe
df = pd.DataFrame(None, columns=['Date','Open','High','Low','Close','Volume'])
# Change date format as there appears to be two versions of the date on the FT website for different sized browsers
def format_date(date):
date = date.split(',')[-2][1:] date.split(',')[-1]
return pd.Series({'Date': date})
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36'}
# Build the scrapping loop
for url in urls:
ISIN = url.split('=')[-1].replace(':', '_')
ISIN = ISIN[:-4]
# Extract HTML element (symbol) from original fund url
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, "html.parser")
elemList = soup.find_all('section', {'class':'mod-tearsheet-add-to-watchlist'})
for elem in elemList:
elemID = elem.get('class')
elemName = elem.get('data-mod-config')
if elemName is None:
pass
elif 'xid' in elemName:
data = json.loads(elemName)
val1 = data['xid']
# We have now extracted the fund xid so we can start the request loop using the API.
while True:
for end, start in zip(datelist, datelist[1:]):
try:
r = requests.get(f'https://markets.ft.com/data/equities/ajax/get-historical-prices?startDate={start}&endDate={end}&symbol={val1}', headers=headers).json()
df_temp = pd.read_html('<table>' r['html'] '</table>')[0]
df_temp.columns=['Date','Open','High','Low','Close','Volume']
df['Date'] = df['Date'].apply(format_date)
df.to_csv(r'/Users//' ISIN '.csv', index=False)
except:
break
break
轉載請註明出處，本文鏈接：https://www.uj5u.com/qukuanlian/331246.html