我有以下抓取腳本，我需要在“items2”回圈中獲取元素。該腳本正在列印所有元素，但稍后資料框將“name”和“tPlan”回傳為 NaN。知道為什么嗎？
import requests
import json
import csv
import sys
from bs4 import BeautifulSoup
base_url = "xxxx"
username = "xxxx"
password = "xxxx"
toget = data
allowed_results = 50
max_results = "maxResults=" str(allowed_results)
tc = "/testcycles?"
result_count = -1
start_index = 0
df = pd.DataFrame(
columns=['id', 'name', 'gId', 'dKey', 'tPlan'])
for eachId in toget['TPlan_ID']:
while result_count != 0:
start_at = "startAt=" str(start_index)
url = url = f'{base_url}{eachId}{tc}&{start_at}&{max_results}'
response = requests.get(url, auth=(username, password))
json_response = json.loads(response.text)
print(json_response)
page_info = json_response["meta"]["pageInfo"]
start_index = page_info["startIndex"] allowed_results
result_count = page_info["resultCount"]
items2 = json_response["data"]
print(items2)
for item in items2:
print (item["id"])
print (item["fields"]["name"])
print (item["fields"]["gId"])
print (item["fields"]["dKey"])
print (item["fields"]["tPlan"])
temporary_df = pd.DataFrame([item], columns=['id', 'name', 'gId', 'dKey', 'tPlan'])
df = df.append(temporary_df, ignore_index=True)
uj5u.com熱心網友回復：
TLDR
使用這個 for 回圈。
for item in items2:
df = df.append({'id': item['id'], **item['fields']}, ignore_index=True)
解釋
我假設它items2看起來像這樣。
items2 = [
{ 'id': 0, 'fields': {'name': 'prop1', 'gId': 100, 'dKey': 'key1', 'tPlan': 'plan1'}},
{ 'id': 1, 'fields': {'name': 'prop2', 'gId': 200, 'dKey': 'key2', 'tPlan': 'plan2'}},
{ 'id': 2, 'fields': {'name': 'prop3', 'gId': 300, 'dKey': 'key3', 'tPlan': 'plan3'}},
]
item由于結構是這樣的，因此您無法創建預期的資料框。
{'id': 2, 'fields': {'name': 'prop3', 'gId': 300, 'dKey': 'key3', 'tPlan': 'plan3'}}
這導致temporary_df填充了NaN。
id name gId dKey tPlan fields
0 0 NaN NaN NaN NaN key1
1 0 NaN NaN NaN NaN 100
2 0 NaN NaN NaN NaN prop1
3 0 NaN NaN NaN NaN plan1
4 1 NaN NaN NaN NaN key2
5 1 NaN NaN NaN NaN 200
6 1 NaN NaN NaN NaN prop2
7 1 NaN NaN NaN NaN plan2
8 2 NaN NaN NaN NaN key3
9 2 NaN NaN NaN NaN 300
10 2 NaN NaN NaN NaN prop3
11 2 NaN NaN NaN NaN plan3
您需要作為引數傳遞給 pd.DataFrame 的是一個 dict 結構，例如
{'id': 2, 'name': 'prop3', 'gId': 300, 'dKey': 'key3', 'tPlan': 'plan3'}
注意這里缺少的fields字典，所有的鍵值對fields都被添加到item. 使用這個改變的 dict 會導致 temporary_df類似
id name gId dKey tPlan
0 0 prop1 100 key1 plan1
1 1 prop2 200 key2 plan2
2 2 prop3 300 key3 plan3
要在專案結構中進行此更改，您應該這樣做
new_item = {'id': item['id']}
for key, value in item['fields'].items():
new_item[key] = value
但是您可以使用解包運算子簡潔地撰寫此代碼 **
new_item = {'id': item['id'], **item['fields']}
現在我們可以使用 passnew_item作為pd.DataFrame.
temp_df = pd.DataFrame({ 'id': item['id'], **item['fields']}, index=(i,)) # i here is the row index of the DataFrame
After making these changes your for loop should look something like this
for i, item in enumerate(items2):
new_item = {'id': item['id'], **item['fields']}
temp_df = pd.DataFrame(new_item, index=(i,))
df = df.append(temp_df, ignore_index=True)
We can make this a bit more concise by directly passing the new_item to pd.DataFrame.append
Thus in the end this code should work.
for item in items2:
new_item = {'id': item['id'], **item['fields']}
df = df.append(new_item, ignore_index=True)
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/448298.html
下一篇：生成所有排列集