Pandas引發錯誤索引包含重復條目，無法重塑

嘗試將資料附加到資料幀但引發錯誤

索引包含重復條目

我想要 csv 檔案中的資料基本上他們在這些行中向我顯示錯誤：

df = df.pivot(index=['v1','v2','v3'], columns='image', values='link').reset_index().fillna('')

代碼：

import requests
from bs4 import BeautifulSoup
import pandas as pd
baseurl='https://twillmkt.com'
headers ={
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r =requests.get('https://twillmkt.com/collections/denim')
soup=BeautifulSoup(r.content, 'html.parser')
tra = soup.find_all('div',class_='ProductItem__Wrapper')
productlinks=[]
for links in tra:
    for link in links.find_all('a',href=True)[:1]:
        comp=baseurl link['href']
        productlinks.append(comp)
data=[]    
u=[]
k=[]
w=[]
n=[]   
for link in productlinks:
    r =requests.get(link,headers=headers)
    soup=BeautifulSoup(r.content, 'html.parser')
    up = soup.find('div',class_='Product__SlideshowNavScroller')
    for e,pro in enumerate(up):
        t=pro.find('img').get('src')
        data.append({'image':'Image ' str(e) ' UI','link':t})
        dup = soup.find_all('div',class_='OptionSelector list-options')
        for ro in dup:
            m=[k.text.strip() for k in ro.find_all('button')]
            variant1=m[0]
            variant2=m[1]
            variant3=m[2]
            data.append({'image':'Image ' str(e) ' UI','link':t,'v1':variant1,'v2':variant2,'v3':variant3})

df = pd.DataFrame(data)
df.image=pd.Categorical(df.image,categories=df.image.unique(),ordered=True)
df = df.pivot(index=['v1','v2','v3'], columns='image', values='link').reset_index().fillna('')
df.to_csv('yt.csv')
```

uj5u.com熱心網友回復：

怎么了？

重復的原因之一是您將資訊附加到data您可能不需要的資訊：

data.append({'image':'Image ' str(e) ' UI','link':t})

因為你還附加：

data.append({'image':'Image ' str(e) 'UI','link':t,'v1':variant1,'v2':variant2,'v3':variant3}) 

但主要問題是您正在使用index=['v1','v2','v3']構建索引，但這不是仍然是unique.

怎么修？

1. 跳過這一行 data.append({'image':'Image ' str(e) ' UI','link':t})

2. 還添加id產品的data.append({'id':t.split('=')[-1],'image':'Image ' str(e) ' UI','link':t,'v1':variant1,'v2':variant2,'v3':variant3})

3. 使用id建立一個唯一索引index=['id','v1','v2','v3']

例子

import requests
from bs4 import BeautifulSoup
import pandas as pd
baseurl='https://twillmkt.com'
headers ={
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r =requests.get('https://twillmkt.com/collections/denim')
soup=BeautifulSoup(r.content, 'html.parser')
tra = soup.find_all('div',class_='ProductItem__Wrapper')
productlinks=[]
for links in tra:
    for link in links.find_all('a',href=True)[:1]:
        comp=baseurl link['href']
        productlinks.append(comp)

data=[]

for link in productlinks:
    r =requests.get(link,headers=headers)
    soup=BeautifulSoup(r.content, 'html.parser')
    up = soup.find('div',class_='Product__SlideshowNavScroller')
    for e,pro in enumerate(up):
        t=pro.find('img').get('src')
        dup = soup.find_all('div',class_='OptionSelector list-options')

        for ro in dup:
            m=[k.text.strip() for k in ro.find_all('button')]
            variant1=m[0]
            variant2=m[1]
            variant3=m[2]
            data.append({'id':t.split('=')[-1],'image':'Image ' str(e) ' UI','link':t,'v1':variant1,'v2':variant2,'v3':variant3})

df = pd.DataFrame(data)
df.image=pd.Categorical(df.image,categories=df.image.unique(),ordered=True)
df = df.pivot(index=['id','v1','v2','v3'], columns='image', values='link').reset_index().fillna('')
df

`

標籤：Python 熊猫 数据框 网页抓取 美汤

上一篇：將過濾器應用于資料后進行網路抓取

下一篇：如何從二進制檔案中正確讀取？