產品分頁資料抓取以獲取所有產品詳細資訊

我想抓取 URL = 'https://www.noon.com/uae-en/home-and-kitchen/home-decor/slipcovers/cushion-cover/' 的“靠墊套”類別的所有產品資料我分析了資料在腳本標簽中，但是如何從所有頁面中獲取資料。我需要所有頁面中所有產品的 URL，并且資料也在不同頁面的 API 中 API='https://www.noon.com/_next/data/B60DhzfamQWEpEl9Q8ajE/uae-en/home-and-kitchen /home-decor/slipcovers/cushion-cover.json?limit=50&page=2&sort[by]=popularity&sort[dir]=desc&catalog=home-and-kitchen&catalog=home-decor&catalog=slipcovers&catalog=cushion-cover'

import requests
import pandas as pd
import json
import csv
from lxml import html

headers ={'authority': 'www.noon.com',
      'accept' : 
'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
      'accept-encoding': 'gzip, deflate, br',
      'accept-language': 'en-US,en;q=0.9',
      }
 produrl = 'https://www.noon.com/uae-en/home-and-kitchen/home-decor/slipcovers/cushion-cover/'
 prodresp = requests.get(produrl, headers = headers, timeout =30)
 prodResphtml = html.fromstring(prodresp.text)
 print(prodresp)


 partjson = prodResphtml.xpath('//script[@id= "__NEXT_DATA__"]/text()')
 partjson = partjson[0]

uj5u.com熱心網友回復：

你即將達到你的目標。您可以使用拉出所有頁面來使下一頁意味著分頁for loop and range function，因為我們知道總頁碼為 192，這就是為什么我以這種健壯的方式進行分頁。因此，要從所有頁面中獲取所有產品url（或任何資料項），您可以按照下一個示例進行操作。

腳本：

import requests
import pandas as pd
import json
from lxml import html

headers ={
      'authority': 'www.noon.com',
      'accept' :'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
      'accept-encoding': 'gzip, deflate, br',
      'accept-language': 'en-US,en;q=0.9',
      }
produrl = 'https://www.noon.com/uae-en/home-and-kitchen/home-decor/slipcovers/cushion-cover/?limit=50&page={page}&sort[by]=popularity&sort[dir]=desc'
data=[]
for page in range(0,192):
    prodresp = requests.get(produrl.format(page=page), headers = headers, timeout =30)
    prodResphtml = html.fromstring(prodresp.text)
    #print(prodresp)

    partjson = prodResphtml.xpath('//script[@id= "__NEXT_DATA__"]/text()')
    #partjson = partjson[0]
    partjson = json.loads(partjson[0])
    #print(partjson)

    # with open('data.json','w',encoding='utf-8') as f:
    #     f.write(partjson)

    for item in partjson['props']['pageProps']['props']['catalog']['hits']:
        link='https://www.noon.com/' item['url']
        data.append(link)

df = pd.DataFrame(data,columns=['URL'])
#df.to_csv('product.csv',index=False)#to save data into your system
print(df)

輸出：

                     URL
0     https://www.noon.com/graphic-geometric-pattern...
1     https://www.noon.com/classic-nordic-decorative...
2     https://www.noon.com/embroidered-iconic-medusa...
3     https://www.noon.com/geometric-marble-texture-...
4     https://www.noon.com/traditional-damask-motif-...
...                                                 ...
9594  https://www.noon.com/geometric-printed-cushion...
9595  https://www.noon.com/chinese-style-art-printed...
9596  https://www.noon.com/chinese-style-art-printed...
9597  https://www.noon.com/chinese-style-art-printed...
9598  https://www.noon.com/chinese-style-art-printed...

[9599 rows x 1 columns]

uj5u.com熱心網友回復：

我使用了重新庫。換句話說，我使用了正則運算式，使用 JavaScript 抓取任何頁面要好得多

import requests
import pandas as pd
import json
import csv
from lxml import html
import re
headers ={
    'authority': 'www.noon.com',
    'accept' : 'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
      }
url = "https://www.noon.com/uae-en/home-and-kitchen/home-decor/slipcovers/cushion-cover/"
prodresp = requests.get(url, headers = headers, timeout =30)
jsonpage = re.findall(r'type="application/json">(.*?)</script>', prodresp.text)
jsonpage = json.loads(jsonpage[0])

標籤：网页抓取 lxml

上一篇：從R中的多個頁面中抓取評論

下一篇：在R中抓取評論