我嘗試使用scrapy抓取以下網站-
當我只從概覽頁面（如名稱、價格、鏈接）中抓取資訊時，它作業正常，它回傳 1535 行。
import scrapy
class WhiskeySpider(scrapy.Spider):
name = "whisky"
allowed_domains = ["whiskyshop.com"]
start_urls = ["https://www.whiskyshop.com/scotch-whisky"]
def parse(self, response):
for products in response.css("div.product-item-info"):
tmpPrice = products.css("span.price::text").get()
if tmpPrice == None:
tmpPrice = "Sold Out"
else:
tmpPrice = tmpPrice.replace("\u00a3",""),
yield {
"name": products.css("a.product-item-link::text").get(),
"price": tmpPrice,
"link": products.css("a.product-item-link").attrib["href"],
}
nextPage = response.css("a.action.next").attrib["href"]
if nextPage != None:
nextPage = response.urljoin(nextPage)
yield response.follow(nextPage, callback=self.parse)
現在我還想為每個專案抓取一些額外的詳細資訊（如升、百分比、面積），我希望有一行包含 3 個主要資訊和 3 個詳細資訊
我使用以下代碼進行了嘗試 - 但這效果不佳：
import scrapy
class WhiskeySpider(scrapy.Spider):
name = "whiskyDetail"
allowed_domains = ["whiskyshop.com"]
start_urls = ["https://www.whiskyshop.com/scotch-whisky"]
def parse(self, response):
for products in response.css("div.product-item-info"):
tmpPrice = products.css("span.price::text").get()
tmpLink = products.css("a.product-item-link").attrib["href"]
tmpLink = response.urljoin(tmpLink)
if tmpPrice == None:
tmpPrice = "Sold Out"
else:
tmpPrice = tmpPrice.replace("\u00a3",""),
yield {
"name": products.css("a.product-item-link::text").get(),
"price": tmpPrice,
"link": tmpLink,
}
yield scrapy.Request(url=tmpLink, callback=self.parseDetails)
nextPage = response.css("a.action.next").attrib["href"]
if nextPage != None:
nextPage = response.urljoin(nextPage)
yield response.follow(nextPage, callback=self.parse)
def parseDetails(self, response):
tmpDetails = response.css("p.product-info-size-abv span::text").getall()
yield {
"litre": tmpDetails[0],
"percent": tmpDetails[1],
"area": tmpDetails[2]
}
代碼似乎在無限回圈中運行 在日志中，我看到他有時以 429 未知狀態重試
2021-11-05 22:24:14 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.whiskyshop.com/benrinnes-10-year-old-that-boutique-y-whisky-company>
{'litre': '50cl', 'percent': '56.8% abv', 'area': 'Speyside'}
2021-11-05 22:24:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.whiskyshop.com/bruichladdich-28-year-old-batch-19-that-boutique-y-whisky-company> (referer: https://www.whiskyshop.com/scotch-whisky)
2021-11-05 22:24:17 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.whiskyshop.com/bruichladdich-28-year-old-batch-19-that-boutique-y-whisky-company>
{'litre': '50cl', 'percent': '48.5%% abv', 'area': 'Islay'}
2021-11-05 22:24:17 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.whiskyshop.com/westport-21-year-old-batch-1-that-boutique-y-whisky-company> (failed 1 times): 429 Unknown Status
2021-11-05 22:24:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.whiskyshop.com/strathmill-22-year-old-batch-7-that-boutique-y-whisky-company> (referer: https://www.whiskyshop.com/scotch-whisky)
2021-11-05 22:24:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.whiskyshop.com/strathmill-22-year-old-batch-7-that-boutique-y-whisky-company>
{'litre': '50cl', 'percent': '49.6% abv', 'area': 'Speyside'}
2021-11-05 22:24:20 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.whiskyshop.com/benromach-40-year-old> (failed 1 times): 429 Unknown Status
2021-11-05 22:24:22 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.whiskyshop.com/monkey-shoulder-fever-tree-gift-pack> (failed 1 times): 429 Unknown Status
2021-11-05 22:24:25 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.whiskyshop.com/catalog/product/view/id/21965/s/nc-nean-organic-single-malt/category/246/> (failed 1 times): 429 Unknown Status
在 json-output 中，兩個資訊都不在一行（主要和詳細資訊）：
{"name": "Port Charlotte Islay Barley 2013 ", "price": ["65.00"], "link": "https://www.whiskyshop.com/port-charlotte-islay-barley-2013"},
{"name": "Bruichladdich Bere Barley 2011 ", "price": ["70.00"], "link": "https://www.whiskyshop.com/bruichladdich-bere-barley-2011"},
{"name": "Glen Grant 1950 68 Year Old ", "price": ["4,999.99"], "link": "https://www.whiskyshop.com/glen-grant-1950-68-year-old"},
{"name": "Linkwood 1981 Private Collection ", "price": ["1,250.00"], "link": "https://www.whiskyshop.com/linkwood-1981-private-collection"},
{"name": "Linkwood 1980 40 Year Old Private Collection ", "price": ["999.99"], "link": "https://www.whiskyshop.com/linkwood-1980-40-year-old-private-collection"},
{"name": "Dimensions Linkwood 2009 12 Year Old", "price": ["89.99"], "link": "https://www.whiskyshop.com/dimensions-linkwood-2009-12-year-old"},
{"name": "Dimensions Highland Park 2007 13 Year Old", "price": ["114.00"], "link": "https://www.whiskyshop.com/dimensions-highland-park-2007-13-year-old"},
{"litre": "70cl", "percent": "54.9% abv", "area": "Highland"},
{"litre": "70cl", "percent": "54.7% abv", "area": "Islay"},
{"litre": "70cl", "percent": "46% abv", "area": "Highland"},
{"litre": "70cl", "percent": "52.1% abv", "area": "Islay"},
{"litre": "70cl", "percent": "43% abv", "area": "Speyside"},
{"litre": "70cl", "percent": "43% abv", "area": "Highland"},
我做錯了什么，如何在一行中獲取主要和詳細資訊？（并且沒有重試錯誤）
uj5u.com熱心網友回復：
您必須下載延遲，否則會被 429 狀態代碼/重試/連接丟失等阻止。我的 settings.py 檔案：
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36'
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 4
在概覽頁面和詳細資訊頁面的情況下獲取資料的替代和最簡單的解決方案是使用CrawlSpider.And 我在 start_urls 中進行了分頁，您可以根據需要增加或減少頁碼范圍。這里每頁??包含 100 個專案。
代碼：
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class ShopSpider(CrawlSpider):
name = 'shop'
start_urls = ['https://www.whiskyshop.com/scotch-whisky?p=' str(x) ''for x in range(1,5)]
rules = (
Rule(LinkExtractor(restrict_xpaths='//a[@]'),callback='parse', follow=False),)
def parse(self, response):
yield {
'Name': response.xpath('//h1[@]/text()').get().strip(),
'Price':response.xpath('(//span[@])[1]/text()').get(),
'Litre':response.xpath('(//*[@]/span)[1]/text()').get(),
'Percent':response.xpath('(//*[@]/span)[2]/text()').get(),
'Area':response.xpath('(//*[@]/span)[3]/text()').get(),
'LINK': response.url}
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/351282.html