我已經達到了個人里程碑scrapy。目的是正確理解callback和cb_kwargs，我已經無數次閱讀了檔案，但我通過可視化代碼、實踐和解釋來學習最好。
我有一個示例刮板，目的是獲取書名、價格并進入每個書頁并提取一條資訊。我也在嘗試了解如何正確獲取接下來幾頁的資訊，我知道這取決于對回呼操作的理解。
當我運行我的腳本時它只回傳第一頁的結果，我如何獲得其他頁面？
這是我的刮刀：
class BooksItem(scrapy.Item):
items = Field(output_processor = TakeFirst())
price = Field(output_processor = TakeFirst())
availability = Field(output_processor = TakeFirst())
class BookSpider(scrapy.Spider):
name = "books"
start_urls = ['https://books.toscrape.com']
def start_request(self):
for url in self.start_url:
yield scrapy.Request(
url,
callback = self.parse)
def parse(self, response):
data = response.xpath('//div[@class = "col-sm-8 col-md-9"]')
for books in data:
loader = ItemLoader(BooksItem(), selector = books)
loader.add_xpath('items','.//article[@]/h3/a//text()')
loader.add_xpath('price','.//p[@]//text()')
for url in [books.xpath('.//a//@href').get()]:
yield scrapy.Request(
response.urljoin(url),
callback = self.parse_book,
cb_kwargs = {'loader':loader})
for next_page in [response.xpath('.//div/ul[@]/li[@]/a//@href').get()]:
if next_page is not None:
yield response.follow(next_page, callback=self.parse)
def parse_book(self, response, loader):
book_quote = response.xpath('//p[@]//text()').get()
loader.add_value('availability', book_quote)
yield loader.load_item()
我相信問題出在我嘗試抓取接下來幾頁的部分。我已經嘗試使用以下替代方法：
def start_request(self):
for url in self.start_url:
yield scrapy.Request(
url,
callback = self.parse,
cb_kwargs = {'page_count':0}
)
def parse(self, response, next_page):
if page_count > 3:
return
...
...
page_count = 1
for next_page in [response.xpath('.//div/ul[@]/li[@]/a//@href').get()]:
yield response.follow(next_page, callback=self.parse, cb_kwargs = {'page_count': page_count})
但是，使用這種方法時出現以下錯誤：
型別錯誤：parse() 缺少 1 個必需的位置引數：'page_cntr'
uj5u.com熱心網友回復：
它應該是
start_requests, 和self.start_urls（在函式內部）。get()將回傳第一個結果，你想要的是getall()為了回傳一個串列。“next_page”部分不需要 for 回圈，這不是錯誤，只是不必要。
在這一行中，
for url in books.xpath您將每個 url 兩次獲取，同樣不是錯誤，但仍然......在這里，
data = response.xpath('//div[@class = "col-sm-8 col-md-9"]')您不是逐一選擇書籍，而是選擇整個書籍容器，您可以檢查len(data.getall()) == 1.book_quote = response.xpath('//p[@]//text()').get()將回傳\n，查看源代碼并嘗試找出原因（提示：'i' 標簽）。
將您的代碼與此進行比較，看看我更改了什么：
import scrapy
from scrapy import Field
from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst
class BooksItem(scrapy.Item):
items = Field(output_processor=TakeFirst())
price = Field(output_processor=TakeFirst())
availability = Field(output_processor=TakeFirst())
class BookSpider(scrapy.Spider):
name = "books"
start_urls = ['https://books.toscrape.com']
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url,
callback=self.parse)
def parse(self, response):
data = response.xpath('//div[@class = "col-sm-8 col-md-9"]//li')
for books in data:
loader = ItemLoader(BooksItem(), selector=books)
loader.add_xpath('items', './/article[@]/h3/a//text()')
loader.add_xpath('price', './/p[@]//text()')
for url in books.xpath('.//h3/a//@href').getall():
yield scrapy.Request(
response.urljoin(url),
callback=self.parse_book,
cb_kwargs={'loader': loader})
next_page = response.xpath('.//div/ul[@]/li[@]/a//@href').get()
if next_page:
yield response.follow(next_page, callback=self.parse)
def parse_book(self, response, loader):
# option 1:
book_quote = response.xpath('//p[@]/i/following-sibling::text()').get().strip()
# option 2:
# book_quote = ''.join(response.xpath('//div[contains(@class, "product_main")]//p[@]//text()').getall()).strip()
loader.add_value('availability', book_quote)
yield loader.load_item()
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/392387.html