我寫了這段代碼：
curl_command = "curl blah blah"
class MySpider(scrapy.Spider):
name = 'myspider'
allowed_domains = ['some_domain', ]
start_urls = ['someurl', ]
postal_codes = ['some_postal_code', ]
def start_requests(self):
for postal_code in self.postal_codes:
curl_req = scrapy.Request.from_curl(curl_command=curl_command)
curl_req._cb_kwargs = {'page': 0}
yield curl_req
def parse(self, response, **kwargs):
cur_page = kwargs.get('page', 1)
logging.info("Doing some logic")
num_pages = do_some_logic()
yield mySpiderItem
if cur_page < num_pages:
logging.info("New Request")
curl_req = scrapy.Request.from_curl(curl_command=curl_command)
curl_req._cb_kwargs = {'page': cur_page 1}
yield curl_req
yield scrapy.Request(url="https://jsonplaceholder.typicode.com/posts")
現在的問題是 parse 方法只被呼叫一次。換句話說，日志看起來像這樣：
Doing some logic
New Request
Spider closing
我不明白新請求發生了什么。從邏輯上講，新請求也應該導致Doing some logic日志，但由于某種原因它沒有。
我在這里錯過了什么嗎？還有其他方法可以產生新的請求嗎？
uj5u.com熱心網友回復：
從代碼示例中很難確切地知道問題出在哪里，但我想可能是您沒有在請求中使用頁碼。
例如，我修改了其他網站的代碼：
import scrapy
import logging
curl_command = 'curl "https://scrapingclub.com/exercise/list_basic/"'
class MySpider(scrapy.Spider):
name = 'myspider'
allowed_domains = ['scrapingclub.com']
#start_urls = ['someurl', ]
postal_codes = ['some_postal_code', ]
def start_requests(self):
for postal_code in self.postal_codes:
curl_req = scrapy.Request.from_curl(curl_command=curl_command, dont_filter=True)
curl_req._cb_kwargs = {'page': 1}
yield curl_req
def parse(self, response, **kwargs):
cur_page = kwargs.get('page', 1)
logging.info("Doing some logic")
#num_pages = do_some_logic()
#yield mySpiderItem
num_pages = 4
if cur_page < num_pages:
logging.info("New Request")
curl_req = scrapy.Request.from_curl(curl_command=f'{curl_command}?page={str(cur_page 1)}', dont_filter=True)
curl_req._cb_kwargs = {'page': cur_page 1}
yield curl_req
yield scrapy.Request(url="https://jsonplaceholder.typicode.com/posts")
輸出：
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://scrapingclub.com/exercise/list_basic/> (referer: None)
[root] INFO: Doing some logic
[root] INFO: New Request
[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'jsonplaceholder.typicode.com': <GET https://jsonplaceholder.typicode.com/posts>
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://scrapingclub.com/exercise/list_basic/?page=2> (referer: https://scrapingclub.com/exercise/list_basic/)
[root] INFO: Doing some logic
[root] INFO: New Request
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://scrapingclub.com/exercise/list_basic/?page=3> (referer: https://scrapingclub.com/exercise/list_basic/?page=2)
[root] INFO: Doing some logic
[root] INFO: New Request
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://scrapingclub.com/exercise/list_basic/?page=4> (referer: https://scrapingclub.com/exercise/list_basic/?page=3)
Scrapy 有一個默認啟用的內置重復過濾器。如果您不想要這種行為，您可以設定 'dont_filter = True' 以避免忽略重復請求。
uj5u.com熱心網友回復：
我想你忘記了請求中的回呼部分。檢查我從檔案中獲得的代碼。在你的情況下應該是 callback=self.parse
class MySpider(scrapy.Spider):
name = 'myspider'
def start_requests(self):
return [scrapy.FormRequest("http://www.example.com/login",
formdata={'user': 'john', 'pass': 'secret'},
callback=self.logged_in)]
def logged_in(self, response):
# here you would extract links to follow and return Requests for
# each of them, with another callback
pass
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/361676.html