出于學習的目的，我一直在嘗試遞回地抓取和抓取所有 URL https://triniate.com/images/，但 Scrapy 似乎只想抓取和抓取 TXT、HTML 和 PHP 的 URL。
這是我的蜘蛛代碼
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.selector import Selector
from HelloScrapy.items import PageInfoItem
class HelloSpider(CrawlSpider):
#Identifier when executing scrapy from CLI
name = 'hello'
#Domains that allow spiders to explore
allowed_domains = ["triniate.com"]
#Starting point(Start exploration)URL
start_urls = ["https://triniate.com/images/"]
#Specific rule with LinkExtractor argument(For example, scrape only pages that include new in the URL)Can be specified, but this time there is no argument because it targets all pages
#When you download a page that matches the Rule, the function specified in callback will be called.
#If follow is set to True, the search will be performed recursively.
rules = [Rule(LinkExtractor(), callback='parse_pageinfo', follow=True)]
def parse_pageinfo(self, response):
item = PageInfoItem()
item['URL'] = response.url
#Specify which part of the page to scrape
#In addition to specifying in xPath format, it is also possible to specify in CSS format
item['title'] = "idc"
return item
items.py 包含
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
from scrapy.item import Item, Field
class PageInfoItem(Item):
URL = Field()
title = Field()
pass
控制臺輸出是
2022-04-21 22:30:50 [scrapy.core.engine] INFO: Closing spider (finished)
2022-04-21 22:30:50 [scrapy.extensions.feedexport] INFO: Stored json feed (175 items) in: haxx.json
2022-04-21 22:30:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 59541,
'downloader/request_count': 176,
'downloader/request_method_count/GET': 176,
'downloader/response_bytes': 227394,
'downloader/response_count': 176,
'downloader/response_status_count/200': 176,
'dupefilter/filtered': 875,
'elapsed_time_seconds': 8.711563,
'feedexport/success_count/FileFeedStorage': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2022, 4, 22, 3, 30, 50, 142416),
'httpcompression/response_bytes': 402654,
'httpcompression/response_count': 175,
'item_scraped_count': 175,
'log_count/DEBUG': 357,
'log_count/INFO': 11,
'request_depth_max': 5,
'response_received_count': 176,
'scheduler/dequeued': 176,
'scheduler/dequeued/memory': 176,
'scheduler/enqueued': 176,
'scheduler/enqueued/memory': 176,
'start_time': datetime.datetime(2022, 4, 22, 3, 30, 41, 430853)}
2022-04-21 22:30:50 [scrapy.core.engine] INFO: Spider closed (finished)
有人可以建議我應該如何更改我的代碼以反映我想要的結果嗎？
編輯：為了澄清，我試圖獲取URL，而不是影像或檔案本身。
uj5u.com熱心網友回復：
為此，您需要了解 Scrapy 的作業原理。首先，您應該撰寫一個爬蟲來遞回地從根 URL 爬取所有目錄。在訪問頁面時提取所有影像鏈接。
所以我為你寫了這段代碼，并在你提供的網站上進行了測驗。它完美地作業。
import scrapy
class ImagesSpider(scrapy.Spider):
name = "images"
image_ext = ['png', 'gif']
images_urls = set()
start_urls = [
'https://triniate.com/images/',
# if there are some other urls you want to scrape the same way
# add them in this list
]
for url in start_urls:
yield scrapy.Request(url=url, callback=self.get_images)
def get_images(self, response):
all_hrefs = response.css('a::attr(href)').getall()
all_images_links = list(filter(lambda x: x.split('.')[-1] in self.image_ext, all_hrefs))
for link in all_images_links:
self.images_urls.add(link)
yield {'link': f'{response.request.url}{link}'}
next_page_links = list(filter(lambda x: x[-1]=='/', all_hrefs))
for link in next_page_links:
yield response.follow(link, callback=self.get_images)
因此，通過這種方式，您可以獲得此頁面上提供的所有影像的所有鏈接以及任何內部目錄（遞回）。
該get_images方法搜索頁面中的任何影像。它獲取所有影像鏈接，然后還放置任何目錄鏈接以進行爬網。所以它獲取所有目錄的所有影像鏈接。
我提供的代碼導致其中包含您想要的所有鏈接：
[
{"link": "https://triniate.com/images/ChatIcon.png"},
{"link": "https://triniate.com/images/Sprite1.gif"},
{"link": "https://triniate.com/images/a.png"},
...
...
...
{"link": "https://triniate.com/images/objects/house_objects/workbench.png"}
]
image_ext注意：我在屬性中指定了影像檔案的擴展名。您可以將它擴展到所有可用的影像擴展，或者像我一樣只包含網站中存在的擴展。你的選擇。
uj5u.com熱心網友回復：
我嘗試使用基本的蜘蛛和scrapy selenium。它有效。
基本的.py
import scrapy
from scrapy_selenium import SeleniumRequest
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
class BasicSpider(scrapy.Spider):
name = 'basic'
allowed_domains = ['triniate.com']
def start_requests(self):
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.set_window_size(1920, 1080)
driver.get("https://triniate.com/images/")
links = driver.find_elements(By.XPATH, "//html/body/table/tbody/tr/td[2]/a")
for link in links:
href= link.get_attribute('href')
yield SeleniumRequest(
url = href
)
driver.quit()
return super().start_requests()
def parse(self, response):
yield {
'URL': response.url
}
設定.py
添加
DOWNLOADER_MIDDLEWARES = {
'scrapy_selenium.SeleniumMiddleware': 800
}
輸出
2022-04-22 12:03:51 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/stand_right.gif>
{'URL': 'https://triniate.com/images/stand_right.gif'}
2022-04-22 12:03:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://triniate.com/images/walk_right_transparent.gif> (referer: None)
2022-04-22 12:03:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/walk_back.gif>
{'URL': 'https://triniate.com/images/walk_back.gif'}
2022-04-22 12:03:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/walk_left_transparent.gif>
{'URL': 'https://triniate.com/images/walk_left_transparent.gif'}
2022-04-22 12:03:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/walk_front_transparent.gif>
{'URL': 'https://triniate.com/images/walk_front_transparent.gif'}
2022-04-22 12:03:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/walk_back_transparent.gif>
{'URL': 'https://triniate.com/images/walk_back_transparent.gif'}
2022-04-22 12:03:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/walk_right.gif>
{'URL': 'https://triniate.com/images/walk_right.gif'}
2022-04-22 12:03:52 [scrapy.core.scraper] DEBUG: Scraped from <200 https://triniate.com/images/walk_right_transparent.gif>
{'URL': 'https://triniate.com/images/walk_right_transparent.gif'}
2022-04-22 12:03:52 [scrapy.core.engine] INFO: Closing spider (finished)
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/462929.html