我之前做過一個使用 BeautifulSoup 抓取房地產網站的小專案，但是抓取大約 5,000 個資料點需要很長時間。我想學習多執行緒處理并用 BS 實作它，但有人告訴我用 Scrapy 爬網可能更快更容易。此外，我已經從使用 Spyder 切換到 Pycharm 作為我的 IDE。這仍然是令人不快的經歷，但我正在努力適應它。
我已經瀏覽過一次檔案，并使用 Scrapy 遵循了一些抓取示例，但我仍然遇到困難。我打算使用我之前創建的 BS 抓取腳本作為基礎，并創建一個新的 Scrapy 專案來抓取房地產資料。但是，我不知道如何以及從哪里開始。非常感謝任何和所有幫助。謝謝你。
預期結果：使用 Scrapy 從多個 URL 中抓取多個頁面。通過進入公寓串列鏈接并從每個值中獲取資料來抓取多個值。
Scrapy 腳本（到目前為止）：
# -*- coding: utf-8 -*-
# Import library
import scrapy
# Create Spider class
class UneguiApartmentSpider(scrapy.Spider):
name = 'apartments'
allowed_domains = ['www.unegui.mn']
start_urls = [
'https://www.unegui.mn/l-hdlh/l-hdlh-zarna/oron-suuts-zarna/'
]
# headers
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
}
def parse(self, response):
for listings in response.xpath("//div[@class='list-announcement']"):
item = ApartmentsItem()
item['name'] = listings.xpath('text()').extract()
item['link'] = listings.xpath('href').extract()
yield item
BeautifulSoup 腳本：
這個腳本仍然有一些我試圖解決的問題，比如抓取城市和價格。例如，對于 4 臥室公寓 url (/4-r/)，它會創建錯誤或空值，因為有 VIP 串列
# -*- coding: utf-8 -*-
import requests
from bs4 import BeautifulSoup as BS
from datetime import datetime, timedelta
from dateutil.relativedelta import relativedelta
from timeit import default_timer as timer
import pandas as pd
import re
import csv
dt_today = datetime.today()
date_today = dt_today.strftime('%Y-%m-%d')
date_today2 = dt_today.strftime('%Y%m%d')
date_yesterday = (dt_today-relativedelta(day=1)).strftime('%Y-%m-%d')
def main():
page = 0
name = []
date = []
address = []
district = []
city = []
price = []
area_sqm = []
rooms = []
floor = []
commission_year = []
building_floors = []
garage = []
balcony = []
windows = []
window_type = []
floor_type = []
door_type = []
leasing = []
description = []
link = []
for i in range (5,6):
BASE = 'https://www.unegui.mn'
URL = f'{BASE}/l-hdlh/l-hdlh-zarna/oron-suuts-zarna/{i}-r/?page='
COLUMNS=['Name','Date','Address','District','City','Price','Area_sqm','Rooms','Floor','Commission_year',
'Building_floors','Garage', 'Balcony','Windows','Window_type','Floor_type','door_type','Leasing','Description','Link']
with requests.Session() as session:
while True:
(r := session.get(f'{URL}{page 1}')).raise_for_status()
m = re.search('.*page=(\d )$', r.url)
if m and int(m.group(1)) == page:
break
page = 1
start = timer()
print(f'Scraping {i} bedroom apartments page {page}')
soup = BS(r.text, 'lxml')
for tag in soup.findAll('div', class_='list-announcement-block'):
_name = tag.find('a', attrs={'itemprop': 'name'})
name.append(_name.get('content', 'N/A'))
if (_link := _name.get('href', None)):
link.append(f'{BASE}{_link}')
(_r := session.get(link[-1])).raise_for_status()
_spanlist = BS(_r.text, 'lxml').find_all('span', class_='value-chars')
floor_type.append(_spanlist[0].get_text().strip())
balcony.append(_spanlist[1].get_text().strip())
garage.append(_spanlist[2].get_text().strip())
window_type.append(_spanlist[3].get_text().strip())
door_type.append(_spanlist[4].get_text().strip())
windows.append(_spanlist[5].get_text().strip())
_alist = BS(_r.text, 'lxml').find_all('a', class_='value-chars')
commission_year.append(_alist[0].get_text().strip())
building_floors.append(_alist[1].get_text().strip())
area_sqm.append(_alist[2].get_text().strip())
floor.append(_alist[3].get_text().strip())
leasing.append(_alist[4].get_text().strip())
district.append(_alist[5].get_text().strip())
address.append(_alist[6].get_text().strip())
rooms.append(tag.find('div', attrs={'announcement-block__breadcrumbs'}).get_text().split('?')[1].strip())
description.append(tag.find('div', class_='announcement-block__description').get_text().strip())
date.append(tag.find('div', class_='announcement-block__date').get_text().split(',')[0].strip())
city.append(tag.find('div', class_='announcement-block__date').get_text().split(',')[1].strip())
# if ( _price := tag.find('div', class_='announcement-block__price _premium')) is None:
# _price = tag.find('meta', attrs={'itemprop': 'price'})['content']
# price.append(_price)
end = timer()
print(timedelta(seconds=end-start))
df = pd.DataFrame(zip(name, date, address, district, city,
price, area_sqm, rooms, floor, commission_year,
building_floors, garage, balcony, windows, window_type,
floor_type, door_type, leasing, description, link), columns=COLUMNS)
return(df)
df['Date'] = df['Date'].replace('?н??д?р', date_today)
df['Date'] = df['Date'].replace('?чигд?р', date_yesterday)
df['Area_sqm'] = df['Area_sqm'].replace('м2', '')
df['Balcony'] = df['Balcony'].replace('тагттай', '')
if __name__ == '__main__':
df = main()
df.to_csv(f'{date_today2}HPD.csv', index=False)
uj5u.com熱心網友回復：
Scrapy 是一個異步回呼驅動的框架。
該parse()方法是對 all 的默認回呼start_urls。現在每個回呼都可以產生：
- item - 如果有，它將發送到管道并輸出
- request -
scrapy.Request可以產生物件以繼續抓取。
因此，如果您有多個頁面抓取器并且您想抓取所有專案，則您的邏輯將如下所示：
class MySpider(Spider):
name = "pagination_spider"
start_urls = ["first_page_url"]
def parse(self, response):
total_pages = ... # find total pages number in first page
for page in range(1, total_pages):
url = base_url page # form page url
yield scrapy.Request(url, callback=self.parse_page)
# also don't forget to parse first page
yield from self.parse_page(self, response)
def parse_page(self, response):
"""parse listing items from single pagination page"""
item = {} # parse page response for listing items
yield item
在這里，蜘蛛將請求第一頁，然后同時安排對所有剩余頁面的請求——這意味著您可以充分利用scrapy 的速度來獲取所有串列。
uj5u.com熱心網友回復：
這是抓取同一網站的多個 URL 的示例，例如該網站是 amazon 嬰兒類別的第一個 URL 第二個類別的第二個 URL
import time
import scrapy
from scrapy_splash import SplashRequest
from scrapy.crawler import CrawlerProcess
class spiders(scrapy.Spider):
name = "try"
start_urls = ["https://www.amazon.sg/gp/bestsellers/baby/ref=zg_bs_nav_0",'https://www.amazon.sg/gp/browse.html?node=6537678051&ref_=nav_em__home_appliances_0_2_4_4']
def parse(self, response):
for url in response.css('.mr-directory-item a::attr(href)').getall(): #loop for each href
yield scrapy.Request(f'https://muckrack.com{url}', callback=self.parse_products,
dont_filter=True)
def parse_products(self, response):
#these are for another website
full_name = response.css('.mr-font-family-2.top-none::text').get()
Media_outlet = response.css('.mr-person-job-item a::text').get()
yield {'Full Name': full_name, 'Media outlet':Media_outlet,'URL': response.url}
如果你想為每個 URL 執行不同的程序，你應該使用
import scrapy
from scrapy_splash import SplashRequest
from scrapy.crawler import CrawlerProcess
class spiders(scrapy.Spider):
name = "try"
def start_requests(self):
yield scrapy.Request('url1',callback=self.parse1)
yield scrapy.Request('url2',callback=self.parse2)
def parse1(self, response):
for url in response.css('.mr-directory-item a::attr(href)').getall():#loop for each href
yield scrapy.Request(f'https://muckrack.com{url}', callback=self.parse_products,
dont_filter=True)
def parse2(self, response):
for url in response.css('.mr-directory-item a::attr(href)').getall():#loop for each href
yield scrapy.Request(f'https://muckrack.com{url}', callback=self.parse_products,
dont_filter=True)
def parse_products(self, response):
#these are for another website
full_name = response.css('.mr-font-family-2.top-none::text').get()
Media_outlet = response.css('.mr-person-job-item a::text').get()
#yield {'header':'data'}
yield {'Full Name': full_name, 'Media outlet':Media_outlet,'URL': response.url}```
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/341150.html