我需要在出租房屋的 Zillow 上對加利福尼亞州的 3 頁進行網路抓取，并將所有資料放入熊貓資料框中。我需要提取每個串列的所有特征 - 地址、城市、臥室和浴室的數量、房子的大小、地塊的大小、建造年份、租金價格、租金日期
我的代碼：
from bs4 import BeautifulSoup
import requests
import time
import os
import random
import re
!pip install selenium
!pip install webdriver-manager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
import pandas as pd
import scipy as sc
import numpy as np
import sys
req_headers = {
'accept': 'text/html,application/xhtml xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.8',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951 Safari/537.36'
}
response = requests.get("https://www.zillow.com/homes/for_rent/CA/house_type/",headers=req_headers)
print(response)
soup = BeautifulSoup(response.content, 'html.parser')
print(soup.prettify())
listing_urls = []
listings = soup.find_all("article", {"class": "list-card list-card-additional-attribution list-card_not-saved"})
for listing in listings:
listing_url = listing.find("a")["href"]
print(listing_url)
listing_urls.append(listing_url)
我被困在這里 - 我收到以下錯誤：
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_24224/2055957203.py in <module>
4
5 for listing in listings:
----> 6 listing_url = listing.find("a")["href"]
7
8 print(listing_url)
TypeError: 'NoneType' object is not subscriptable
此外，代碼僅列印整個頁面的 2 個鏈接（每頁有 40 個待租房屋/公寓串列）
謝謝 ！:)
uj5u.com熱心網友回復：
資料通過 api 從外部源生成，并在 html 注釋下以 json 格式存盤在腳本中。所以你可以很容易地使用re or API.Here 我使用re模塊提取所有資料
import requests
import re
import json
r = requests.get('https://www.zillow.com/ca/rentals/2_p/?searchQueryState={"pagination":{"currentPage":2},"usersSearchTerm":"CA","mapBounds":{"west":-150.332039625,"east":-96.19141462500001,"south":21.05981671373876,"north":54.12119640028577},"regionSelection":[{"regionId":9,"regionType":2}],"isMapVisible":false,"filterState":{"fsba":{"value":false},"fsbo":{"value":false},"nc":{"value":false},"fore":{"value":false},"cmsn":{"value":false},"auc":{"value":false},"fr":{"value":true},"ah":{"value":true},"mf":{"value":false},"manu":{"value":false},"land":{"value":false}},"isListVisible":true,"mapZoom":4}',headers = {'User-Agent':'Mozilla/5.0'})
data = json.loads(re.search(r'!--(\{"queryState".*?)-->', r.text).group(1))
for item in data['cat1']['searchResults']['listResults']:
listing_url= 'https://www.zillow.com' item['detailUrl']
print(listing_url)
輸出：
https://www.zillow.com/b/morgan-park-vacaville-ca-5XjWBr/
https://www.zillow.com/b/renaissance-park-apartments-davis-ca-5XjQ6D/
https://www.zillow.com/b/alivia-apartments-whittier-ca-BwXXhk/
https://www.zillow.comhttps://www.zillow.com/homedetails/7845-Stewart-And-Gray-Rd-783498CC7-Downey-CA-90241/2065561183_zpid/
https://www.zillow.com/b/admirals-cove-alameda-ca-BPfvbS/
https://www.zillow.com/b/parkside-union-city-ca-5XjLjS/
https://www.zillow.com/b/coventry-park-roseville-ca-5XjQjF/
https://www.zillow.com/b/capitol-yards-west-sacramento-ca-5hHzYJ/
https://www.zillow.comhttps://www.zillow.com/homedetails/8950-Arrow-Rte-APT-145-Rancho-Cucamonga-CA-91730/2063289079_zpid/
https://www.zillow.com/b/twin-palms-apartments-san-jose-ca-5ZsGBN/
https://www.zillow.com/b/pavona-apartments-san-jose-ca-5XjTRx/
https://www.zillow.com/b/imt-pleasanton-pleasanton-ca-5XkG5W/
https://www.zillow.comhttps://www.zillow.com/homedetails/1516-Sylvan-Way-8328CE707-Lodi-CA-95242/2066702720_zpid/
https://www.zillow.com/b/vineyard-terrace-apartments-napa-ca-5XjN2m/
https://www.zillow.com/b/autumn-springs-livermore-ca-5XjRGt/
https://www.zillow.com/b/tamarack-woods-apartment-homes-brea-ca-5XjSZJ/
https://www.zillow.com/b/serrano-apartments-west-covina-ca-5XnMXG/
https://www.zillow.com/b/castlerock-riverside-ca-5XkJLW/
https://www.zillow.com/b/lasselle-place-moreno-valley-ca-5XjThf/
https://www.zillow.com/b/the-crossing-at-arroyo-trail-livermore-ca-5XjR44/
https://www.zillow.com/b/canyon-park-riverside-ca-5XjVV8/
https://www.zillow.com/b/55+-community-fountainglen-grand-isle-murrieta-ca-5XjT3M/
https://www.zillow.com/b/calavo-woods-apartments-spring-valley-ca-5XjSpC/
https://www.zillow.com/b/axiom-tustin-tustin-ca-BWTZgR/
https://www.zillow.com/b/colonnade-riverside-ca-5XjxP7/
https://www.zillow.com/b/pinnacle-lemoli-hawthorne-ca-5j35wR/
https://www.zillow.com/b/verge-san-diego-ca-5hJ5ML/
https://www.zillow.com/b/palmia,-age-55+-luxury-apartments-fremont-ca-C2vgyw/https://www.zillow.com/b/sage-creek-south-simi-valley-ca-5XtP4x/
https://www.zillow.com/b/dublin-station-by-windsor-dublin-ca-63Pjjc/
https://www.zillow.com/b/wilshire-margot-los-angeles-ca-5XjVmT/
https://www.zillow.com/b/carefree-north-natomas-senior-apartments-sacramento-ca-5XjW9V/
https://www.zillow.com/b/tradition-carlsbad-ca-5XjQWM/
https://www.zillow.com/b/sherwood-at-iron-point-folsom-ca-5XjT5w/
https://www.zillow.comhttps://www.zillow.com/homedetails/139-Santa-Rosa-Dr-00749DD7E-Oceanside-CA-92058/2066712796_zpid/
https://www.zillow.com/b/casa-madrid-anaheim-ca-5XjKvK/
https://www.zillow.com/b/metro-gateway-riverside-ca-B8GcqC/
https://www.zillow.comhttps://www.zillow.com/homedetails/18427-Studebaker-Rd-C99807DD9-Cerritos-CA-90703/2066760882_zpid/
https://www.zillow.com/b/the-vineyard-luxury-apartments-petaluma-ca-5Xj7CH/
https://www.zillow.com/b/renew-one59-san-leandro-ca-5XjxJb/
使用 selenium，您將永遠無法從每頁中提取 40 個專案，因為它們受到評論的限制，并且只會得到 8 個專案。
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get('https://www.zillow.com/homes/for_rent/CA/house_type/')
time.sleep(5)
driver.maximize_window()
soup = BeautifulSoup(driver.page_source, 'html.parser')
info_list = soup.select('div.list-card-info > a')
for link in info_list:
get_url = link['href']
print(get_url)
輸出：
https://www.zillow.com/homedetails/176-W-Festivo-Ln-Mountain-House-CA-95391/69031126_zpid/
https://www.zillow.com/homedetails/1967-Linden-Ln-Milpitas-CA-95035/19473336_zpid/
https://www.zillow.com/homedetails/3341-Idlewild-Way-San-Diego-CA-92117/2086677504_zpid/
/b/960-market-st-san-francisco-ca-9NJsbZ/
https://www.zillow.com/homedetails/2915-Tuna-Canyon-Rd-Topanga-CA-90290/95700373_zpid/
https://www.zillow.com/homedetails/7332-Pacific-View-Dr-Los-Angeles-CA-90068/20045509_zpid/
https://www.zillow.com/homedetails/2267-Filbert-St-San-Francisco-CA-94123/2063204947_zpid/
https://www.zillow.com/homedetails/28145-Via-Princesa-APT-8-Murrieta-CA-92563/2082116207_zpid/https://www.zillow.com/homedetails/312-Birkdale-Way-Bakersfield-CA-93309/18957251_zpid/
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/479399.html
上一篇：多選中的硒選擇選項