使用Selenium在Python中抓取多個頁面-回圈不起作用-有解無憂

我對 python 很陌生，并且使用 selenium 撰寫了一個腳本來抓取網站。我已經嘗試了一切，但無法讓回圈回圈瀏覽頁面。它目前只是將第一頁上的資料重復 5 次。我想為“BR1”抓取所有頁面，任何幫助都會很棒，目前下面的腳本只抓取第一頁 5 次。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

with open('rightmove.csv', 'w') as file:
    file.write('PropertyCardcontent \n')

PATH = ("/usr/local/bin/chromedriver")
driver = webdriver.Chrome(PATH)

driver.get("https://www.rightmove.co.uk/house-prices.html")
print(driver.title)

elem = driver.find_element(By.NAME, 'searchLocation')  # Find the search box
elem.send_keys('BR1'   Keys.RETURN)

try:
    content = WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.ID,'content'))
            )

finally:
    time.sleep(3)

for p in range(5):
    sold = content.find_elements(By.CLASS_NAME, 'sold-prices-content-wrapper ')
    for solds in sold:
        address = solds.find_elements(By.CLASS_NAME, 'sold-prices-content ')
        for addresses in address:
            result = addresses.find_elements(By.CLASS_NAME, 'results ')
            for results in result:
                card = results.find_elements(By.CLASS_NAME,'propertyCard')
                for propertyCard in card:
                    header = propertyCard.find_elements(By.CLASS_NAME,'propertyCard-content')
                    for propertyCardcontent in header:
                        road = propertyCardcontent.find_elements(By.CLASS_NAME,'title')
                    for propertyCardcontent in header:
                        road = propertyCardcontent.find_elements(By.CLASS_NAME,'subTitle')
                        for subtitle in road:
                            bed = subtitle.find_elements(By.CLASS_NAME, 'propertyType')
    with open('rightmove.csv', 'a') as file:
        for i in range(len(result)):
            file.write(header[i].text   '\n')
        
        button = driver.find_element(By.XPATH, '//*[@id="content"]/div[2]/div[2]/div[4]/div[27]/div[3]/div')
        button.click()
    file.close()

time.sleep(3)
driver.quit()

uj5u.com熱心網友回復：

您不必逐個逐個訪問 dom elem，您只需使用 xpath 或 class_name（如果它是唯一的，否則最好使用 xpath 或 css-selector）并獲取您正在尋找的專案。無論如何遵循這個：

import time
import selenium.webdriver as webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome("/usr/local/bin/chromedriver")
driver.get("https://www.rightmove.co.uk/house-prices.html")

# send query
query = "BR1"
search_bar = driver.find_element(By.XPATH, '//input[@]')
search_bar.send_keys(query)
search_bar.send_keys(Keys.ENTER)

# wait to result been loaded
WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'propertyCard'))
    )

#get amount of pages
pages = driver.find_element(By.XPATH, '//span[@][2]').text
pages = int(pages.replace('of ', ''))
data = []
i = 1

while i <= pages:
    WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, '//div[contains(text(), "Next")]'))
    ).click()
    # wait page load result
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, '//div//div[@]'))
    )
    propertyCards = driver.find_elements(By.XPATH, '//div//div[@]')
    # loop over result and store data
    for propertyCard in propertyCards:
        title = propertyCard.find_element(By.CLASS_NAME, 'title').text
        propertyType = propertyCard.find_element(By.CLASS_NAME, 'propertyType').text
        data.append((title, propertyType))
    time.sleep(1)
    i  = 1
    
print("you reach the last page")

#get number of results
printf(data)
driver.close()

我在您的示例中使用了一個元組串列，因為您想要存盤 2 個專案，如果您想要存盤更多資料，您可以使用 dict，然后直接使用 Dictwriter 轉換為 csv。享受。

uj5u.com熱心網友回復：

由于網站鏈接上有頁碼，我建議您將基本 url 設定為“https://www.rightmove.co.uk/house-prices/br1.html?page=1”，并在頁面中回圈使用格式字串等方法更改 url 的最后一個索引。

另一件事，您不需要實作所有這些 for 回圈，您可以簡單地將每個變數分配給它的特定值，因為您需要的一切都在一個易于導航的 html 塊中。

更新：

我很抱歉遲到了，有意想不到的東西（...）。

我在使用 Brave 時進行了一些更改，因此請確保您選擇瀏覽器，Chrome 我相信，chromedriver(ver:102) 保持不變（或取決于您的 Chrome 版本）。

我還得到了價格和日期并將它們存盤在一個元組中。每條記錄都存盤在一個串列中[Title, propertyType, tupleof(Price_Date)]

最后，它創建一個 csv 并用“;”將所有內容存盤在里面作為分隔符。

如果您愿意將價格和日期分開以供以后使用，您可以自行決定。

注意：此回圈方法僅適用于 URL 中包含頁數的網站。在這種情況下，鍵和頁碼都包含在 URL 中。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
import time
import random
import itertools


options = Options()
options.binary_location = r'C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe'
driver = webdriver.Chrome(options = options, service = Service("chromedriver.exe"))

key_word = "BR1".lower()
base_url = f"https://www.rightmove.co.uk/house-prices/{key_word}.html?page=1"
driver.get(base_url)

#Number of pages
pages = driver.find_element(By.XPATH, '//span[@][2]').text
pages = int(pages.strip('of'))


WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'results '))
    )

data = []
pc = 0
for p in range(1,pages 1):
    driver.get(f"https://www.rightmove.co.uk/house-prices/{key_word}.html?page={p}")

    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, '//div//div[@]'))
    )
    propertyCards = driver.find_elements(By.XPATH, '//div//div[@]')

    for propertyCard in propertyCards:
        title = propertyCard.find_element(By.CLASS_NAME, 'title').text
        propertyType = propertyCard.find_element(By.CLASS_NAME, 'propertyType').text        

        price_list = propertyCard.find_elements(By.CLASS_NAME, 'price')
        date_list = propertyCard.find_elements(By.CLASS_NAME, 'date-sold')

        data.append([title,propertyType])
        
        for p, d in itertools.zip_longest(price_list, date_list , fillvalue = None):
            try:
                price = p.text
                date = d.text
                data[pc].append((price, date))
            except Exception as e:
                print(e)
        pc =1
        
    time.sleep(random.randint(1,4))
print(data)

with open('rightmove.csv', 'w') as file:
    header = "Title;propertyType;Price_Date\n"
    file.write(header)
    for record in data:
        file.write("{};{};{}\n".format(record[0],record[1],record[2:]))

driver.quit()

轉載請註明出處，本文鏈接：https://www.uj5u.com/qiye/497457.html

標籤：Python 硒

上一篇：創建和編輯資料框

下一篇：在Javascript中，在將宣告的類作為引數傳遞給Object.getPrototypeOf時，結果參考父類。這是為什么？