PYTHONscrapyseleniumWebDriverWait

這里的專家，如果您不介意，我正在尋求您的幫助。

最近，我正在開發一個在 python 中使用 scrapy 和 selenium 的網路爬蟲。我的心已經碎了。

我只是想問你是否有可能即使你使用了陳述句你仍然會空著

WebDriverWait(driver, 100, 0.1).until(EC.presence_of_all_elements_located((By.XPATH,xxxxx)))

獲取這些元素。而且，它甚至不需要 100 秒才能變空。為什么？

順便說一句，這是一件隨機的事情，這意味著這種現象隨時隨地都會發生。

變空是否與我的網路連接有關？

您能幫我或就上述問題給我一些意見和建議嗎？

非常感謝！

-----------------------補充說明-----------

感謝您的提醒。

總而言之，我使用scrapyandselenium爬取了一個關于評論的網站，并將用戶名、發布時間、評論內容等寫入.xlsx filevia pipeline.py，我希望它在收集完整資訊的同時盡可能快。

一個有很多人評論的頁面，由于評論文本太長而被擱置，這意味著每頁有近 20 條評論有他們的展開按鈕。

因此，我需要使用seleniumtoclick the expand button然后使用驅動程式來獲取完整的評論。常識規定點擊展開按鈕后加載需要一點時間，我相信需要的時間取決于網路的速度。所以WebDriverWait在這里使用似乎是一個明智的選擇。經過我的實踐，默認引數timeout=10和poll_frequency=0.5似乎太慢而且容易出錯。所以我考慮使用 和 的timeout=100規格poll_frequency=0.1。

但是，問題是每次通過cmd陳述句跑專案scrapy crawl spider，總是有好幾個評論爬取為空，而且每次空的位置都不一樣。我曾考慮過使用time.sleep()強制停止，但如果每個頁面都這樣做，那將花費大量時間，而且它肯定是獲取完整資訊的更有用的方式。此外，在我看來，它看起來并不那么優雅而且有點笨拙。

我是否清楚地表達了我的問題？

- - - - - - - - - - - - - - - -添點什么 - - - - - - - - - ---------------

的確切含義 I got somwhere empty如下圖所示。

---------------------------------------添加我的代碼-------------------- ------2022/5/18

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get('https://movie.douban.com/subject/5045678/reviews?start=0')

users = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'a[class=name]')]
dates = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'span[class=main-meta]')]
full_content, words = [], []
unfolds = WebDriverWait(driver,100,0.1).until(EC.presence_of_all_elements_located((By.XPATH,"//a[@class='unfold']")))

# Here's how I think about and design my loop body.
# I click the expansion bottun, then grab the text, then put it away, then move on to the next one.
for i in range(len(unfolds)):
    unfolds[i].click()
    time.sleep(1)
    # After the javascript, the `div[@class='review-content clearfix']` appear,
    # and some of the full review content will be put in a `<p></p>` label
    find_full_content_p = WebDriverWait(driver,100,0.1).until(EC.presence_of_all_elements_located((By.XPATH,"//div[@class='review-content clearfix']/p")))
    full_content_p = [j.text for j in find_full_content_p]
    # and some of them will just put in `div[@class='review-content clearfix']` itself.
    find_full_content_div = WebDriverWait(driver,100,0.1).until(EC.presence_of_all_elements_located((By.XPATH,"//div[@class='review-content clearfix']")))
    full_content_div = [j.text for j in find_full_content_div]
    
    # and I made a list merge
    full_content_p.extend(full_content_div)
    full_content.append("".join(full_content_p))
    words.append(len("".join(full_content_p)))
    time.sleep(1)
    
    # then put it away
    WebDriverWait(driver,100,0.1).until(EC.element_to_be_clickable((By.XPATH,"//a[@class='fold']"))).click()
driver.close()
pd.DataFrame({"users":users, "dates":dates, "full_content":full_content, "words":words})

而且，這是我真正尊重的名為聲波的專家的代碼。（這里稍作修改，核心代碼沒變）

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
# from selenium.webdriver.chrome.service import Service
    
driver = webdriver.Chrome()

driver.get('https://movie.douban.com/subject/5045678/reviews?start=0')

users = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'a[class=name]')]
dates = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'span[class=main-meta]')]
reviews, words = [], []
for review in driver.find_elements(By.CSS_SELECTOR, 'div.review-short'):
    show_more = review.find_elements(By.CSS_SELECTOR, 'a.unfold')
    if show_more:
        # scroll to the show more button, needed to avoid ElementClickInterceptedException
        driver.execute_script('arguments[0].scrollIntoView({block: "center"});', show_more[0])
        show_more[0].click()
        review = review.find_element(By.XPATH, 'following-sibling::div')
        while review.get_attribute('class') == 'hidden':
            time.sleep(0.2)
        review = review.find_element(By.CSS_SELECTOR, 'div.review-content')
    reviews.append(review.text)
    words.append(len(review.text))
    print('done',len(reviews),end='\r')
 pd.DataFrame({"users":users,"dates":dates,"reviews":reviews,"words":words})

uj5u.com熱心網友回復：

新的

為網站添加了代碼douban。要將抓取的資料匯出到 csv，請參閱下面 OLD 部分中的 pandas 代碼

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
    
driver = webdriver.Chrome(service=Service('...'))

driver.get('https://movie.douban.com/subject/5045678/reviews?start=0')

users = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'a[class=name]')]
dates = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'span[class=main-meta]')]
reviews = []
for review in driver.find_elements(By.CSS_SELECTOR, 'div.review-short'):
    show_more = review.find_elements(By.CSS_SELECTOR, 'a.unfold')
    if show_more:
        # scroll to the show more button, needed to avoid ElementClickInterceptedException
        driver.execute_script('arguments[0].scrollIntoView({block: "center"});', show_more[0])
        show_more[0].click()
        review = review.find_element(By.XPATH, 'following-sibling::div')
        while review.get_attribute('class') == 'hidden':
            time.sleep(0.2)
        review = review.find_element(By.CSS_SELECTOR, 'div.review-content')
    reviews.append(review.text)
    print('done',len(reviews),end='\r')

老的

對于您提到的網站（imdb.com），為了抓取隱藏的內容，無需單擊顯示更多按鈕，因為文本已經加載到 HTML 代碼中，只是沒有顯示在網站上。因此，您可以一次抓取所有評論。下面的代碼將用戶、日期和評論存盤在單獨的串列中，最后將資料保存到.csv檔案中。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
  
driver = webdriver.Chrome(service=Service(chromedriver_path))

driver.get('https://www.imdb.com/title/tt1683526/reviews')
# sets a maximum waiting time for .find_element() and similar commands
driver.implicitly_wait(10)

reviews = [el.get_attribute('innerText') for el in driver.find_elements(By.CSS_SELECTOR, 'div.text')]
users = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'span.display-name-link')]
dates = [el.text for el in driver.find_elements(By.CSS_SELECTOR, 'span.review-date')]

# store data in a csv file
import pandas as pd
df = pd.DataFrame(list(zip(users,dates,reviews)), columns=['user','date','review'])
df.to_csv(r'C:\Users\your_name\Desktop\data.csv', index=False)

要列印單個評論，您可以執行以下操作

i = 0
print(f'User: {users[i]}\nDate: {dates[i]}\n{reviews[i]}')

輸出（截斷）是

User: dschmeding
Date: 26 February 2012
Wow! I was not expecting this movie to be this engaging. Its one of those films...

標籤：Python 硒 刮擦 网络驱动程序等待

上一篇：使用具有奇怪屬性的xpath獲取按鈕

下一篇：如何將字串轉換為ruby中的物件參考