Seleniumwebdriver回圈遍歷所有頁面，但只抓取第一頁的資料

我正在嘗試從網站上通過多個頁面 (36) 抓取資料，以收集每個可用檔案的檔案號和修訂號，并將其保存到兩個不同的串列中。如果我為每個單獨的頁面運行下面的代碼塊，它會完美運行。但是，當我添加 while 回圈以遍歷所有 36 頁時，它會回圈，但只保存第一頁的資料。

#sam.gov website
url = 'https://sam.gov/search/?index=sca&page=1&sort=-modifiedDate&pageSize=25&sfm[status][is_active]=true&sfm[wdPreviouslyPerformedWrapper][previouslyPeformed]=prevPerfNo/'

#webdriver
driver = webdriver.Chrome(options = options_, executable_path = r'C:/Users/439528/Python Scripts/Spyder/chromedriver.exe' )
driver.get(url)

#get rid of pop up window
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#sds-dialog-0 > button > usa-icon > i-bs > svg'))).click()

#list of revision numbers
revision_num = []

#empty list for all the WD links
WD_num = []
substring = '2015'

current_page = 0

while True:
    
    current_page  = 1
    if current_page == 36:
        #find all elements on page named "field name". For each one, get the text. if the text is 'Revision Date'
        #then, get the 'sibling' element, which is the actual revision number. append the date text to the revision_num list. 
        elements = driver.find_elements_by_class_name('sds-field__name')
        wd_links = driver.find_elements_by_class_name('usa-link')
        for i in elements:
            element = i.text
            if element == 'Revision Number':
                revision_numbers = i.find_elements_by_xpath("./following-sibling::div")
                
                for x in revision_numbers:
                    a = x.text
                    revision_num.append(a)
                    
            
        #finding all links that have the partial text 2015 and putting the wd text into the WD_num list
        for link in wd_links:
            wd = link.text
            if substring in wd:
                WD_num.append(wd)
            
        
        print('Last Page Complete!')
        break
             
    else:
        #find all elements on page named "field name". For each one, get the text. if the text is 'Revision Date'
        #then, get the 'sibling' element, which is the actual revision number. append the date text to the revision_num list. 
        elements = driver.find_elements_by_class_name('sds-field__name')
        wd_links = driver.find_elements_by_class_name('usa-link')
        for i in elements:
            element = i.text
            if element == 'Revision Number':
                revision_numbers = i.find_elements_by_xpath("./following-sibling::div")
                
                for x in revision_numbers:
                    a = x.text
                    revision_num.append(a)
                    
            

        #finding all links that have the partial text 2015 and putting the wd text into the WD_num list
        for link in wd_links:
            wd = link.text
            if substring in wd:
                WD_num.append(wd)
            
        #click on next page
        click_icon = WebDriverWait(driver, 5, 0.25).until(EC.visibility_of_element_located([By.ID,'bottomPagination-nextPage']))
        click_icon.click()
        WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.ID, 'main-container')))

我嘗試過的事情：

• 我添加了 WebDriverWait 以減慢腳本的速度以使頁面加載和/或可單擊/定位的元素
• 我在回圈外宣告了空串列，因此它不會覆寫每次迭代
• 我已經多次編輯了while回圈以計數到36（而current_page <37）或將計數器移動到回圈的頂部或底部）

有任何想法嗎？TIA。

編輯：添加了“欄位名稱”的螢屏截圖

uj5u.com熱心網友回復：

我已經重構了您的代碼并使事情變得非常簡單。

    driver = webdriver.Chrome(options = options_, executable_path = r'C:/Users/439528/Python Scripts/Spyder/chromedriver.exe' )
    
    revision_num = []
    WD_num = []
    
    for page in range(1,37):
        url = 'https://sam.gov/search/?index=sca&page={}&sort=-modifiedDate&pageSize=25&sfm[status][is_active]=true&sfm[wdPreviouslyPerformedWrapper][previouslyPeformed]=prevPerfNo/'.format(page)
        driver.get(url)
        if page==1:
            WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#sds-dialog-0 > button > usa-icon > i-bs > svg'))).click()
    
        
        elements =  WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH,"//a[contains(@class,'usa-link') and contains(.,'2015')]")))
        wd_links = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH,"//div[@class='sds-field__name' and text()='Revision Number']/following-sibling::div")))
        for element in elements:
            revision_num.append(element.text)
    
        for wd_link in wd_links:
            WD_num.append(wd_link.text)
    
    print(revision_num)
    print(WD_num)

1. 如果您只知道要迭代的 36 個頁面，則可以在 url 中傳遞該值。
2. 使用 webdriverwait 等待元素可見
3. 以這種方式構造您的 xpath，以便可以在沒有 if 的情況下唯一地識別元素，但是。

我的終端上的控制臺輸出：

標籤：Python 硒 循环 硒网络驱动程序 网页抓取

上一篇：BeautfiulSoup在抓取YouTubechanel時回傳空串列

下一篇：如何使用seleniumpython下載電子郵件附件？