在使用Selenium進行網頁抓取時需要幫助-有解無憂

代碼試驗：

from gettext import find
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
path="C:\Program Files (x86)\chromedriver.exe"
driver=webdriver.Chrome(path)
driver.get("https://targetstudy.com/school/state-board-schools-in-himachal-pradesh.html")
section=driver.find_element_by_class_name("section")
print(section.text)
driver.quit()

我能夠寫這個，但之后我無法提取所需的文本。看圖片我想要突出顯示的文本：

我要提取的這些突出顯示的文本

在這個excel格式中

我想要的excel格式

1 頁中有 25 個條目，因此我還必須在 Selenium 的幫助下單擊下一步按鈕。

uj5u.com熱心網友回復：

網址不是動態的。所以不需要使用硒。您可以使用 BeautifulSoup 提取所有資料。下面給出了一個示例和您的任務的其余部分。

import requests
from bs4 import BeautifulSoup

headers={'User-Agent':'mozilla/5.0'}
url='https://targetstudy.com/school/state-board-schools-in-himachal-pradesh.html'
req =requests.get(url,headers=headers)
print(req)  
soup = BeautifulSoup(req.content, 'lxml')
cards = soup.select('div.card-body')

for card in cards:
    try:
        school_name = card.select_one('.media-body a h4').text   
    except:
        pass
    try:
        address = card.select_one('p.card-subtitle.mt-0').get_text().split('Indiaphone')[0].replace('\n','').replace('\xa0','')    
    except:
        pass
    try:
        phone=card.select_one('p.card-subtitle.mt-0').get_text().split('Indiaphone')[1].replace('_iphone','').strip()
        
    except:
        pass

    print([school_name,address,phone])

輸出：

['Aadhar Public School', 'location_on Bir (Bagera)  Hamirpur - 176110, Himachal Pradesh, ', '9418125341']
['Aakash Model School', 'location_on Lahra Galore  Hamirpur - 177026, Himachal Pradesh, ', '(01972)-243201phone  9418118090']
['Aastha Public School - Banuti', 'location_on Banuti  Shimla - 171011, Himachal Pradesh, ', '(0177)-2802404phone  9418022024']
['Aastha Public School - Hatpang', 'location_on Hatpang  Kangra - 176022, Himachal Pradesh, ', '9459068853']
['Abhi Public School', 'location_on Goli (Dalhousie)  Chamba - 176305, Himachal Pradesh, ', '9418410844, 9418093564']
['Abhinav Vidya Mandir High School', 'location_on Haroli  Una - 177220, Himachal Pradesh, ', '(01975)-284209phone  9418537523']
['Abhishek Public High School', 'location_on Rait  Kangra - 176208, Himachal Pradesh, ', '(01892)-238524phone  9816297368']
['ACE Public School', 'location_on Sarah  Kangra - 176215, Himachal Pradesh, ', '9418427824'] 
['Adarsh Bal Jyoti Public School', 'location_on Beru Thona  Mandi - 175049, Himachal Pradesh, 
', '9857560105, 9817163967']
['Adarsh Bal Mandir High School', 'location_on Bhawarna  Kangra - 176083, Himachal Pradesh, ', '(01894)-247115']
['Adarsh Bal Vidya Mandir School', 'location_on Rajpur  Sirmaur - 173025, Himachal Pradesh, ', '9816235897, 9816208863']
['Adarsh Bharti Public School', 'location_on NH-154, Jassur  Kangra - 177201, Himachal Pradesh, ', '(01893)-226945phone  9418476945']
['Adarsh Bharti Public School', 'location_on Nagrota Surian  Kangra - 176027, Himachal Pradesh, ', '9816461493']
['Adarsh Bharti Public School', 'location_on Samkehar  Kangra - 176023, Himachal Pradesh, ', '9418013823']
['Adarsh Bharti Public School', 'location_on Village Dak & PO Dahab, Tehsil Nurpur  Kangra - 176051, Himachal Pradesh, ', '9418356123, 9816923352']
['Adarsh Bhartiya Model School', 'location_on Gharan (Bhojpur)  Kangra - 176402, Himachal Pradesh, ', '(01893)-245117phone  9816636653']
['Adarsh Gyan Public School', 'location_on Tatwali  Kangra - 176058, Himachal Pradesh, ', '9816789311, 9872242147']
['Adarsh High School', 'location_on Ghumarwin  Bilaspur - 174021, Himachal Pradesh, ', '(01978)-255605phone  9418042523']
['Adarsh High School', 'location_on Kuthera  Bilaspur - 174026, Himachal Pradesh, ', '(01978)-275690phone  9817050966']
['Adarsh Jyoti Primary School', ' Mandi - 175001, Himachal Pradesh, ', '9817567850']
['Adarsh Jyoti Public School', 'location_on Beru Thona  Mandi - 175049, Himachal Pradesh, ', '9857560105, 9817163967']
['Adarsh Model High School', 'location_on Chatta Khad  Kangra - 176025, Himachal Pradesh, ', '9816123223']
['Adarsh Model School', 'location_on Kariyara  Kangra - 177017, Himachal Pradesh, ', '9218949303, 9736590988']
['Adarsh Primary School', 'location_on Umari-Gander  Kangra - 176097, Himachal Pradesh, ', '9816123581']
['Adarsh Public School', 'location_on Mandrighat  Bilaspur - 174013, Himachal Pradesh, ', '(01907)-283019phone  9817304028, 9418059027']
['Adarsh Public School', 'location_on Mandrighat  Bilaspur - 174013, Himachal Pradesh, ', '(01907)-283019phone  9817304028, 9418059027']
['Adarsh Public School', 'location_on Mandrighat  Bilaspur - 174013, Himachal Pradesh, ', '(01907)-283019phone  9817304028, 9418059027']
['Adarsh Public School', 'location_on Mandrighat  Bilaspur - 174013, Himachal Pradesh, ', '(01907)-283019phone  9817304028, 9418059027']

uj5u.com熱心網友回復：

每個專案的所需文本位于以下元素內：

<div class="media-body">

解決方案

要提取所需的文本，您必須為visibility_of_all_elements_located()引入WebDriverWait，您可以使用以下任一定位器策略：

使用XPATH：

driver.get("https://targetstudy.com/school/state-board-schools-in-himachal-pradesh.html")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='media-body'][.//a/h4]")))])

控制臺輸出：

['Aadhar Public School\nlocation_on Bir (Bagera)\n  Hamirpur - 176110, Himachal Pradesh, India\nphone_iphone 9418125341', 'Aakash Model School\nlocation_on Lahra Galore\n  Hamirpur - 177026, Himachal Pradesh, India\nphone (01972)-243201\nphone_iphone 9418118090', 'Aastha Public School - Banuti\nlocation_on Banuti\n  Shimla - 171011, Himachal Pradesh, India\nphone (0177)-2802404\nphone_iphone 9418022024', 'Aastha Public School - Hatpang\nlocation_on Hatpang\n  Kangra - 176022, Himachal Pradesh, India\nphone_iphone 9459068853', 'Abhi Public School\nlocation_on Goli (Dalhousie)\n  Chamba - 176305, Himachal Pradesh, India\nphone_iphone 9418410844, 9418093564', 'Abhinav Vidya Mandir High School\nlocation_on Haroli\n  Una - 177220, Himachal Pradesh, India\nphone (01975)-284209\nphone_iphone 9418537523', 'Abhishek Public High School\nlocation_on Rait\n  Kangra - 176208, Himachal Pradesh, India\nphone (01892)-238524\nphone_iphone 9816297368', 'ACE Public School\nlocation_on Sarah\n  Kangra - 176215, Himachal Pradesh, India\nphone_iphone 9418427824', 'Adarsh Bal Jyoti Public School\nlocation_on Beru Thona\n  Mandi - 175049, Himachal Pradesh, India\nphone_iphone 9857560105, 9817163967', 'Adarsh Bal Mandir High School\nlocation_on Bhawarna\n  Kangra - 176083, Himachal Pradesh, India\nphone (01894)-247115', 'Adarsh Bal Vidya Mandir School\nlocation_on Rajpur\n  Sirmaur - 173025, Himachal Pradesh, India\nphone_iphone 9816235897, 9816208863', 'Adarsh Bharti Public School\nlocation_on NH-154, Jassur\n  Kangra - 177201, Himachal Pradesh, India\nphone (01893)-226945\nphone_iphone 9418476945', 'Adarsh Bharti Public School\nlocation_on Nagrota Surian\n  Kangra - 176027, Himachal Pradesh, India\nphone_iphone 9816461493', 'Adarsh Bharti Public School\nlocation_on Samkehar\n  Kangra - 176023, Himachal Pradesh, India\nphone_iphone 9418013823', 'Adarsh Bharti Public School\nlocation_on Village Dak & PO Dahab, Tehsil Nurpur\n  Kangra - 176051, Himachal Pradesh, India\nphone_iphone 9418356123, 9816923352', 'Adarsh Bhartiya Model School\nlocation_on Gharan (Bhojpur)\n  Kangra - 176402, Himachal Pradesh, India\nphone (01893)-245117\nphone_iphone 9816636653', 'Adarsh Gyan Public School\nlocation_on Tatwali\n  Kangra - 176058, Himachal Pradesh, India\nphone_iphone 9816789311, 9872242147', 'Adarsh High School\nlocation_on Ghumarwin\n  Bilaspur - 174021, Himachal Pradesh, India\nphone (01978)-255605\nphone_iphone 9418042523', 'Adarsh High School\nlocation_on Kuthera\n  Bilaspur - 174026, Himachal Pradesh, India\nphone (01978)-275690\nphone_iphone 9817050966', 'Adarsh Jyoti Primary School\n\n  Mandi - 175001, Himachal Pradesh, India\nphone_iphone 9817567850', 'Adarsh Jyoti Public School\nlocation_on Beru Thona\n  Mandi - 175049, Himachal Pradesh, India\nphone_iphone 9857560105, 9817163967', 'Adarsh Model High School\nlocation_on Chatta Khad\n  Kangra - 176025, Himachal Pradesh, India\nphone_iphone 9816123223', 'Adarsh Model School\nlocation_on Kariyara\n  Kangra - 177017, Himachal Pradesh, India\nphone_iphone 9218949303, 9736590988', 'Adarsh Primary School\nlocation_on Umari-Gander\n  Kangra - 176097, Himachal Pradesh, India\nphone_iphone 9816123581', 'Adarsh Public School\nlocation_on Mandrighat\n  Bilaspur - 174013, Himachal Pradesh, India\nphone (01907)-283019\nphone_iphone 9817304028, 9418059027']

注意：您必須添加以下匯入：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

uj5u.com熱心網友回復：

查找可以從當前網頁訪問下一個網頁的位置并使用

.click

在元素上

轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/466286.html

標籤：Python 硒网页抓取路径网络驱动程序等待

上一篇：seleniumpython中等待頁面上所有元素加載的命令是什么？

下一篇：使用Seleniumwebdriver創建新的python類