我有一些代碼決議動態網頁的特定部分。早些時候我用刮擦劑做這件事，它作業得很好，但由于它使用的信用系統，我決定轉向硒。但是，當我嘗試使用 selenium 運行我的代碼時，它并沒有獲取動態生成的 HTML 內容，而是給了我非瀏覽器請求獲得的無用 HTML，因此它沒有找到我的代碼在使用 srapingant 之前找到的任何元素. 有任何想法嗎？我的代碼如下
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url='https://www.mesitis.com.cy/Search.aspx?&isAsc=0&isRent=0&districts=Lefkosia&status=1&apartFloor=&types=0dfe6c47-04c5-e511-ae61-a4badb3ceace&refno=&priceFrom=170000&priceTo=260000&priceRentFrom=0&priceRentTo=10000&areaFrom=0&areaTo=1000&intAreaFrom=130&intAreaTo=500&densityFrom=0&densityTo=200¤tPage=1'
# instance of Options class allows us to configure Headless Chrome
options = Options()
# this parameter tells Chrome that it should be run without UI (Headless)
options.headless = True
# initializing webdriver for Chrome with our options
driver = webdriver.Chrome(options=options, executable_path='C:\Downloads\chromedriver_win32\chromedriver.exe')
# getting webpage - this is where something seems to be going wrong, and I don't get the dynamically generated HTML content
driver.get(url)
page_source=driver.page_source
soup = BeautifulSoup(page_source,"html.parser")
body=soup.find('body')
properties=body.find('div', attrs={'class':'row properties-list'})
#find links of each property
allinks=properties.find_all('h3')
for d in allinks:
print('https://www.mesitis.com.cy/' d.a.get('href'))
uj5u.com熱心網友回復：
這是獲取該資料的一種方法，無需 Selenium 的開銷：僅使用請求即可：
import requests
import pandas as pd
import json
url = 'https://www.mesitis.com.cy/Search.aspx/SearchPropertiesFast'
big_df = pd.DataFrame()
headers = {
'origin': 'https://www.mesitis.com.cy',
'content-type': 'application/json; charset=UTF-8',
'sec-fetch-site': 'same-origin',
'x-requested-with': 'XMLHttpRequest',
'accept-language': 'en-US,en;q=0.9',
'accept': 'application/json, text/javascript, */*; q=0.01',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36',
}
payload = '{"formVars":[{"name":"__VIEWSTATE","value":"qDd5lhcP4H3kN2jINvhRjt92XZj6ix9IaKWrw9oIHKjw6zeDrrw2zWsQbGcDIZB/6U buWRwKUTyFye1K8vyTDeTj8MbT xSputXV4722aOhy672wwqzKKKDppNXgYLEZoM1aO0tlGNuGZrJ1EKT30FF2345a6oUPfKdrzHPyWAzxoceJ2/qFVuGQXMjAc/i"},{"name":"__VIEWSTATEGENERATOR","value":"BBBC20B8"},{"name":"__EVENTVALIDATION","value":"/8kjpsksY0pg1DuYQRFVE06InIT4LHDuZT8OFTY/pxYB3czd0Ukmi9thbd9XYDPNRXLkdm oJzu0U6g3bqVMZU5Azgj6C2V/LyMKsLq4tYuQ KE/GKWokA74mPf53qg 8tJ75T92dXShsE32JMlhiQ=="},{"name":"isAsc","value":"0"},{"name":"isRent","value":"0"},{"name":"districts","value":"Lefkosia"},{"name":"status","value":"1"},{"name":"apartFloor","value":""},{"name":"types","value":"0dfe6c47-04c5-e511-ae61-a4badb3ceace"},{"name":"refno","value":""},{"name":"priceFrom","value":170000},{"name":"priceTo","value":260000},{"name":"priceRentFrom","value":0},{"name":"priceRentTo","value":10000},{"name":"areaFrom","value":0},{"name":"areaTo","value":1000},{"name":"intAreaFrom","value":130},{"name":"intAreaTo","value":500},{"name":"densityFrom","value":0},{"name":"densityTo","value":200},{"name":"currentPage","value":1}]}'
r = requests.post(url, data = payload, headers=headers)
data_obj = json.loads(r.json()['d'])
df = pd.json_normalize(data_obj, record_path=['DataObject'])
df.to_csv('houses.csv')
print(df)
終端列印的結果：
Price id ReferenceNo Title Area Type Sold IsRent PlotArea ProjectID ... LastUpdated propImageID IsProject District CoveredArea VAT IsNew PropertyId ApartFloor EnergyEfficiency
0 170000.0 cf241874-a168-ec11-aa4c-246e960cbc4d 16365 2 Bedroom Old House For Sale In Kaimakli Kaimakli House/Villa 0 False 270 None ... 2022-06-17T06:55:26.247 00000000-0000-0000-0000-000000000000 0 Lefkosia 0 0 ce241874-a168-ec11-aa4c-246e960cbc4d None 20
1 190000.0 cd1a0b90-07aa-ec11-aa4c-246e960cbc4d 18001 3 Bedroom House For Sale In Anthoupoli, Nicosia Anthoupoli House/Villa 1 False 0 None ... 2022-10-13T10:36:39.8 09667ce7-07aa-ec11-aa4c-246e960cbc4d 0 Lefkosia 141 1 288943c7-d6a9-ec11-aa4c-246e960cbc4d None 80
2 200000.0 c42dc379-5f67-e811-bb4e-a4badb3ceace 2605 2 Storey Modern 3 Bedroom House For Sale In Da... Dali House/Villa 0 False 210 None ... 2021-06-20T15:09:05.183 f37cf839-6667-e811-bb4e-a4badb3ceace 0 Lefkosia 180 0 c32dc379-5f67-e811-bb4e-a4badb3ceace None 91
3 200000.0 6370763d-61ab-e811-b319-a4badb3ceacd 2962 Very Nice And Spacious 3 Bedroom Detached Hous... Lakatameia House/Villa 1 False 0 None ... 2022-10-13T10:36:25.23 8384a75b-61ab-e811-b319-a4badb3ceacd 0 Lefkosia 170 1 6270763d-61ab-e811-b319-a4badb3ceacd None 91
4 200000.0 7da50193-266b-e811-bb4e-a4badb3ceace 2633 3 Bedroom Under Construction Detached 4 Houses... Tseri House/Villa 0 False 173 None ... 2021-06-20T15:09:38.88 c1b54437-0d81-e811-ae53-a4badb3ceacd 0 Lefkosia 150 VAT 0 7ca50193-266b-e811-bb4e-a4badb3ceace None 100
5 210000.0 96d0fb0a-89bd-ec11-aa4e-246e960cbc4d 18927 3 Bedroom House For Sale In Agios Dometios, Ni... Agios Dometios House/Villa 0 False 293 None ... 2022-08-08T13:20:09.877 d1fadd0c-8ebd-ec11-aa4e-246e960cbc4d 0 Lefkosia 140 0 95d0fb0a-89bd-ec11-aa4e-246e960cbc4d None 38
6 211000.0 1881b78e-52d7-ec11-aa4e-246e960cbc4d 19481 4 Bedroom House For Sale In Archangelos, Nicosia Archangelos House/Villa 0 False 296 None ... 2022-08-10T09:32:52.27 e0c3a52d-53d7-ec11-aa4e-246e960cbc4d 0 Lefkosia 130 0 1781b78e-52d7-ec11-aa4e-246e960cbc4d None 68
7 215000.0 eaa2b630-e685-ec11-aa4c-246e960cbc4d 16834 In Excellent Location 3 Bedrooms House In Arch... Archangelos House/Villa 0 False 180 None ... 2022-10-10T13:53:48.94 95726746-e685-ec11-aa4c-246e960cbc4d 0 Lefkosia 160 1 e9a2b630-e685-ec11-aa4c-246e960cbc4d None 80
8 220000.0 2ad40da2-a190-ec11-aa4c-246e960cbc4d 17075 Incomplete residential development in Politiko... Politiko House/Villa 0 False 0 None ... 2022-02-18T10:00:43.46 2bd40da2-a190-ec11-aa4c-246e960cbc4d 0 Lefkosia 509 0 29d40da2-a190-ec11-aa4c-246e960cbc4d None 0
9 220000.0 a19ad42e-cd59-e911-8b16-a4badb3ceacd 3846 3 Bedroom House For Sale In Spilia With Great ... Spilia House/Villa 0 False 0 None ... 2022-05-25T11:57:46.77 9214ee5b-df59-e911-8b16-a4badb3ceacd 0 Lefkosia 0 0 a09ad42e-cd59-e911-8b16-a4badb3ceacd None 68
10 225000.0 4bf71317-9cee-e911-b59c-a4badb3ceacd 5397 3 Bedroom House For Sale In Strovolos, Nicosia Strovolos House/Villa 0 False 280 None ... 2022-05-16T06:40:32.417 491a8e48-a6ee-e911-b59c-a4badb3ceacd 0 Lefkosia 149 0 4af71317-9cee-e911-b59c-a4badb3ceacd None 20
11 225000.0 082e2088-7baf-e911-935a-a4badb3ceacd 4738 4 Bedroom House For Sale In Nisou, Nicosia Nisou House/Villa 0 False 210 None ... 2021-09-06T14:05:58.267 dfed3499-7baf-e911-935a-a4badb3ceacd 0 Lefkosia 225 0 072e2088-7baf-e911-935a-a4badb3ceacd None 91
12 230000.0 8a978c4b-7383-ec11-aa4c-246e960cbc4d 16788 3 Bedroom Upper For Sale In Strovolos, Nicosia Strovolos House/Villa 0 False 0 None ... 2022-02-03T14:20:16.57 a7df5018-f384-ec11-aa4c-246e960cbc4d 0 Lefkosia 190 0 89978c4b-7383-ec11-aa4c-246e960cbc4d None 68
13 230000.0 960d522b-dc39-e911-a389-a4badb3ceacd 3710 Very Nice 3 Bedroom House For Sale Very Close ... Kakopetria House/Villa 0 False 0 None ... 2022-03-09T14:51:47.3 00000000-0000-0000-0000-000000000000 0 Lefkosia 0 0 950d522b-dc39-e911-a389-a4badb3ceacd None 80
14 234000.0 1c3cbfed-2030-ec11-aa4c-246e960cbc4d 15488 5 Bedroom House For Sale In Spilia, Nicosia Spilia House/Villa 0 False 762 None ... 2022-05-25T11:57:23.79 dab17ebe-2130-ec11-aa4c-246e960cbc4d 0 Lefkosia 180 0 1b3cbfed-2030-ec11-aa4c-246e960cbc4d None 20
15 235000.0 b306f83b-7723-ed11-aa4e-246e960cbc4d 20722 4 Bedroom House For Sale In Lakatamia Lakatameia House/Villa 0 False 500 None ... 2022-10-06T12:52:50.527 d36a70ee-8623-ed11-aa4e-246e960cbc4d 0 Lefkosia 180 1 b206f83b-7723-ed11-aa4e-246e960cbc4d None 91
16 235000.0 9925f398-c76d-ea11-a1df-a4badb3ceacd 7596 3 Bedroom House For Sale In Tseri, Nicosia Tseri House/Villa 0 False 0 None ... 2021-08-03T14:02:02.54 636187c3-fe6f-ea11-a1df-a4badb3ceacd 0 Lefkosia 170 0 9825f398-c76d-ea11-a1df-a4badb3ceacd None 91
17 235000.0 b536908d-d0a7-e911-935a-a4badb3ceacd 4605 3 Bedroom House/Villa For Sale in Dali, Nicosia Dali House/Villa 0 False 293 None ... 2022-10-13T11:01:48.23 4c888960-d3a7-e911-935a-a4badb3ceacd 0 Lefkosia 180 1 b436908d-d0a7-e911-935a-a4badb3ceacd None 80
18 238000.0 e2519c9f-7188-e611-878e-a4badb3ceace 1450 3 Bedroom Detached House For Sale In Geri, Ni... Geri House/Villa 0 False 200 None ... 2022-02-18T09:06:00.217 c831a074-def9-eb11-aa49-246e960cbc4d 0 Lefkosia 165 0 e1519c9f-7188-e611-878e-a4badb3ceace None 91
19 238000.0 bfe95fce-0959-e811-bb4e-a4badb3ceace 2547 A Very Nice 3 Bedroom Beautiful House For Sale... Kalo Chorio Oreinis House/Villa 0 False 0 None ... 2021-06-28T08:59:28.997 48c399a4-9568-e811-bb4e-a4badb3ceace 0 Lefkosia 230 0 bee95fce-0959-e811-bb4e-a4badb3ceace None 68
20 238350.0 fcfdc748-7990-ea11-a1df-a4badb3ceacd 8196 3 Bedroom House For Sale In Latsia, Nicosia Latsia (Lakkia) House/Villa 0 False 256 None ... 2022-06-23T09:23:39.51 f18a96be-d5f2-ec11-aa4e-246e960cbc4d 0 Lefkosia 143 0 fbfdc748-7990-ea11-a1df-a4badb3ceacd None 80
21 239000.0 3f5369dc-8bf0-ec11-aa4e-246e960cbc4d 20038 4 Bedroom House For Sale In Dali, Nicosia Dali House/Villa 0 False 258 None ... 2022-06-30T11:56:33.08 00000000-0000-0000-0000-000000000000 0 Lefkosia 190 0 3e5369dc-8bf0-ec11-aa4e-246e960cbc4d None 80
22 240000.0 93eac30c-cd29-ed11-aa4e-246e960cbc4d 20805 3 Bedroom House For Sale In Lakatameia, Anthou... Lakatameia House/Villa 0 False 159 None ... 2022-09-07T07:56:28.5 6e6adaea-692e-ed11-aa4e-246e960cbc4d 0 Lefkosia 143 1 92eac30c-cd29-ed11-aa4e-246e960cbc4d None 80
23 243500.0 753a3401-d75b-ea11-a1df-a4badb3ceacd 7339 3 Bedroom House For Sale Ιn Tseri, Nicosia Tseri House/Villa 0 False 700 None ... 2022-06-09T12:06:17.957 001fea44-d75b-ea11-a1df-a4badb3ceacd 0 Lefkosia 227 VAT 0 743a3401-d75b-ea11-a1df-a4badb3ceacd None 91
24 245000.0 0fd2cd26-9e23-ed11-aa4e-246e960cbc4d 20733 3 Bedroom Houses For Sale In Pera Chorio, Nicosia Pera Chorio House/Villa 0 False 581 None ... 2022-10-06T12:42:59.047 c1ad0059-9e23-ed11-aa4e-246e960cbc4d 0 Lefkosia 200 1 0ed2cd26-9e23-ed11-aa4e-246e960cbc4d None 68
25 245000.0 bb9bd1f8-c729-ed11-aa4e-246e960cbc4d 20804 3 Bedroom Plus Office House For Sale In Tseri,... Tseri House/Villa 0 False 215 None ... 2022-09-14T09:42:17.427 dac9ee2a-532f-ed11-aa4e-246e960cbc4d 0 Lefkosia 178 1 ba9bd1f8-c729-ed11-aa4e-246e960cbc4d None 91
26 245000.0 0de63389-f8ff-eb11-aa4a-246e960cbc4d 14744 3 Bedroom House For Sale In Kaimakli Kaimakli House/Villa 0 False 0 None ... 2021-08-24T07:46:06.86 2d97036a-faff-eb11-aa4a-246e960cbc4d 0 Lefkosia 180 0 0ce63389-f8ff-eb11-aa4a-246e960cbc4d None 38
27 250000.0 c6caa20d-b910-ec11-aa4a-246e960cbc4d 14865 3 Bedroom House For Sale In Mammari, Nicosia Mammari House/Villa 0 False 586 None ... 2022-08-02T07:51:38.443 1ebcd136-7311-ed11-aa4e-246e960cbc4d 0 Lefkosia 190 0 c5caa20d-b910-ec11-aa4a-246e960cbc4d None 80
28 250000.0 c1b1fcc5-9251-ea11-a1df-a4badb3ceacd 7090 Houses For Sale In Strovolos Nicosia Strovolos House/Villa 0 False 301 None ... 2022-10-06T12:17:01.56 22c7750a-9351-ea11-a1df-a4badb3ceacd 0 Lefkosia 322 1 c0b1fcc5-9251-ea11-a1df-a4badb3ceacd None 68
29 250000.0 b8823644-babb-eb11-aa46-246e960cbc4d 13710 4 Bedroom House For Sale In Dali, Nicosia Dali House/Villa 0 False 280 None ... 2022-07-04T12:39:25.063 66596a80-bbbb-eb11-aa46-246e960cbc4d 0 Lefkosia 190 0 b7823644-babb-eb11-aa46-246e960cbc4d None 100
30 250000.0 8cee923f-bc5c-ec11-aa4c-246e960cbc4d 16227 3 Bedroom House For Sale In Nisou Nisou House/Villa 0 False 456 None ... 2021-12-21T08:42:29.78 418ba317-cf61-ec11-aa4c-246e960cbc4d 0 Lefkosia 170 0 ff0e8d10-bc5c-ec11-aa4c-246e960cbc4d None 54
31 255000.0 ed27019b-f2db-ea11-a1df-a4badb3ceacd 8867 3 Bedroom House For Sale In Ag.Dometios, Nicosia Agios Dometios House/Villa 0 False 341 None ... 2021-09-08T13:05:39.39 e756e5d2-00dc-ea11-a1df-a4badb3ceacd 0 Lefkosia 143 0 ec27019b-f2db-ea11-a1df-a4badb3ceacd None 80
32 258000.0 c453e74a-8bba-e811-91b1-a4badb3ceacd 3048 4 Bedroom Under Construction Detached House Fo... Pera Chorio House/Villa 0 False 332 None ... 2021-06-19T12:41:37.69 fa1f8fa2-8bba-e811-91b1-a4badb3ceacd 0 Lefkosia 185 VAT 0 c353e74a-8bba-e811-91b1-a4badb3ceacd None 100
33 260000.0 58fcb64e-443a-ed11-aa4e-246e960cbc4d 21097 In Excellent Condition 3 Bedroom Plus Attic Ho... Anthoupoli House/Villa 0 False 190 None ... 2022-09-22T10:19:04.283 4e45580e-453a-ed11-aa4e-246e960cbc4d 0 Lefkosia 180 1 b9497873-8039-ed11-aa4e-246e960cbc4d None 91
34 260000.0 aef929d7-f7a3-eb11-aa45-246e960cbc4e 13205 5 Bedroom House For Sale In Lakatamia, Nicosia Lakatameia House/Villa 0 False 260 None ... 2022-06-09T13:03:20.523 271593f7-f7a3-eb11-aa45-246e960cbc4e 0 Lefkosia 293 0 adf929d7-f7a3-eb11-aa45-246e960cbc4e None 68
35 260000.0 d2106e0e-092a-ed11-aa4e-246e960cbc4d 20822 3 Bedroom House For Sale In Palouriotissa, Nic... Palouriotissa House/Villa 0 False 281 None ... 2022-09-06T14:36:45.15 bfd8748a-cd2a-ed11-aa4e-246e960cbc4d 0 Lefkosia 170 1 9aff1698-0f29-ed11-aa4e-246e960cbc4d None 80
36 rows × 21 columns
??請參閱此處的請求檔案：https ://requests.readthedocs.io/en/latest/
對于熊貓：https ://pandas.pydata.org/docs/
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/514458.html