我正在嘗試使用 Selenium-“https://jobsearch.az/vacancies”滾動網頁。但是，當您打開它并單擊空缺時，您會看到兩個頁面并排滾動。我需要滾動中間的那個，所以 selenium 也可以從其他作業中獲取資訊。但是，現在，它在 14 時停止，如果您不滾動，您只能看到這些。
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
import pandas as pd
from selenium.webdriver.common.action_chains import ActionChains
path = "C:/Users/nihad/OneDrive/Documents/aInternship/chromedriver.exe"
driver = webdriver.Chrome(path)
url = "https://jobsearch.az/vacancies"
driver.get(url)
time.sleep(10)
soup = BeautifulSoup(driver.page_source, 'html.parser')
lists = soup.find_all('div', {'class': 'list__item'})
jobs_list=[]
print('A')
x=True
while x:
print('B')
driver.maximize_window()
for index, _list in enumerate(lists):
link = _list.find(
'a', class_="list__item__text")["href"]
current_url = "https://jobsearch.az" link
driver.get(current_url)
time.sleep(10)
soup = BeautifulSoup(driver.page_source, 'html.parser')
jobs = soup.find_all('div', {'class': 'vacancy'})
for index, job in enumerate(jobs):
company = job.find( # div.a.
'div', class_="vacancy__start").text.strip()
print(f'Company:{company}')
category = job.find( # div.a.
'span', class_="company__industry").text
print(f'Category:{category}')
# key_requirements = job.find( # div.a.
# 'div', class_="content-text").text
# print(f'Key requirements:{key_requirements}')
job_title = job.find( # div.a.
'h1', class_="vacancy__title").text.strip()
print(f'Job title: {job_title}')
deadline = job.find( # div.a.
'span', class_="vacancy__deadline").text.strip()
print(f'Deadline: {deadline}')
views = _list.find('div', class_="list__item__end").text.strip()
print(f'Views: {views}')
data = {
"job_title":job_title,
"company":company,
"category":category,
"deadline":deadline,
"views":views
}
jobs_list.append(data)
driver.minimize_window()
print('C')
driver.find_element_by_xpath('//h3[@]').click()
driver.execute.script("window.scrollBy(0, document.body.scrollHeight)")
for i in range(5):
driver.find_element_by_tag_name('h3').send_keys(Keys.END)
time.sleep(4)
driver.execute_script("window.scrollBy(0, 1000)", "")
time.sleep(5)
print('after executing scrolling')
# element = driver.find_element_by_class_name('vacancy__title')
# actions = ActionChains(driver)
# actions.move_to_element(element).perform()
dataframe = pd.DataFrame(jobs_list)
dataframe
driver.close()
我寫了所有 3 種技術，但沒有任何效果。
uj5u.com熱心網友回復：
我不知道你是否需要使用硒。但我在另一個圖書館有一個例子 - 請求。由于 api 使用 cookie，我們必須使用 session 來傳遞 XSRF 令牌。
import requests
import pandas as pd
jobs_list = []
def foo(url):
headers = {
'accept': 'application/json, text/plain, */*',
'x-requested-with': 'XMLHttpRequest'
}
response = session.request("GET", url, headers=headers)
for job in response.json()['items']:
data = {
"job_title": job['title'],
"company": job['company']['title'],
"category": job['category']['title'],
"deadline": job['deadline_at'],
"views": job['view_count']
}
jobs_list.append(data)
if 'next' in response.json():
foo(response.json()['next'])
session = requests.session()
response = session.get('https://jobsearch.az/vacancies')
foo('https://jobsearch.az/api-az/vacancies-az?hl=az')
dataframe = pd.DataFrame(jobs_list)
dataframe
輸出：
job_title ... views
0 Revenue manager ... 356
1 Operator (Sat?? t?msil?isi) ... 236
2 Sat?c? (xan?m) ... 766
3 ADM ISO 9001 Beyn?lxalq Sertifikat Proqram?n? ... ... 1.6K
4 Avto-s?nayed? operator montaj??s? ... 218
... ... ... ...
1656 Receptionist (gec? n?vb?si) ... 735
1657 Складчик ... 400
1658 Android proqram?? ... 358
1659 Credit Risk Specialist ... 587
1660 ?poteka üzr? müt?x?ssis, apar?c? müt?x?ssis (B... ... 439
[1661 rows x 5 columns]
轉載請註明出處，本文鏈接：https://www.uj5u.com/qiye/494114.html
標籤：javascript Python 爪哇 硒 滚动