我想從這個域https://stacker.com/stories/1587/100-best-movies-all-time抓取資料
uj5u.com熱心網友回復：
只有添加標題才能獲取資料User-Agent
from bs4 import BeautifulSoup as BS
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0',
}
url = 'https://www.the-numbers.com/movie/Avengers-Endgame-(2019)#tab=cast-and-crew'
response = requests.get(url, headers=headers)
# --- response ---
#print(response.status_code)
#print(response.text[:1000])
soup = BS(response.text, 'html.parser')
all_items = soup.find_all('div', id="cast-and-crew")
for item in all_items:
print(item.get_text(strip=True, separator='\n'))
結果：
Lead Ensemble Members
Robert Downey, Jr.
Tony Stark/Iron Man
Chris Evans
Steve Rogers/Captain America
Mark Ruffalo
Bruce Banner/Hulk
Chris Hemsworth
Thor
Scarlett Johansson
Natasha Romanoff/Black Widow
Jeremy Renner
Clint Barton/Hawkeye
Don Cheadle
...
uj5u.com熱心網友回復：
我和你一樣是新手，我嘗試過使用 beatifulsoap 它確實收到了請求，也許是某種型別的安全性，但我嘗試用 selenium 做你想做的事情，它可以作業，檢查一下：
from selenium import webdriver
website = "https://www.the-numbers.com/movie/Avengers-Endgame-(2019)#tab=cast-and-crew"
path = "/"
chrome_options = webdriver.ChromeOptions();
chrome_options.add_experimental_option("excludeSwitches", ['enable-logging'])
driver = webdriver.Chrome(options=chrome_options);
driver.get(website)
box = driver.find_element_by_class_name("cast_new")
matches = box.find_elements_by_xpath('//*[@id="cast-and-crew"]/div[5]/table/tbody/tr[1]/td[1]/b/a')
for match in matches:
print(match.text)
driver.quit()
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/449244.html