我正在嘗試構建我的第一個網路抓取工具，但我不知道如何阻止我的程式尋找“下一頁”鏈接。
#get URLs for all pages
def page_parse(main_url, url_list):
page = requests.get(main_url);
soup = BeautifulSoup(page.content, 'html.parser');
#check if next page button inactive
if soup.find('a.next.ajax-page', href=True) == None:
print('debug');
return url_list;
next_page = soup.select_one('a.next.ajax-page', href=True)['href']
next_page = (f'http://www.yellowpages.com{next_page}')
url_list.append(next_page);
print(str(url_list))
page_parse(next_page, url_list);
return url_list;
我知道錯誤是什么我只是不知道如何檢查“下一頁”按鈕是否處于活動狀態。我嘗試尋找第一頁和最后一頁的“下一頁”按鈕之間的 html 差異（第一頁使用 a.next.ajax-page，而最后一頁使用 div.next）。根據我對代碼所做的更改，要么點擊 print('debug')，要么到達最后一頁并點擊 TypeError [見下文]。我認為問題在于不呼叫元素就無法檢查元素是否存在。
錯誤代碼：
['http://www.yellowpages.com/omaha-ne/towing?page=2']
['http://www.yellowpages.com/omaha-ne/towing?page=2', 'http://www.yellowpages.com/omaha-ne/towing?page=3']
['http://www.yellowpages.com/omaha-ne/towing?page=2', 'http://www.yellowpages.com/omaha-ne/towing?page=3', 'http://www.yellowpages.com/omaha-ne/towing?page=4']
['http://www.yellowpages.com/omaha-ne/towing?page=2', 'http://www.yellowpages.com/omaha-ne/towing?page=3', 'http://www.yellowpages.com/omaha-ne/towing?page=4', 'http://www.yellowpages.com/omaha-ne/towing?page=5']
['http://www.yellowpages.com/omaha-ne/towing?page=2', 'http://www.yellowpages.com/omaha-ne/towing?page=3', 'http://www.yellowpages.com/omaha-ne/towing?page=4', 'http://www.yellowpages.com/omaha-ne/towing?page=5', 'http://www.yellowpages.com/omaha-ne/towing?page=6']
['http://www.yellowpages.com/omaha-ne/towing?page=2', 'http://www.yellowpages.com/omaha-ne/towing?page=3', 'http://www.yellowpages.com/omaha-ne/towing?page=4', 'http://www.yellowpages.com/omaha-ne/towing?page=5', 'http://www.yellowpages.com/omaha-ne/towing?page=6', 'http://www.yellowpages.com/omaha-ne/towing?page=7']
Traceback (most recent call last):
File "c:\Users\-\Documents\code\Python Projects\webscrape2.py", line 49, in <module>
url_list = page_parse(main_url, url_list);
File "c:\Users\-\Documents\code\Python Projects\webscrape2.py", line 19, in page_parse
page_parse(next_page, url_list);
File "c:\Users\-\Documents\code\Python Projects\webscrape2.py", line 19, in page_parse
page_parse(next_page, url_list);
File "c:\Users\-\Documents\code\Python Projects\webscrape2.py", line 19, in page_parse
page_parse(next_page, url_list);
[Previous line repeated 3 more times]
File "c:\Users\-\Documents\code\Python Projects\webscrape2.py", line 15, in page_parse
next_page = soup.select_one('a.next.ajax-page', href=True)['href']
TypeError: 'NoneType' object is not subscriptable
抱歉，如果這令人困惑，這是我第一次發布問題。
uj5u.com熱心網友回復：
這里的問題是您正在嘗試訪問一個NoneType變數。 next_page = soup.select_one('a.next.ajax-page', href=True)不回傳任何內容，因此您無法訪問['href']內部
uj5u.com熱心網友回復：
怎么了？
您的選擇soup.find('a.next.ajax-page', href=True)沒有以任何方式找到您正在搜索的元素，因為它是語法（查找和 css 選擇器）的混合，并且將始終回傳None- 因此它也將無法訪問屬性值。
怎么修？
更改檢查下一頁元素的行：
if soup.find('a.next.ajax-page', href=True) == None:
到：
if soup.find('a',{'class':'next ajax-page'}) == None:
或者
if soup.select_one('a.next.ajax-page') == None:
您還應該能夠抓取搜索結果的所有基本資訊并將其存盤在一個步驟中，而不是回傳搜索頁面的 url 串列：
def page_parse(url):
data = []
while True:
page = requests.get(url)
soup = BeautifulSoup(page.text)
for item in soup.select('div.result'):
data.append({
'title':item.h2.text,
'url':f"{baseUrl}{item.a['href']}"
})
if (url := soup.select_one('a.next.ajax-page')):
url = f"{baseUrl}{url['href']}"
else:
return data
例子
import requests
from bs4 import BeautifulSoup
baseUrl = 'http://www.yellowpages.com'
def page_parse(url):
data = []
while True:
page = requests.get(url)
soup = BeautifulSoup(page.text)
for item in soup.select('div.result'):
data.append({
'title':item.h2.text,
'url':f"{baseUrl}{item.a['href']}"
})
if (url := soup.select_one('a.next.ajax-page')):
url = f"{baseUrl}{url['href']}"
else:
return data
page_parse('http://www.yellowpages.com/omaha-ne/towing')
輸出
[{'title': "1. Keith's BP",
'url': 'http://www.yellowpages.com/omaha-ne/mip/keiths-bp-460502890?lid=1002059325385'},
{'title': '2. Neff Towing Svc',
'url': 'http://www.yellowpages.com/omaha-ne/mip/neff-towing-svc-21969600?lid=1000282974083#gallery'},
{'title': '3. A & A Towing',
'url': 'http://www.yellowpages.com/omaha-ne/mip/a-a-towing-505777665?lid=1002056319136'},
{'title': '4. Cross Electronic Recycling',
'url': 'http://www.yellowpages.com/omaha-ne/mip/cross-electronic-recycling-473693798?lid=1000236876513'},
{'title': '5. 24 Hour Towing',
'url': 'http://www.yellowpages.com/omaha-ne/mip/24-hour-towing-521607477?lid=1001918028003'},
{'title': '6. A & A Towing Fast Friendly',
'url': 'http://www.yellowpages.com/omaha-ne/mip/a-a-towing-fast-friendly-478453697?lid=1000090213043'},
{'title': '7. Austin David Towing',
'url': 'http://www.yellowpages.com/omaha-ne/mip/austin-david-towing-465037110?lid=1001788338357'},...]
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/418822.html
標籤：
下一篇：我無法使用bs4訪問某些元素？