我有一個學校專案，我必須在比利時搜索大約 88 000 家公司。有些網站需要更長的時間才能在網站上找到一個詞，所以我認為這只是一個更大的網站。但是，當我訪問 DHL 的網站 ( www.dhl.com ) 時，我的程式什么也沒做。是否有原因這是不可能的，或者公司可以禁止抓取他們的網站？我認為我的代碼沒有任何問題，但我已將其放在下面。變數網站只是一個包含公司所有 URL 的陣列。
counter = 0
url = ""
for url in websites:
counter = 1
word = 'de'
print(f'{counter}: {url}')
try:
r = req.get('http://' url.strip())
r.encode = 'utf-8'
html_content = r.text
soup = bs(html_content, 'lxml')
to_search = soup.find_all(text = lambda text : text and word.lower() in text)
if len(to_search) > 0:
print(f'\tamount of "{word}" on website: {len(to_search)}')
else:
print(f'\t"{word}" never occured')
except:
print(f'\t{url} is unavailable')
uj5u.com熱心網友回復：
是的，公司可以通過檢測機器人來防止報廢。但是，如果您訪問 dhl 站點，它似乎會重定向，因此不確定是否是問題所在。當我運行你的腳本時，它就會停止。
不過，這也提出了另一個問題。當您搜索這個詞"de"時，您是專門尋找 ONLY 的'de'嗎？因為當您對其進行編碼時，它將回傳任何包含'de'. 例如，對于 dhl.com 站點的此代碼，它''bazadebezolkohpepadr="1231685289"''會在 26 個中找到'de'它計數的 1 個。
到那時，這也將包括 html 中的任何功能或腳本。例如，這也算作 dhl.com 的“de”字：
'!function(a){var e="https://s.go-mpulse.net/boomerang/",t="addEventListener";if("True"=="True")a.BOOMR_config=a.BOOMR_config||{},a.BOOMR_config.PageParams=a.BOOMR_config.PageParams||{},a.BOOMR_config.PageParams.pci=!0,e="https://s2.go-mpulse.net/boomerang/";if(window.BOOMR_API_key="RSVGU-547KJ-ZUMZD-ZW27F-P4RHY",function(){function n(e){a.BOOMR_onload=e&&e.timeStamp||(new Date).getTime()}if(!a.BOOMR||!a.BOOMR.version&&!a.BOOMR.snippetExecuted){a.BOOMR=a.BOOMR||{},a.BOOMR.snippetExecuted=!0;var i,_,o,r=document.createElement("iframe");if(a[t])a[t]("load",n,!1);else if(a.attachEvent)a.attachEvent("onload",n);r.src="javascript:void(0)",r.title="",r.role="presentation",(r.frameElement||r).style.cssText="width:0;height:0;border:0;display:none;",o=document.getElementsByTagName("script")[0],o.parentNode.insertBefore(r,o);try{_=r.contentWindow.document}catch(O){i=document.domain,r.src="javascript:var d=document.open();d.domain=\'" i "\';void(0);",_=r.contentWindow.document}_.open()._l=function(){var a=this.createElement("script");if(i)this.domain=i;a.id="boomr-if-as",a.src=e "RSVGU-547KJ-ZUMZD-ZW27F-P4RHY",BOOMR_lstart=(new Date).getTime(),this.body.appendChild(a)},_.write("<bo" \'dy onl oad="document._l();">\'),_.close()}}(),"".length>0)if(a&&"performance"in a&&a.performance&&"function"==typeof a.performance.setResourceTimingBufferSize)a.performance.setResourceTimingBufferSize();!function(){if(BOOMR=a.BOOMR||{},BOOMR.plugins=BOOMR.plugins||{},!BOOMR.plugins.AK){var e=""=="true"?1:0,t="",n="uxq4jklij32vkyqodtiq-f-5a918ce10-clientnsv4-s.akamaihd.net",i="false"=="true"?2:1,_={"ak.v":"32","ak.cp":"640765","ak.ai":parseInt("326248",10),"ak.ol":"0","ak.cr":100,"ak.ipv":4,"ak.proto":"http/1.1","ak.rid":"1eef41a0","ak.r":42169,"ak.a2":e,"ak.m":"a","ak.n":"essl","ak.bpcip":"165.225.196.0","ak.cport":60467,"ak.gh":"23.52.43.86","ak.quicv":"","ak.tlsv":"tls1.3","ak.0rtt":"","ak.csrc":"-","ak.acc":"bbr","ak.t":"1645092049","ak.ak":"hOBiQwZUYzCg5VSAfCLimQ==gPk5vdO2N xaMvOHluPHPA75P2qfqUmFzBjJVMEHxXiQzYo6BKsg3x M21NSbrw9cCziPvIOGYUIJBB4srzYCVE S3n8bUaeN3xk/ pwl6 CHMsGzVTAqySt3i0ewJ1GI5g6CypA2BL8kALrH1Z8sCNqZy0eMS1nmkOjppm315f4Qf l8qhUvekcsNZ3NALIP0UqjlcEZtDpHsWEC AoFQQTILEGMxqi4IctzOqS3RG 5KW2/TQ7YbksSvp0tOo9JIjguIAqM2gK8d/uOsFF2CIlIYFYtXQCoDnlosdxnVQjkIGLqOVcM81NauDexVgIRgCIbQWCO1VA2ZuuapLm5Jc7Noh83C3XodGjRNlS9oiLRZT7YI z9GVtfANw8nVQn7TMhIt8F9KlMmzcHkKEwBFhXYozOkhx0YluLmLTaj8=","ak.pv":"278","ak.dpoabenc":"","ak.tf":i};if(""!==t)_["ak.ruds"]=t;var o={i:!1,av:function(e){var t="http.initiator";if(e&&(!e[t]||"spa_hard"===e[t]))_["ak.feo"]=void 0!==a.aFeoApplied?1:0,BOOMR.addVar(_)},rv:function(){var a=["ak.bpcip","ak.cport","ak.cr","ak.csrc","ak.gh","ak.ipv","ak.m","ak.n","ak.ol","ak.proto","ak.quicv","ak.tlsv","ak.0rtt","ak.r","ak.acc","ak.t","ak.tf"];BOOMR.removeVar(a)}};BOOMR.plugins.AK={akVars:_,akDNSPreFetchDomain:n,init:function(){if(!o.i){var a=BOOMR.subscribe;a("before_beacon",o.av,null,null),a("onbeacon",o.rv,null,null),o.i=!0}return this},is_complete:function(){return!0}}}}()}(window);'
我會在這里改變一些事情：
- 改成
'https://' - 在標題中包含用戶代理
- 在請求中包含超時引數（除了您的代碼不會掛起之外，不會解決問題
話雖如此，amount of "de" on website: 21為 dhl 找到的代碼如下：
import requests as req
from bs4 import BeautifulSoup as bs
websites = ['www.dhl.com']
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}
counter = 0
url = ""
for url in websites:
counter = 1
word = 'de'
print(f'{counter}: {url}')
try:
r = req.get('https://' url.strip(), headers=headers, timeout=10)
r.encode = 'utf-8'
html_content = r.text
soup = bs(html_content, 'lxml')
to_search = soup.find_all(text = lambda text : text and word.lower() in text)
if len(to_search) > 0:
print(f'\tamount of "{word}" on website: {len(to_search)}')
else:
print(f'\t"{word}" never occured')
except:
print(f'\t{url} is unavailable')
輸出：
amount of "de" on website: 21
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/426939.html