webmagic爬取https網站,報錯javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
彭大勃 2020-08-18 14:41:23 117 收藏
分類專欄： 經驗分享 文章標簽： java https 經驗分享 資料挖掘
著作權
目前webmagic最新版是0.7.3版本，在爬取只支持TLS1.2的https站點會報錯,作者說會在0.7.4版本發布修復錯誤,但是等了3年了都沒等到0.7.4版本的發布.
javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
at sun.security.ssl.Alert.createSSLException(Alert.java:131)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:187)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1198)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1107)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:400)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:372)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:404)
at us.codecraft.webmagic.Spider.access$000(Spider.java:61)
at us.codecraft.webmagic.Spider$1.run(Spider.java:320)
at us.codecraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
目前的解決辦法是：臨時適配方式，修改HttpClientGenerator中的buildSSLConnectionSocketFactory方法，重寫自己實作的HttpClientDownloader，并設定到Spider中。修改內容如下:
return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
null,
new DefaultHostnameVerifier())
目前我已經把原始碼修改好了,并編譯為jar包,有被同樣問題困擾的朋友可以直接下載使用.
webmagic-core-0.7.3.jar
提取碼:webm
jar包使用方法:將下載好的 jar包,放在你的maven倉庫的us\codecraft\webmagic-core\0.7.3路徑下,替換原有jar包即可解決問題.
uj5u.com熱心網友回復：
可以解決ssl轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/195418.html
標籤：Java相關