WebMagic簡介
webmagic是不需要配置，便捷資料挖掘的爬蟲框架，其擁有簡單且靈活的api，webmagic整體采用模塊化架構，整個爬蟲的生命周期：提取連接——>頁面下載——>內容提取——>資料持久化，并且支持多執行緒挖掘，支持分布式挖掘，支持自動重試，自定義cookies，模塊可定制化等功能，
Selenium簡介
selenium是一款遵守Apache License 2.0協議的開源框架，用于Web程式自動化測驗工具，selenium測驗運行在瀏覽器中，就像真的用戶在操作一樣，包括Firefox、Safari、Chrome、Opera等等，
Tesseract-OCR簡介
一款由HP實驗室開發由Google維護的開源OCR引擎，與MODI相比，可以不斷的訓練的庫，使影像轉文本的能力不斷增強，
一、專案需求
總所周知，在資料挖掘領域，其中及其重要的就是資料的爬取，而在大資料時代的到來之后對資料量的需求更加的大，這迫使爬蟲需要在短時間內爬取更多的資料，但是現在許多的網站都設定了反扒機制，最常見的反扒機制就是封鎖段時間請求過多的ip，而解決方式之一就是使用代理服務器，通過請求道代理服務器，代理服務器去請求目的地站點，這樣即使被封ip也是代理服務器被封鎖，而我們通常沒有那么多代理服務器，市面上有許多的代理服務商，比如我們今天要爬取的物件米撲代理，它雖然有收費的代理服務器，但是它的免費代理也是可以用的，而我們的任務就是爬去ip、埠、型別、匿名度、國家（省市）、運營商、回應時間、傳輸速度、驗證日期等等，
專案成品gitee地址：mipuproxy: webmagic+selenium+tesseract-ocr實作米撲代理代理爬去
二、技術可行性分析
先上圖，
很顯然除了埠之外還是比較好處理的，由于埠是一張圖片，就聯想到使用tesseract-ocr進行識別，為了可以直觀的無阻礙的模擬人訪問使用selenium進行輔助模擬人的操作，整體的爬蟲系統使用基于Java撰寫的WebMagic實作，
三、技術實施
專案整體使用SpringBoot工程化，如下圖：
分層明確，dao層為資料訪問層、entity為資料庫物體、service服務層、以及webmagic的任務層 ，
如下為本次專案maven所需的包的坐標：
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<!-- <dependency>-->
<!-- <groupId>com.google.guava</groupId>-->
<!-- <artifactId>guava</artifactId>-->
<!-- <version>23.0</version>-->
<!-- </dependency>-->
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>0.7.4</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>0.7.4</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-selenium</artifactId>
<version>0.7.4</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-chrome-driver</artifactId>
</dependency>
<dependency>
<groupId>net.sourceforge.tess4j</groupId>
<artifactId>tess4j</artifactId>
<version>4.5.4</version>
<exclusions>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
</exclusion>
<exclusion>
<groupId>net.sourceforge.lept4j</groupId>
<artifactId>lept4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>4.4.0</version>
</dependency>
<dependency>
<groupId>net.sourceforge.lept4j</groupId>
<artifactId>lept4j</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid-spring-boot-starter</artifactId>
<version>1.2.6</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>dao層介面：
package icu.smile.proxy.dao;
import icu.smile.proxy.entity.ProxyMiPu;
import org.springframework.data.jpa.repository.JpaRepository;
/**
* <p>
* dao層
* </p>
*
* @author starrysky
* @since 2021/6/7
*/
public interface MiPuDao extends JpaRepository<ProxyMiPu, Long> {
}
與資料庫互動用的物體類：
package icu.smile.proxy.entity;
import lombok.Data;
import lombok.experimental.Accessors;
import javax.persistence.*;
/**
* <p>
* 封裝物體
* </p>
*
* @author starrysky
* @since 2021/6/6
*/
@Entity
@Table(name = "proxy_mipu")
@Data
@Accessors(chain = true)
public class ProxyMiPu {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String ip;
private Integer port;
private String type;
private String anonymous;
private String location;
private String operator;
private String responseTime;
private String transmissionTime;
private String verificationTime;
}
service服務：
package icu.smile.proxy.service;
import icu.smile.proxy.entity.ProxyMiPu;
import java.util.List;
/**
* <p>
* 服務層
* </p>
*
* @author starrysky
* @since 2021/6/7
*/
public interface MiPuService {
void save(ProxyMiPu proxyMiPu);
List<ProxyMiPu> findAll(ProxyMiPu proxyMiPu);
void saveAll(List<ProxyMiPu> entityList);
}
age icu.smile.proxy.service.impl;
import icu.smile.proxy.dao.MiPuDao;
import icu.smile.proxy.entity.ProxyMiPu;
import icu.smile.proxy.service.MiPuService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Example;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import java.util.List;
import java.util.logging.Logger;
/**
* <p>
* TODO
* </p>
*
* @author starrysky
* @since 2021/6/7
*/
@Service
public class MiPuServiceImpl implements MiPuService {
@Autowired
private MiPuDao miPuDao;
@PersistenceContext
protected EntityManager entityManager;
private static final Logger LOGGER = Logger.getLogger(MiPuServiceImpl.class.getName());
@Override
@Transactional
public void save(ProxyMiPu proxyMiPu) {
ProxyMiPu miPum = new ProxyMiPu();
miPum.setIp(proxyMiPu.getIp());
miPum.setPort(proxyMiPu.getPort());
List<ProxyMiPu> list = this.findAll(miPum);
if (list.size()==0){
this.miPuDao.saveAndFlush(proxyMiPu);
}
}
@Override
@Transactional(propagation = Propagation.REQUIRED)
public void saveAll(List<ProxyMiPu> entityList){
miPuDao.saveAll(entityList);
}
@Override
public List<ProxyMiPu> findAll(ProxyMiPu proxyMiPu) {
Example example = Example.of(proxyMiPu);
List<ProxyMiPu> list = this.miPuDao.findAll(example);
return list;
}
}
webmagic的task任務：
package icu.smile.proxy.task;
import org.apache.commons.lang3.StringUtils;
import org.springframework.stereotype.Component;
import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.processor.PageProcessor;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.logging.Logger;
/**
* <p>
* 爬蟲處理器
* </p>
*
* @author starrysky
* @since 2021/6/6
*/
@Component
public class MiPuPageProcessor implements PageProcessor {
private static final Logger LOGGER = Logger.getLogger(MiPuPageProcessor.class.getName());
private static String cmdPrefix = "tesseract ";
private static String cmdSuffix = " stdout";
private static Process process = null;
private static BufferedReader bufferedReader = null;
private static String ImageResultOCR = null;
private Site site = Site.me()
.setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36")
.setDomain("proxy.mimvp.com")
.addCookie("UMIMVPSESSID", "hct11388aottkqvlgkoecsqna6")
.addCookie("Hm_lvt_51e3cc975b346e7705d8c255164036b3", "1622948301")
.addCookie("Hm_lpvt_51e3cc975b346e7705d8c255164036b3", "1622949348")
.setCharset("UTF-8")
.setTimeOut(5000)
.setRetrySleepTime(1000)
.setRetryTimes(3);
/***
* <p>
* 總的處理方法
* </p>
* @author starrysky
* @since 2021/6/6 22:14
* @param page page頁面
* @return void 無回傳值
*/
@Override
public void process(Page page) {
final List<String> proxyTabes = proxyTabes(page);
page.addTargetRequests(proxyTabes);
page.putField("ip", page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-ip']/text()").all());
page.putField("port", proxyPort(page));
page.putField("type", page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-type']/text()").all());
page.putField("anonymous", page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-anonymous']/text()").all());
page.putField("location", fixLocation(page));
page.putField("operator", fixOperator(page));
page.putField("responseTime", page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-pingtime']/@title").all());
page.putField("transmissionTime", page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-transfertime']/@title").all());
page.putField("verificationTime", page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-checkdtime']/text()").all());
page.addTargetRequests(fixSmileUrl(page));
page.addTargetRequests(fixListnav(page));
}
/***
* <p>
* 將page中的三個代理分類的url抽出
* </p>
* @author starrysky
* @since 2021/6/6 22:11
* @param page page頁面
* @return java.util.List<java.lang.String> 存盤三個代理方式的分類url
*/
public List<String> proxyTabes(Page page) {
List<String> tabes = new ArrayList<>();
for (String value : page.getHtml().css("div.free-proxytype-tabs").xpath("//a/@href").all()) {
tabes.add(page.getUrl().toString().substring(0, 23) + value);
}
return tabes;
}
/***
* <p>
* 將頁面中的記錄著代理埠的圖片抽出
* </p>
* @author starrysky
* @since 2021/6/6 22:12
* @param page page頁面
* @return java.util.List<java.lang.Integer> 存盤轉換之后的埠
*/
public List<Integer> proxyPort(Page page) {
List<Integer> protoPort = new ArrayList<>();
for (String value : page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-port']//img/@src").all()) {
protoPort.add(convertImageInteger(page.getUrl().toString().substring(0, 23) + value));
}
return protoPort;
}
/***
* <p>
* 將從圖片中提取出來
* </p>
* @author starrysky
* @since 2021/6/6 22:13
* @param url 每一個記錄著埠的圖片url
* @return java.lang.Integer 埠號
*/
public synchronized Integer convertImageInteger(String url) {
try {
process = Runtime.getRuntime().exec(cmdPrefix + url + cmdSuffix);
bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
ImageResultOCR = bufferedReader.readLine()
.replace(",", "")
.replace(".", "")
.replace("!", "")
.replace("@", "")
.replace("#", "")
.replace("$", "")
.replace("%", "")
.replace("^", "")
.replace("&", "")
.replace("*", "");
//處理8080被識別成為B080情況
if (ImageResultOCR.contains("B") && ImageResultOCR.length() != 3) {
ImageResultOCR = ImageResultOCR.replace("B", "8");
//處理三位的埠其本來為兩位，去除被干擾的B，比如B80，這個方法有點風險后期會改進
} else if (ImageResultOCR.contains("B") && ImageResultOCR.length() == 3) {
ImageResultOCR = ImageResultOCR.replace("B", "");
} else if (ImageResultOCR.contains("s") && ImageResultOCR.contains("e")) {
ImageResultOCR = ImageResultOCR.replace("s", "5").replace("e", "2");
} else if (ImageResultOCR.contains("s")) {
ImageResultOCR = ImageResultOCR.replace("s", "5");
} else if (ImageResultOCR.contains("e")) {
ImageResultOCR = ImageResultOCR.replace("e", "2");
}
return (Integer) Integer.parseInt(ImageResultOCR);
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
ImageResultOCR = "0";
LOGGER.info("OCR識別出錯,將使用0填充埠專案.");
}
return null;
}
public List<String> fixLocation(Page page) {
List<String> location = new ArrayList<>();
for (String loc : page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-country']/text()").all()) {
location.add(loc.replace("(", "").replace(")", "").trim());
}
return location;
}
/***
* <p>
* 修復運營商描述
* </p>
* @author starrysky
* @since 2021/6/6 23:22
* @param page page頁面
* @return java.util.List<java.lang.String> 回傳運營商的描述
*/
public List<String> fixOperator(Page page) {
List<String> operator = new ArrayList<>();
for (String oper : page.getHtml().xpath("//td[@class='free-proxylist-tbl-proxy-isp']/text()").all()) {
operator.add(oper == null ? "暫無運營商" : oper);
}
return operator;
}
/***
* <p>
* 處理串列頁面
* </p>
* @author starrysky
* @since 2021/6/6 23:23
* @param page page頁面
* @return java.util.List<java.lang.String> 串列頁面
*/
public List<String> fixListnav(Page page) {
List<String> listnv = new ArrayList<>();
//判斷串列尋址中包含...則需要根據收src和尾部src數字生成List
if (page.getHtml().css("div#listnav").css("ul").xpath("//li/text()").all().contains("...")) {
//構造條件【/freesecret?proxy，in_hp&sort=&pag，1】
final String[] origin = page.getHtml().css("div#listnav").css("ul").xpath("//li//a/@href").all().get(0).toString().split("=");
// 【/freesecret?proxy，in_hp&sort=&pag]
final String[] urlBody = Arrays.copyOf(origin,origin.length-1);
final String[] firstElement = page.getHtml().css("div#listnav").css("ul").xpath("//li//a/@href").all().get(0).toString().split("=");
//獲取1 這個id
int firstId = Integer.parseInt(firstElement[firstElement.length-1]);
final String[] lastElement = page.getHtml().css("div#listnav").css("ul").xpath("//li//a/@href").all().get(page.getHtml().css("div#listnav").css("ul").xpath("//li//a/@href").all().size() - 1).toString().split("=");
//獲取最后一個48 這個id
int lastId = Integer.parseInt(lastElement[lastElement.length-1]);
// 使用firstId、lastId構造for生成全部的url
for (int i = firstId; i <= lastId ; i++) {
listnv.add("https://proxy.mimvp.com/"+StringUtils.join(urlBody,"=")+"="+i);
}
return listnv;
}
//沒有...說明頁面少，直接提取生成完整url
for (String url:page.getHtml().css("div#listnav").css("ul").xpath("//li//a/@href").all()){
listnv.add("https://proxy.mimvp.com/"+url);
}
return listnv;
}
public List<String> fixSmileUrl(Page page){
List<String> smileurl = new ArrayList<>();
for (String url:page.getHtml().xpath("//div[@class='free-httptype-tabs']//a/@href").all()){
smileurl.add("https://proxy.mimvp.com/"+url);
}
return smileurl;
}
@Override
public Site getSite() {
return site;
}
}
package icu.smile.proxy.task;
import icu.smile.proxy.entity.ProxyMiPu;
import icu.smile.proxy.service.MiPuService;
import lombok.extern.java.Log;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import us.codecraft.webmagic.ResultItems;
import us.codecraft.webmagic.Task;
import us.codecraft.webmagic.pipeline.Pipeline;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Logger;
/**
* <p>
* 持久化層
* </p>
*
* @author starrysky
* @since 2021/6/6
*/
@Component
@Log
public class MiPuPipeline implements Pipeline {
@Autowired
private MiPuService miPuService;
private volatile static int curIndex = 0;
private static final Logger LOGGER = Logger.getLogger(MiPuPipeline.class.getName());
@Override
public void process(ResultItems resultItems, Task task) {
final ProxyMiPu proxyMiPu = new ProxyMiPu();
final List<String> ip = resultItems.get("ip");
final List<Integer> port = resultItems.get("port");
final List<String> type = resultItems.get("type");
final List<String> anonymous = resultItems.get("anonymous");
final List<String> location = resultItems.get("location");
final List<String> operator = resultItems.get("operator");
final List<String> responseTime = resultItems.get("responseTime");
final List<String> transmissionTime = resultItems.get("transmissionTime");
final List<String> verificationTime = resultItems.get("verificationTime");
final List<ProxyMiPu> entityList = new ArrayList<>();
if (ip.size() == 0 || port.size() == 0) {
return;
}
for (int i = curIndex; i <= ip.size() - 1; i++) {
proxyMiPu.setIp(ip.get(i))
.setPort(port.get(i))
.setType(type.size() == 0 ? "頁面無型別描述" : type.get(i))
.setAnonymous(anonymous.size() == 0 ? "頁面無描述" : anonymous.get(i))
.setLocation(location.size() == 0 ? "頁面無地址描述" : location.get(i))
.setOperator(operator.size() == 0 ? "頁面無運營商描述" : operator.get(i))
.setResponseTime(responseTime.size() == 0 ? "頁面無回應時間描述" : responseTime.get(i))
.setTransmissionTime(transmissionTime.size() == 0 ? "頁面無傳輸時間描述" : transmissionTime.get(i))
.setVerificationTime(verificationTime.size() == 0 ? "頁面無驗證時間描述" : verificationTime.get(i));
entityList.add(proxyMiPu);
LOGGER.info(proxyMiPu.toString());
}
miPuService.saveAll(entityList);
curIndex = 0;
}
}
SpringBoot啟動類：
package icu.smile;
import icu.smile.proxy.task.MiPuPageProcessor;
import icu.smile.proxy.task.MiPuPipeline;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.downloader.selenium.SeleniumDownloader;
import us.codecraft.webmagic.scheduler.BloomFilterDuplicateRemover;
import us.codecraft.webmagic.scheduler.QueueScheduler;
/**
* <p>
* 爬去米撲代理
* </p>
*
* @author starrysky
* @since 2021/6/6
*/
@SpringBootApplication
public class MiPuProxyApplication {
private static String URL = "https://proxy.mimvp.com/freeopen";
public static void main(String[] args) {
final ConfigurableApplicationContext ctx = SpringApplication.run(MiPuProxyApplication.class, args);
System.setProperty("selenuim_config","src/main/resources/config.ini");
Spider.create(new MiPuPageProcessor())
.addUrl(URL)
.setDownloader(new SeleniumDownloader())
.addPipeline(ctx.getBean(MiPuPipeline.class))
.setScheduler(new QueueScheduler().setDuplicateRemover(new BloomFilterDuplicateRemover(10000000)))
.thread(1).runAsync();
}
}
resources資源：
application.yml:
server:
port: 8849
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://47.111.237.28:3388/mipuproxy?characterEncoding=utf8&useSSL=false&serverTimezone=Asia/Shanghai
username: root
password: root
type: com.alibaba.druid.pool.DruidDataSource
# 連接池配置
druid:
# 初始化連接池連接數量
initial-size: 5
# 最小連接數量
min-idle: 5
# 最大連接池數量
max-active: 20
# 配置獲取練級等待超時時間
max-wait: 60000
# 配置間隔多久進行一次檢測時間，檢測時需要關閉空閑時間，單位為毫秒
time-between-eviction-runs-millis: 60000
# 配置連接池最小生存時間
min-evictable-idle-time-millis: 30000
validation-query: SELECT 1 FROM DUAL
test-while-idle: true
test-on-borrow: true
test-on-return: false
# 是否快取preparedStatement，也就是PSCache 官方建議MySQL下建議關閉 個人建議如果想用SQL防火墻 建議打開
pool-prepared-statements: true
max-pool-prepared-statement-per-connection-size: 20
# 配置監控統計攔截的filters，去掉后監控界面sql無法統計，'wall'用于防火墻
filter:
stat:
merge-sql: true
slow-sql-millis: 5000
# 基礎監控配置
web-stat-filter:
enabled: true
url-pattern: /*
# 設定不統計哪些URL
exclusions: "*.js,*.gif,*.jpg,*.png,*.css,*.ico,/druid/*"
session-stat-enable: true
session-stat-max-count: 100
stat-view-servlet:
enabled: true
url-pattern: /druid/*
reset-enable: true
# 設定監控頁面的登錄名和密碼
login-username: admin
login-password: admin
# 允許訪問的IP
allow: 127.0.0.1
# 不允許訪問的IP
#deny: 192.168.1.100
default-auto-commit: true
jpa:
database: mysql
show-sql: true
open-in-view: false
hibernate:
naming:
physical-strategy: org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
devtools:
restart:
enabled: true
selenium.ini:
driver=chrome
chrome_exec_path=/usr/local/bin/chromedriver
safari_driver_loglevel=DEBUG
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/413374.html
標籤：其他