主頁 >  其他 > 基于Kafka+SparkStreaming+Hbase的實時數倉案例-計算榷訓

基于Kafka+SparkStreaming+Hbase的實時數倉案例-計算榷訓

2021-08-08 07:20:49 其他

一、基礎工程構建

創建父工程

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.atguigu.gmall2019.dw</groupId>
    <artifactId>gmall2019-dw</artifactId>
    <version>1.0-SNAPSHOT</version>
    <parent>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-parent</artifactId>
            <version>1.5.10.RELEASE</version>
            <relativePath/> <!-- lookup parent from repository -->
        </parent>

<properties>
    <spark.version>2.1.1</spark.version>
    <scala.version>2.11.8</scala.version>
    <log4j.version>1.2.17</log4j.version>
    <slf4j.version>1.7.22</slf4j.version>

    <fastjson.version>1.2.47</fastjson.version>
    <httpclient.version>4.5.5</httpclient.version>
    <httpmime.version>4.3.6</httpmime.version>

    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <java.version>1.8</java.version>
</properties>

<dependencies>
    <!--此處放日志包，所有專案都要參考-->
    <!-- 所有子專案的日志框架 -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>jcl-over-slf4j</artifactId>
        <version>${slf4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>${slf4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>${slf4j.version}</version>
    </dependency>
    <!-- 具體的日志實作 -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>${log4j.version}</version>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>${httpclient.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpmime</artifactId>
            <version>${httpmime.version}</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>${fastjson.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>
    </project>

創建common模塊

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>gmall2019-dw</artifactId>
        <groupId>com.atguigu.gmall2019.dw</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>gmall2019-common</artifactId>
    <dependencies>
            <dependency>
                <groupId>com.alibaba</groupId>
                <artifactId>fastjson</artifactId>
            </dependency>

            <dependency>
                <groupId>org.apache.httpcomponents</groupId>
                <artifactId>httpclient</artifactId>
            </dependency>
            <dependency>
                <groupId>org.apache.httpcomponents</groupId>
                <artifactId>httpmime</artifactId>
            </dependency>

        </dependencies>
    </project>

創建Mock模塊

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>gmall2019_dw</artifactId>
        <groupId>com.atguigu.gmall2019.dw</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>dw-mocker</artifactId>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->


        <dependency>
            <groupId>com.atguigu.gmall2019.dw</groupId>
            <artifactId>dw-common</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>

    </dependencies>

</project>

創建工具類

RandomDate

import java.util.Date;
import java.util.Random;

public class RandomDate {
    Long logDateTime =0L;//
    int maxTimeStep=0 ;


     public RandomDate (Date startDate , Date  endDate,int num) {

         Long avgStepTime = (endDate.getTime()- startDate.getTime())/num;
        this.maxTimeStep=avgStepTime.intValue()*2;
        this.logDateTime=startDate.getTime();

    }


     public  Date  getRandomDate() {
           int  timeStep = new Random().nextInt(maxTimeStep);
           logDateTime = logDateTime+timeStep;
            return new Date( logDateTime);
      }

}

RanOpt

public class RanOpt<T>{
        T value ;
        int weight;

        public RanOpt ( T value, int weight ){
            this.value=value ;
            this.weight=weight;
        }

      public T getValue() {
          return value;
      }

      public int getWeight() {
          return weight;
      }
  }

RandomOptionGroup

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class RandomOptionGroup<T> {

    int totalWeight=0;

    List<RanOpt> optList=new ArrayList();

    public   RandomOptionGroup(RanOpt<T>... opts) {
        for (RanOpt opt : opts) {
            totalWeight+=opt.getWeight();
            for (int i = 0; i <opt.getWeight() ; i++) {
                optList.add(opt);
            }

        }
    }

    public RanOpt<T> getRandomOpt() {
        int i = new Random().nextInt(totalWeight);
        return optList.get(i);
    }


    public static void main(String[] args) {
        RanOpt[] opts= {new RanOpt("zhang3",20),new RanOpt("li4",30),new RanOpt("wang5",50)};
        RandomOptionGroup randomOptionGroup = new RandomOptionGroup(opts);
        for (int i = 0; i <10 ; i++) {
            System.out.println(randomOptionGroup.getRandomOpt().getValue());
        }

    }

 }

RandomNum

import java.util.Random;

public class RandomNum {
    public static final  int getRandInt(int fromNum,int toNum){
       return   fromNum+ new Random().nextInt(toNum-fromNum+1);
    }
}

發送日志工具類 ：LogUploader

import org.apache.http.HttpEntity;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;

import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;

public class LogUploader {

    


    public static void sendLogStream(String log){
        try{
            //不同的日志型別對應不同的URL

            URL url  =new URL("http://logserver/log");

            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            //設定請求方式為post
            conn.setRequestMethod("POST");

            //時間頭用來供server進行時鐘校對的
            conn.setRequestProperty("clientTime",System.currentTimeMillis() + "");
            //允許上傳資料
            conn.setDoOutput(true);
            //設定請求的頭資訊,設定內容型別為JSON
            conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

            System.out.println("upload" + log);

            //輸出流
            OutputStream out = conn.getOutputStream();
            out.write(("logString="+log).getBytes());
            out.flush();
            out.close();
            int code = conn.getResponseCode();
            System.out.println(code);
        }
        catch (Exception e){
            e.printStackTrace();
        }
    }

}

日志生成類JsonMocker

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.atguigu.gmall.dw.mock.utils.*;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Random;

public class JsonMocker {

      int startupNum=100000;
     int eventNum=200000 ;

     RandomDate logDateUtil= null;


      RanOpt[] osOpts= {new RanOpt("ios",3),new RanOpt("andriod",7) };
      RandomOptionGroup<String> osOptionGroup= new RandomOptionGroup(osOpts);
    Date startTime= null;
    Date endTime= null;

    RanOpt[] areaOpts= {new RanOpt("beijing",10),
            new RanOpt("shanghai",10),new RanOpt("guangdong",20),new RanOpt("hebei",5),
            new RanOpt("heilongjiang",5),new RanOpt("shandong",5),new RanOpt("tianjin",5),
            new RanOpt("shan3xi",5),new RanOpt("shan1xi",5),new RanOpt("sichuan",5)
    };
    RandomOptionGroup<String>  areaOptionGroup= new RandomOptionGroup(areaOpts);

    String appId="gmall2019";

    RanOpt[] vsOpts= {new RanOpt("1.2.0",50),new RanOpt("1.1.2",15),
            new RanOpt("1.1.3",30),
            new RanOpt("1.1.1",5)
    };

    RandomOptionGroup<String>  vsOptionGroup= new RandomOptionGroup(vsOpts);

    RanOpt[] eventOpts= {new RanOpt("addFavor",10),new RanOpt("addComment",30),
            new RanOpt("addCart",20), new RanOpt("clickItem",40)
    };

    RandomOptionGroup<String>  eventOptionGroup= new RandomOptionGroup(eventOpts);

    RanOpt[] channelOpts= {new RanOpt("xiaomi",10),new RanOpt("huawei",20),
            new RanOpt("wandoujia",30), new RanOpt("360",20), new RanOpt("tencent",20)
            , new RanOpt("baidu",10), new RanOpt("website",10)
    };

    RandomOptionGroup<String>  channelOptionGroup= new RandomOptionGroup(channelOpts);

    RanOpt[] quitOpts= {   new RanOpt(true,20),new RanOpt(false,80)};

    RandomOptionGroup<Boolean>  isQuitGroup= new RandomOptionGroup(quitOpts);

    public JsonMocker( )  {

    }

    public JsonMocker(String startTimeString ,String endTimeString,int startupNum,int eventNum) {
        try {
            startTime= new SimpleDateFormat("yyyy-MM-dd").parse(startTimeString);
            endTime= new SimpleDateFormat("yyyy-MM-dd").parse(endTimeString);
        } catch (ParseException e) {
            e.printStackTrace();
        }
        logDateUtil= new RandomDate(startTime,endTime,startupNum+eventNum);
    }

    String initEventLog(String startLogJson){
            /*`type` string   COMMENT '日志型別',
             `mid` string COMMENT '設備唯一 表示',
            `uid` string COMMENT '用戶標識',
            `os` string COMMENT '作業系統',
            `appid` string COMMENT '應用id',
            `area` string COMMENT '地區' ,
            `evid` string COMMENT '事件id',
            `pgid` string COMMENT '當前頁',
            `npgid` string COMMENT '跳轉頁',
            `itemid` string COMMENT '商品編號',
            `ts` bigint COMMENT '時間',*/

        JSONObject startLog = JSON.parseObject(startLogJson);
         String mid= startLog.getString("mid");
         String uid=  startLog.getString("uid");
         String os= startLog.getString("os");
         String appid=this.appId;
          String area=startLog.getString("area");
          String evid = eventOptionGroup.getRandomOpt().getValue();
          int pgid = new Random().nextInt(50)+1;
          int npgid = new Random().nextInt(50)+1;
          int itemid = new Random().nextInt(50);
        //  long ts= logDateUtil.getRandomDate().getTime();

        JSONObject jsonObject = new JSONObject();
        jsonObject.put("type","event");
        jsonObject.put("mid",mid);
        jsonObject.put("uid",uid);
        jsonObject.put("os",os);
        jsonObject.put("appid",appid);
        jsonObject.put("area",area);
        jsonObject.put("evid",evid);
        jsonObject.put("pgid",pgid);
        jsonObject.put("npgid",npgid);
        jsonObject.put("itemid",itemid);
        return  jsonObject.toJSONString();
    }


    String initStartupLog( ){
            /*`type` string   COMMENT '日志型別',
             `mid` string COMMENT '設備唯一標識',
      `uid` string COMMENT '用戶標識',
      `os` string COMMENT '作業系統', ,
      `appId` string COMMENT '應用id', ,
     `vs` string COMMENT '版本號',
     `ts` bigint COMMENT '啟動時間', ,
     `area` string COMMENT '城市' */


        String mid= "mid_"+ RandomNum.getRandInt(1,500);
        String uid=""+ RandomNum.getRandInt(1,500);
        String os=osOptionGroup.getRandomOpt().getValue();
        String appid=this.appId;
        String area=areaOptionGroup.getRandomOpt().getValue();
        String vs = vsOptionGroup.getRandomOpt().getValue();
        //long ts= logDateUtil.getRandomDate().getTime();
        String ch=os.equals("ios")?"appstore": channelOptionGroup.getRandomOpt().getValue();



        JSONObject jsonObject = new JSONObject();
        jsonObject.put("type","startup");
        jsonObject.put("mid",mid);
        jsonObject.put("uid",uid);
        jsonObject.put("os",os);
        jsonObject.put("appid",appid);
        jsonObject.put("area",area);
        jsonObject.put("ch",ch);
        jsonObject.put("vs",vs);
        return  jsonObject.toJSONString();
    }

public static void genLog() {
    JsonMocker jsonMocker = new JsonMocker();
    jsonMocker.startupNum = 1000000;
    for (int i = 0; i < jsonMocker.startupNum; i++) {
        String startupLog = jsonMocker.initStartupLog();
        jsonMocker.sendLog(startupLog);
        while (!jsonMocker.isQuitGroup.getRandomOpt().getValue()) {
            String eventLog = jsonMocker.initEventLog(startupLog);
            jsonMocker.sendLog(eventLog);
        }
        try {
            Thread.sleep(20);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }


}

    public void sendLog(String log)   {
       LogUploader.sendLogStream(log);
    }

    public static void main(String[] args)  {
        genLog();
    }

}

二、日志采集系統構建

基于SpringBoot接收埠資料，轉發到Kakfa

springboot整合kafka

application.propeties

#============== kafka ===================
# 指定kafka 代理地址，可以多個
spring.kafka.bootstrap-servers=hadoop1:9092


# 指定訊息key和訊息體的編解碼方式
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

LogJsonController

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.web.bind.annotation.*;

import   com.atguigu.gmall.constant.GmallConstants;

@Slf4j
@RestController //Controller+responsebody
public class LoggerController {

    @Autowired
    KafkaTemplate<String,String> kafkaTemplate;
    
    @PostMapping("log")
    public String doLog(@RequestParam("logString") String logString ){

        // 0 補充時間戳
        JSONObject jsonObject = JSON.parseObject(logString);
        jsonObject.put("ts",System.currentTimeMillis());
        // 1 落盤 file
        String jsonString = jsonObject.toJSONString();
        log.info(jsonObject.toJSONString());


        // 2 推送到kafka
        if( "startup".equals( jsonObject.getString("type"))){
            kafkaTemplate.send(GmallConstants.KAFKA_TOPIC_STARTUP,jsonString);
        }else{
            kafkaTemplate.send(GmallConstants.KAFKA_TOPIC_EVENT,jsonString);
        }

        return "success";
    }

}

在common模塊添加常量

public class GmallConstants {

    public static final String KAFKA_TOPIC_STARTUP="GMALL_STARTUP";
    public static final String KAFKA_TOPIC_EVENT="GMALL_EVENT";
    public static final String KAFKA_TOPIC_NEW_ORDER="GMALL_NEW_ORDER";
    public static final String KAFKA_TOPIC_ORDER_DETAIL="GMALL_ORDER_DETAIL";

    public static final String ES_INDEX_DAU="gmall2019_dau";
    public static final String ES_INDEX_NEW_MID="gmall2019_new_mid";
    public static final String ES_INDEX_NEW_ORDER="gmall2019_new_order";
    public static final String ES_INDEX_SALE_DETAIL="gmall2019_sale_detail";

}

增加log4j.properties

log4j.appender.atguigu.MyConsole=org.apache.log4j.ConsoleAppender
log4j.appender.atguigu.MyConsole.target=System.err
log4j.appender.atguigu.MyConsole.layout=org.apache.log4j.PatternLayout    
log4j.appender.atguigu.MyConsole.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %10p (%c:%M) - %m%n 

log4j.appender.atguigu.File=org.apache.log4j.DailyRollingFileAppender
log4j.appender.atguigu.File.file=d:/applog/gmall2019/log/app.log
log4j.appender.atguigu.File.DatePattern='.'yyyy-MM-dd
log4j.appender.atguigu.File.layout=org.apache.log4j.PatternLayout
log4j.appender.atguigu.File.layout.ConversionPattern=%m%n

log4j.logger.com.atguigu.xxxxxx.XXXXcontroller=info,atguigu.File,atguigu.MyConsole

打包發布到服務器

安裝Nginx負載均衡，修改組態檔：nginx.conf

http{
   ..........
    upstream logserver{
      server    hadoop1:8080 weight=1;  
      server    hadoop2:8080 weight=1;
      server    hadoop3:8080 weight=1;
 
    }
    server {
        listen       80;
        server_name  logserver;
 
        location / {
            root   html;
            index  index.html index.htm;
            proxy_pass http://logserver;
            proxy_connect_timeout 10;
 
         }
   ..........
}

三、實時處理，創建realtime模塊

消費kafka中的資料，
利用redis過濾當日已經計入的榷訓設備，
把每批次新增的當日榷訓資訊保存到HBASE或ES中，
從ES中查詢出資料，發布成資料介面，通可視化化工程呼叫，

3.1代碼開發之消費Kafka

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>gmall2019_dw</artifactId>
        <groupId>com.atguigu.gmall2019.dw</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>dw-realtime</artifactId>

    <dependencies>

        <dependency>
            <groupId>com.atguigu.gmall2019.dw</groupId>
            <artifactId>dw-common</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        </dependency>


        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.0</version>
        </dependency>

 
    

    </dependencies>
    <build>
        <plugins>
            <!-- 該插件用于將Scala代碼編譯成class檔案 -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <!-- 宣告系結到maven的compile階段 -->
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            
        </plugins>
    </build>
</project>

3.2、創建工具類

3.2.1 MykafkaUtil

import org.apache.kafka.common.serialization.StringDeserializer
import java.util.Properties

import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.dstream.InputDStream
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}

object MyKafkaUtil {
  private val properties: Properties = PropertiesUtil.load("config.properties")
  val broker_list = properties.getProperty("kafka.broker.list")

  // kafka消費者配置
  val kafkaParam = Map(
    "bootstrap.servers" -> broker_list,//用于初始化鏈接到集群的地址
    "key.deserializer" -> classOf[StringDeserializer],
    "value.deserializer" -> classOf[StringDeserializer],
    //用于標識這個消費者屬于哪個消費團體
    "group.id" -> "gmall_consumer_group",
    //如果沒有初始化偏移量或者當前的偏移量不存在任何服務器上，可以使用這個配置屬性
    //可以使用這個配置，latest自動重置偏移量為最新的偏移量
    "auto.offset.reset" -> "latest",
    //如果是true，則這個消費者的偏移量會在后臺自動提交,但是kafka宕機容易丟失資料
    //如果是false，會需要手動維護kafka偏移量
    "enable.auto.commit" -> (true: java.lang.Boolean)
  )

  // 創建DStream，回傳接收到的輸入資料
  // LocationStrategies：根據給定的主題和集群地址創建consumer
  // LocationStrategies.PreferConsistent：持續的在所有Executor之間分配磁區
  // ConsumerStrategies：選擇如何在Driver和Executor上創建和配置Kafka Consumer
  // ConsumerStrategies.Subscribe：訂閱一系列主題

  def getKafkaStream(topic: String,ssc:StreamingContext): InputDStream[ConsumerRecord[String,String]]={
    val dStream = KafkaUtils.createDirectStream[String,String](ssc, LocationStrategies.PreferConsistent,ConsumerStrategies.Subscribe[String,String](Array(topic),kafkaParam))
    dStream
  }
}

3.2.2 PropertiesUtil

object PropertiesUtil {

  def main(args: Array[String]): Unit = {
        val properties: Properties = PropertiesUtil.load("config.properties")

        println(properties.getProperty("kafka.broker.list"))
  }

   def load(propertieName:String): Properties ={
     val prop=new Properties();
     prop.load(new InputStreamReader(Thread.currentThread().getContextClassLoader.getResourceAsStream(propertieName) , "UTF-8"))
     prop
   }

}

3.3.3 RedisUtil

object RedisUtil {

  var jedisPool:JedisPool=null

  def getJedisClient: Jedis = {
    if(jedisPool==null){
//      println("開辟一個連接池")
      val config = PropertiesUtil.load("config.properties")
      val host = config.getProperty("redis.host")
      val port = config.getProperty("redis.port")

      val jedisPoolConfig = new JedisPoolConfig()
      jedisPoolConfig.setMaxTotal(100)  //最大連接數
      jedisPoolConfig.setMaxIdle(20)   //最大空閑
      jedisPoolConfig.setMinIdle(20)     //最小空閑
      jedisPoolConfig.setBlockWhenExhausted(true)  //忙碌時是否等待
      jedisPoolConfig.setMaxWaitMillis(500)//忙碌時等待時長 毫秒
      jedisPoolConfig.setTestOnBorrow(true) //每次獲得連接的進行測驗

      jedisPool=new JedisPool(jedisPoolConfig,host,port.toInt)
    }
//    println(s"jedisPool.getNumActive = ${jedisPool.getNumActive}")
 //   println("獲得一個連接")
    jedisPool.getResource
  }
}

3.2 制作case class Startuplog

case class StartUpLog(mid:String,
                      uid:String,
                      appid:String,
                      area:String,
                      os:String,
                      ch:String,
                      logType:String,
                      vs:String,
                      var logDate:String,
                      var logHour:String,
                      var ts:Long
                  ) {

}

3.4 業務類—消費kafka

object RealtimeStartupApp {

  def main(args: Array[String]): Unit = {
       val sparkConf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("gmall2019")
       val sc = new SparkContext(sparkConf)
       val ssc = new StreamingContext(sc,Seconds(10))

       val startupStream: InputDStream[ConsumerRecord[String, String]] = MyKafkaUtil.getKafkaStream(GmallConstants.KAFKA_TOPIC_STARTUP,ssc)

//       startupStream.map(_.value()).foreachRDD{ rdd=>
//         println(rdd.collect().mkString("\n"))
//       }

    val startupLogDstream: DStream[StartUpLog] = startupStream.map(_.value()).map { log =>
     // println(s"log = ${log}")
      val startUpLog: StartUpLog = JSON.parseObject(log, classOf[StartUpLog])
      startUpLog
    }
 }

3.5代碼開發2---去重

  1. 把今日新增的活躍用戶保存到redis中
  2. 每條資料經過過濾，去掉redis中已有的用戶
  3. 去掉本批次重復的用戶
import java.util 
import java.text.SimpleDateFormat
import java.util.Date

import com.alibaba.fastjson.JSON
import com.atguigu.gmall.constant.GmallConstants
import com.atguigu.gmall2019.realtime.bean.StartupLog
import com.atguigu.gmall2019.realtime.util.{MyKafkaUtil, RedisUtil}
import org.apache.hadoop.conf.Configuration
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.spark.SparkConf
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.dstream.{DStream, InputDStream}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import redis.clients.jedis.Jedis
import org.apache.phoenix.spark._

object DauApp {

  def main(args: Array[String]): Unit = {
     val sparkConf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("dau_app")
     val ssc = new StreamingContext(sparkConf,Seconds(5))
 // 1 消費kafka 
    val inputDstream: InputDStream[ConsumerRecord[String, String]] = MyKafkaUtil.getKafkaStream(GmallConstants.KAFKA_TOPIC_STARTUP,ssc)


    //2 資料流 轉換 結構變成case class 補充兩個時間欄位
    val startuplogDstream: DStream[StartupLog] = inputDstream.map { record =>
      val jsonStr: String = record.value()
      val startupLog: StartupLog = JSON.parseObject(jsonStr, classOf[StartupLog])

      val dateTimeStr: String = new SimpleDateFormat("yyyy-MM-dd HH").format(new Date(startupLog.ts))
      val dateArr: Array[String] = dateTimeStr.split(" ")
      startupLog.logDate = dateArr(0)
      startupLog.logHour = dateArr(1)
      startupLog
    }


    startuplogDstream.cache()


   // 3   利用用戶清單進行過濾 去重  只保留清單中不存在的用戶訪問記錄



    val filteredDstream: DStream[StartupLog] = startuplogDstream.transform { rdd =>
      val jedis: Jedis = RedisUtil.getJedisClient //driver //按周期執行
    val dateStr: String = new SimpleDateFormat("yyyy-MM-dd").format(new Date())

      val key = "dau:" + dateStr
      val dauMidSet: util.Set[String] = jedis.smembers(key)
      jedis.close()

      val dauMidBC: Broadcast[util.Set[String]] = ssc.sparkContext.broadcast(dauMidSet)
      println("過濾前：" + rdd.count())
      val filteredRDD: RDD[StartupLog] = rdd.filter { startuplog => //executor
        val dauMidSet: util.Set[String] = dauMidBC.value
        !dauMidSet.contains(startuplog.mid)
      }
      println("過濾后：" + filteredRDD.count())
      filteredRDD

    }

    // 4 批次內進行去重：：按照mid 進行分組，每組取第一個值
    val groupbyMidDstream: DStream[(String, Iterable[StartupLog])] = filteredDstream.map(startuplog=>(startuplog.mid,startuplog)).groupByKey()
    val distictDstream: DStream[StartupLog] = groupbyMidDstream.flatMap { case (mid, startupLogItr) =>
      startupLogItr.toList.take(1)
    }


   // 5 保存今日訪問過的用戶(mid)清單   -->Redis    1 key型別 ： set    2 key ： dau:2019-xx-xx   3 value : mid
    distictDstream.foreachRDD{rdd=>
      //driver
      rdd.foreachPartition{ startuplogItr=>
        val jedis:Jedis=RedisUtil.getJedisClient   //executor
        for (startuplog <- startuplogItr ) {
                  val key= "dau:"+startuplog.logDate
                  jedis.sadd(key,startuplog.mid)
                  println(startuplog)
        }
        jedis.close()
      }


    }

 


    ssc.start()
    ssc.awaitTermination()

  }

}

3.5保存到Hbase 中

添加依賴

<dependency>
    <groupId>org.apache.phoenix</groupId>
    <artifactId>phoenix-spark</artifactId>
    <version>4.14.2-HBase-1.3</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
</dependency>

通過phoenix保存到hbase

引入隱式轉換的包：
import org.apache.phoenix.spark._

//把資料寫入hbase+phoenix
distictDstream.foreachRDD{rdd=>
  rdd.saveToPhoenix("GMALL2019_DAU",Seq("MID", "UID", "APPID", "AREA", "OS", "CH", "TYPE", "VS", "LOGDATE", "LOGHOUR", "TS") ,new Configuration,Some("hadoop1,hadoop2,hadoop3:2181"))
}

四、SpringBoot框架開發榷訓資料查詢介面

1 訪問路徑

總數

http://publisher:8070/realtime-total?date=2019-02-01

分時統計

http://publisher:8070/realtime-hour?id=dau&date=2019-02-01

2 要求資料格式

總數

[{"id":"dau","name":"新增榷訓","value":1200},

{"id":"new_mid","name":"新增設備","value":233} ]

分時統計

{"yesterday":{"11":383,"12":123,"17":88,"19":200 },

"today":{"12":38,"13":1233,"17":123,"19":688 }}

5.1創建應用GmallPublisherApplication，添加掃描包

@SpringBootApplication
@MapperScan(basePackages = "com.atguigu.gmallXXXXXXX.publisher.mapper")
public class Gmall2019PublisherApplication{

  public static void main(String[] args) {
     SpringApplication.run(Gmall2019PublisherApplication.class, args);
  }

}

5.2 controller層

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.atguigu.gmall2019.dw.publisher.service.PublisherService;
import org.apache.commons.lang.time.DateUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.*;

@RestController
public class PublisherController {

  @Autowired
  PublisherService publisherService;

    @GetMapping("realtime-total")
    public String realtimeHourDate(@RequestParam("date") String date) {
         List<Map> list = new ArrayList<Map>();
         // 榷訓總數
        int dauTotal = publisherService.getDauTotal(date);
        Map dauMap=new HashMap<String,Object>();
        dauMap.put("id","dau");
        dauMap.put("name","新增榷訓");
        dauMap.put("value",dauTotal);
        list.add(dauMap);

        // 新增用戶
        int newMidTotal = publisherService.getNewMidTotal(date);
        Map newMidMap=new HashMap<String,Object>();
        newMidMap.put("id","new_mid");
        newMidMap.put("name","新增用戶");
        newMidMap.put("value",newMidTotal);
        list.add(newMidMap);

        return JSON.toJSONString(list);
    }


   @GetMapping("realtime-hours")
  public String realtimeHourDate(@RequestParam("id") String id,@RequestParam("date") String date){

      if( "dau".equals(id)){
          Map dauHoursToday = publisherService.getDauHours(date);
          JSONObject jsonObject = new JSONObject();
          jsonObject.put("today",dauHoursToday);
          String  yesterdayDateString="";
          try {
              Date dateToday = new SimpleDateFormat("yyyy-MM-dd").parse(date);
              Date dateYesterday = DateUtils.addDays(dateToday, -1);
              yesterdayDateString=new SimpleDateFormat("yyyy-MM-dd").format(dateYesterday);

          } catch (ParseException e) {
              e.printStackTrace();
          }
          Map dauHoursYesterday = publisherService.getDauHours(yesterdayDateString);
          jsonObject.put("yesterday",dauHoursYesterday);
          return jsonObject.toJSONString();
      }


      if( "new_order_totalamount".equals(id)){
          String newOrderTotalamountJson = publisherService.getNewOrderTotalAmountHours(date);
          return newOrderTotalamountJson;
      }
       return null;
  }

}

5.3 service層

public interface PublisherService {

public int getDauTotal(String date );

public Map getDauHours(String date );


}

5.4 service層實作類

@Service
public class PublisherServiceImpl implements PublisherService{

    @Autowired
DauMapper dauMapper;

@Override
public Integer getDauTotal(String date) {
    return dauMapper.selectDauTotal(date);
}

@Override
public Map getDauHour(String date) {
    HashMap dauHourMap=new HashMap();
    List<Map> dauHourList = dauMapper.selectDauTotalHourMap(date);
    for (Map map : dauHourList) {
        dauHourMap.put(map.get("LH"),map.get("CT"));
    }

    return dauHourMap;
}

}

5.5 資料層 mapper

import java.util.List;
import java.util.Map;
public interface DauMapper {

    public Integer selectDauTotal(String date);
    public List<Map> selectDauTotalHourMap(String date);
}

5.6 資料層 實作配置

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper SYSTEM "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
<mapper namespace="com.atguigu.gmall2019.publisher.mapper.DauMapper">
    <select id="selectDauTotal" resultType="Integer">
        select count(*) from gmall2019_dau where  logdate=#{date}
    </select>


    <select id="selectDauTotalHourMap" resultMap="dauTotalHour">
        select LOGHOUR lh, count(*) ct from gmall2019_dau where  LOGDATE=#{date}
        group by LOGHOUR
    </select>

<resultMap id="dauTotalHour" type="java.util.Map" autoMapping="true">
</resultMap>


</mapper>

六、附錄1

#啟動Kafka集群腳本
#! /bin/bash

case $1 in
"start"){
        for i in hadoop102 hadoop103 hadoop104
        do
                echo " --------啟動 $i Kafka-------"
                # 用于KafkaManager監控
                ssh $i "export JMX_PORT=9988 && /opt/module/kafka-default/bin/kafka-server-start.sh -daemon /opt/module/kafka-default/config/server.properties "
        done
};;
"stop"){
        for i in hadoop102 hadoop103 hadoop104
        do
                echo " --------停止 $i Kafka-------"
                ssh $i "/opt/module/kafka-default/bin/kafka-server-stop.sh stop"
        done
};;
esac



啟動：./kafka start
停止：./kafka stop


#啟動消費 消費日志
/opt/module/kafka-default/bin/kafka-console-consumer.sh --bootstrap-server  hadoop102:9092,hadoop103:9092,hadoop104:9092 --topic  GMALL_STARTUP
/opt/module/kafka-default/bin/kafka-console-consumer.sh --bootstrap-server  hadoop102:9092,hadoop103:9092,hadoop104:9092 --topic  GMALL_EVENT

#啟動SpringBoot
[deploy@hadoop102 gmall0513]$ java -jar /opt/module/gmall0513/gmall0513-logger-0.0.1-SNAPSHOT.jar --server.port=8090

#日志服務集群啟動腳本 logger-cluster.sh
#!/bin/bash
JAVA_BIN=/opt/module/jdk-default/bin/java
PROJECT=gmall0513
APPNAME=gmall0513-logger-0.0.1-SNAPSHOT.jar
SERVER_PORT=8090
 
case $1 in
 "start")
    {
		echo "========START NGINX==============="
		ssh hadoop103 "/opt/module/nginx/sbin/nginx"
        for i in hadoop102 hadoop103 hadoop104
        do
            echo "========start: $i============="
            ssh $i  "$JAVA_BIN -Xms32m -Xmx64m  -jar /opt/module/$PROJECT/$APPNAME --server.port=$SERVER_PORT >/dev/null 2>&1 &"
        done
        echo "========NGINX STARTED============="
        
    };;
  "stop")
    { 
        echo "========STOP NGINX==============="
        ssh hadoop103  "/opt/module/nginx/sbin/nginx  -s stop"
        for i in  hadoop102 hadoop103 hadoop104
        do
            echo "========stop: $i==============="
            ssh $i "ps -ef|grep $APPNAME |grep -v grep|awk '{print \$2}'|xargs kill" >/dev/null 2>&1
        done
		echo "========NGINX STOPED============="
    };;
    esac

七、附錄二，安裝啟動Phoenix

[deploy@hadoop102 phoenix-default]$ cp phoenix-server-hbase-2.2-5.1.2.jar /opt/module/hbase-default/lib/
[deploy@hadoop102 phoenix-default]$ cp phoenix-client-hbase-2.2-5.1.2.jar /opt/module/hbase-default/lib/  

[deploy@hadoop102 lib]$ xsync phoenix-server-hbase-2.2-5.1.2.jar

[deploy@hadoop102 lib]$ xsync  phoenix-client-hbase-2.2-5.1.2.jar

#PHOENIX
export PHOENIX_HOME=/opt/module/phoenix-default
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin

[root@hadoop102 ~]# xsync /etc/profile

JAVA_HOME=/opt/module/jdk-default
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
export PATH JAVA_HOME CLASSPATH
export JRE_HOME=$JAVA_HOME/jre

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-default
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_PREFIX=$HADOOP_HOME
export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar:${HADOOP_CLASSPATH}
#SCALA_HOME
export SCALA_HOME=/opt/module/scala-default
export PATH=$PATH:$SCALA_HOME/bin

#HIVE_HOME
export HIVE_HOME=/opt/module/hive-default
export PATH=$HIVE_HOME/bin:$PATH
export HADOOP_CLASSPATH=$HIVE_HOME/conf:$HIVE_HOME/lib

#SPARK_HOME
export SPARK_HOME=/opt/module/spark-default
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME

#SQOOP_HOME
export SQOOP_HOME=/opt/module/sqoop-default
export PATH=$SQOOP_HOME/bin:$PATH
export LOGDIR=$SQOOP_HOME/logs

#ES_HOME
export ES_HOME=/opt/module/elasticsearch-default
export PATH=$PATH:$ES_HOME/bin

#KIBANA_HOME
export ES_HOME=/opt/module/kibana-default
export PATH=$PATH:$KIBANA_HOME/bin
#ZK_HOME
export ES_HOME=/opt/module/zookeeper-default
export PATH=$PATH:$ZK_HOME/bin

#HBASE_HOME
export HBASE_HOME=/opt/module/hbase-default
export PATH=$PATH:$HBASE_HOME/bin:$HBASE_HOME/sbin

#AIRFLOW_HOME
export AIRFLOW_HOME=/home/deploy/airflow

#PRESTO_HOME
export PRESTO_HOME=/opt/module/presto-default
export PATH=$PATH:$PRESTO_HOME/bin

#FLINK_HOME
export FLINK_HOME=/opt/module/flink_default
export PATH=$PATH:$FLINK_HOME/bin

#PHOENIX
export PHOENIX_HOME=/opt/module/phoenix-default
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin

啟動Phoenix
sqlline.py hadoop104,hadoop105,hadoop106:2181

//master.HMaster: hbase:meta,,1.1588230740 is NOT online\\\\\\\\\\\\\\\\\
[deploy@hadoop104 bin]$ ./zkCli.sh -server  hadoop104:2181
[zk: hadoop104:2181(CONNECTED) 4] deleteall /hbase

轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/292303.html

標籤：其他

上一篇：從小白開始的人工智能學習（一）（入門機器學習、神經網路，python一路走來所經歷的坑。）

下一篇：【Package】RosBridge——打通Ros與非Ros環境的資料壁壘

標籤雲
其他(157675) Python(38076) JavaScript(25376) Java(17977) C(15215) 區塊鏈(8255) C＃(7972) AI(7469) 爪哇(7425) MySQL(7132) html(6777) 基礎類(6313) sql(6102) 熊猫(6058) PHP(5869) 数组(5741) R(5409) Linux(5327) 反应(5209) 腳本語言(PerlPython)(5129) 非技術區(4971) Android(4554) 数据框(4311) css(4259) 节点.js(4032) C語言(3288) json(3245) 列表(3129) 扑(3119) C++語言(3117) 安卓(2998) 打字稿(2995) VBA(2789) Java相關(2746) 疑難問題(2699) 细绳(2522) 單片機工控(2479) iOS(2429) ASP.NET(2402) MongoDB(2323) 麻木的(2285) 正则表达式(2254) 字典(2211) 循环(2198) 迅速(2185) 擅长(2169) 镖(2155) 功能(1967) .NET技术(1958) Web開發(1951) python-3.x(1918) HtmlCss(1915) 弹簧靴(1913) C++(1909) xml(1889) PostgreSQL(1872) .NETCore(1853) 谷歌表格(1846) Unity3D(1843) for循环(1842)

熱門瀏覽
  • 網閘典型架構簡述

    網閘架構一般分為兩種：三主機的三系統架構網閘和雙主機的2+1架構網閘。 三主機架構分別為內端機、外端機和仲裁機。三機無論從軟體和硬體上均各自獨立。首先從硬體上來看，三機都用各自獨立的主板、記憶體及存盤設備。從軟體上來看，三機有各自獨立的作業系統。這樣能達到完全的三機獨立。對于“2+1”系統，“2”分為 ......

    uj5u.com 2020-09-10 02:00:44 more
  • 如何從xshell上傳檔案到centos linux虛擬機里

    如何從xshell上傳檔案到centos linux虛擬機里及：虛擬機CentOs下執行 yum -y install lrzsz命令，出現錯誤：鏡像無法找到軟體包 前言 一、安裝lrzsz步驟 二、上傳檔案 三、遇到的問題及解決方案 總結 前言 提示：其實很簡單，往虛擬機上安裝一個上傳檔案的工具 ......

    uj5u.com 2020-09-10 02:00:47 more
  • 一、SQLMAP入門

    一、SQLMAP入門 1、判斷是否存在注入 sqlmap.py -u 網址/id=1 id=1不可缺少。當注入點后面的引數大于兩個時。需要加雙引號， sqlmap.py -u "網址/id=1&uid=1" 2、判斷文本中的請求是否存在注入 從文本中加載http請求，SQLMAP可以從一個文本檔案中 ......

    uj5u.com 2020-09-10 02:00:50 more
  • Metasploit 簡單使用教程

    metasploit 簡單使用教程 浩先生， 2020-08-28 16:18:25 分類專欄： kail 網路安全 linux 文章標簽： linux資訊安全 編輯 著作權 metasploit 使用教程 前言 一、Metasploit是什么？ 二、準備作業 三、具體步驟 前言 Msfconsole ......

    uj5u.com 2020-09-10 02:00:53 more
  • 游戲逆向之驅動層與用戶層通訊

    驅動層代碼： #pragma once #include <ntifs.h> #define add_code CTL_CODE(FILE_DEVICE_UNKNOWN,0x800,METHOD_BUFFERED,FILE_ANY_ACCESS) /* 更多游戲逆向視頻www.yxfzedu.com ......

    uj5u.com 2020-09-10 02:00:56 more
  • 北斗電力時鐘（北斗授時服務器）讓網路資料更精準

    北斗電力時鐘（北斗授時服務器）讓網路資料更精準 北斗電力時鐘（北斗授時服務器）讓網路資料更精準 京準電子科技官微——ahjzsz 近幾年，資訊技術的得了快速發展，互聯網在逐漸普及，其在人們生活和生產中都得到了廣泛應用，并且取得了不錯的應用效果。計算機網路資訊在電力系統中的應用，一方面使電力系統的運行 ......

    uj5u.com 2020-09-10 02:01:03 more
  • 【CTF】CTFHub 技能樹 彩蛋 writeup

    ?碎碎念 CTFHub：https://www.ctfhub.com/ 筆者入門CTF時時剛開始刷的是bugku的舊平臺，后來才有了CTFHub。 感覺不論是網頁UI設計，還是題目質量，賽事跟蹤，工具軟體都做得很不錯。 而且因為獨到的金幣制度的確讓人有一種想去刷題賺金幣的感覺。 個人還是非常喜歡這個 ......

    uj5u.com 2020-09-10 02:04:05 more
  • 02windows基礎操作

    我學到了一下幾點 Windows系統目錄結構與滲透的作用 常見Windows的服務詳解 Windows埠詳解 常用的Windows注冊表詳解 hacker DOS命令詳解（net user / type /md /rd/ dir /cd /net use copy、批處理 等） 利用dos命令制作 ......

    uj5u.com 2020-09-10 02:04:18 more
  • 03.Linux基礎操作

    我學到了以下幾點 01Linux系統介紹02系統安裝，密碼啊破解03Linux常用命令04LAMP 01LINUX windows： win03 8 12 16 19 配置不繁瑣 Linux：redhat,centos(紅帽社區版)，Ubuntu server,suse unix:金融機構，證券，銀 ......

    uj5u.com 2020-09-10 02:04:30 more
  • 05HTML

    01HTML介紹 02頭部標簽講解03基礎標簽講解04表單標簽講解 HTML前段語言 js1.了解代碼2.根據代碼 懂得挖掘漏洞 （POST注入/XSS漏洞上傳）3.黑帽seo 白帽seo 客戶網站被黑帽植入劫持代碼如何處理4.熟悉html表單 <html><head><title>TDK標題，描述 ......

    uj5u.com 2020-09-10 02:04:36 more
最新发布
  • 2023年最新微信小程式抓包教程

    01 開門見山 隔一個月發一篇文章，不過分。 首先回顧一下《微信系結手機號資料庫被脫庫事件》，我也是第一時間得知了這個訊息，然后跟蹤了整件事情的經過。下面是這起事件的相關截圖以及近日流出的一萬條資料樣本： 個人認為這件事也沒什么，還不如關注一下之前45億快遞資料查詢渠道疑似在近日復活的訊息。 訊息是 ......

    uj5u.com 2023-04-20 08:48:24 more
  • web3 產品介紹：metamask 錢包 使用最多的瀏覽器插件錢包

    Metamask錢包是一種基于區塊鏈技術的數字貨幣錢包，它允許用戶在安全、便捷的環境下管理自己的加密資產。Metamask錢包是以太坊生態系統中最流行的錢包之一，它具有易于使用、安全性高和功能強大等優點。 本文將詳細介紹Metamask錢包的功能和使用方法。 一、 Metamask錢包的功能 數字資 ......

    uj5u.com 2023-04-20 08:47:46 more
  • vulnhub_Earth

    前言 靶機地址->>>vulnhub_Earth 攻擊機ip：192.168.20.121 靶機ip：192.168.20.122 參考文章 https://www.cnblogs.com/Jing-X/archive/2022/04/03/16097695.html https://www.cnb ......

    uj5u.com 2023-04-20 07:46:20 more
  • 從4k到42k，軟體測驗工程師的漲薪史，給我看哭了

    清明節一過，盲猜大家已經無心上班，在數著日子準備過五一，但一想到銀行卡里的余額……瞬間心情就不美麗了。最近，2023年高校畢業生就業調查顯示，本科畢業月平均起薪為5825元。調查一出，便有很多同學表示自己又被平均了。看著這一資料，不免讓人想到前不久中國青年報的一項調查：近六成大學生認為畢業10年內會 ......

    uj5u.com 2023-04-20 07:44:00 more
  • 最新版本 Stable Diffusion 開源 AI 繪畫工具之中文自動提詞篇

    🎈 標簽生成器 由于輸入正向提示詞 prompt 和反向提示詞 negative prompt 都是使用英文，所以對學習母語的我們非常不友好 使用網址：https://tinygeeker.github.io/p/ai-prompt-generator 這個網址是為了讓大家在使用 AI 繪畫的時候 ......

    uj5u.com 2023-04-20 07:43:36 more
  • 漫談前端自動化測驗演進之路及測驗工具分析

    隨著前端技術的不斷發展和應用程式的日益復雜，前端自動化測驗也在不斷演進。隨著 Web 應用程式變得越來越復雜，自動化測驗的需求也越來越高。如今，自動化測驗已經成為 Web 應用程式開發程序中不可或缺的一部分，它們可以幫助開發人員更快地發現和修復錯誤，提高應用程式的性能和可靠性。 ......

    uj5u.com 2023-04-20 07:43:16 more
  • CANN開發實踐：4個DVPP記憶體問題的典型案例解讀

    摘要：由于DVPP媒體資料處理功能對存放輸入、輸出資料的記憶體有更高的要求（例如，記憶體首地址128位元組對齊），因此需呼叫專用的記憶體申請介面，那么本期就分享幾個關于DVPP記憶體問題的典型案例，并給出原因分析及解決方法。 本文分享自華為云社區《FAQ_DVPP記憶體問題案例》，作者：昇騰CANN。 DVPP ......

    uj5u.com 2023-04-20 07:43:03 more
  • msf學習

    msf學習 以kali自帶的msf為例 一、msf核心模塊與功能 msf模塊都放在/usr/share/metasploit-framework/modules目錄下 1、auxiliary 輔助模塊，輔助滲透（埠掃描、登錄密碼爆破、漏洞驗證等） 2、encoders 編碼器模塊，主要包含各種編碼 ......

    uj5u.com 2023-04-20 07:42:59 more
  • Halcon軟體安裝與界面簡介

    1. 下載Halcon17版本到到本地 2. 雙擊安裝包后 3. 步驟如下 1.2 Halcon軟體安裝 界面分為四大塊 1. Halcon的五個助手 1) 影像采集助手：與相機連接，設定相機引數，采集影像 2) 標定助手：九點標定或是其它的標定，生成標定檔案及內參外參，可以將像素單位轉換為長度單位 ......

    uj5u.com 2023-04-20 07:42:17 more
  • 在MacOS下使用Unity3D開發游戲

    第一次發博客，先發一下我的游戲開發環境吧。 去年2月份買了一臺MacBookPro2021 M1pro(以下簡稱mbp)，這一年來一直在用mbp開發游戲。我大致分享一下我的開發工具以及使用體驗。 1、Unity 官網鏈接： https://unity.cn/releases 我一般使用的Apple ......

    uj5u.com 2023-04-20 07:40:19 more

Copyright © 2010-2020 有解無憂