Springboot+ELK實作日志查詢

日志查詢一直頭疼的問題就是日志檔案太大、請求資料縱橫交錯很不方便查找定位，最近公司難得的清閑就整合ELK可針對日志進行日期、介面地址、用戶查詢日志明細

在這里插入圖片描述

文章目錄

Springboot+ELK實作日志查詢
一、elasticsearch+logstash+filebeat+kibana安裝部署（統一版本:7.7.0）
- 1. docker 安裝部署elasticsearch
- - 下載鏡像
  - 創建掛載的目錄
  - 創建容器并啟動
  - 訪問http://ip:9200如下圖
- 2. docker 安裝部署logstash
- - 下載鏡像
  - 啟動logstash(拷貝資料到持久化目錄)
  - 修改組態檔中的elasticsearch地址
  - 重新啟動logstash
- 3. docker 安裝部署filebeat
- - 下載鏡像
  - 臨時啟動
  - 拷貝資料檔案
  - 編輯組態檔(輸出到logstash)
  - 重新啟動filebeat
  - 在logstash中添加filebeat組態檔
- 4. docker 安裝kibana查看日志
- - 下載鏡像
  - 組態檔
  - 啟動kibana
  - 訪問http://ip:5601
二、整合Springboot可視化查詢
- 1.pom.xml中添加依賴
- 2.ES配置類
- 3.資料查詢

一、elasticsearch+logstash+filebeat+kibana安裝部署（統一版本:7.7.0）

1. docker 安裝部署elasticsearch

下載鏡像

docker pull elasticsearch:7.7.0

創建掛載的目錄

mkdir -p /data/elk/elasticsearch/data
mkdir -p /data/elk/elasticsearch/config
mkdir -p /data/elk/elasticsearch/plugins
// 創建組態檔
echo "http.host: 0.0.0.0" >> /data/elk/elasticsearch/config/elasticsearch.yml

其中data是資料目錄掛載需要權限 chmod -R 777 /data/elk/elasticsearch/data

創建容器并啟動

docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \
-e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
-v /data/elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /dataelk//elasticsearch/data:/usr/share/elasticsearch/data \
-v /dataelk//elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-d elasticsearch:7.7.0

訪問http://ip:9200如下圖

在這里插入圖片描述

2. docker 安裝部署logstash

下載鏡像

docker pull logstash:7.7.0

啟動logstash(拷貝資料到持久化目錄)

docker run -d --name=logstash logstash:7.7.0
mkdir -p /data/elk/logstash/config/conf.d
docker cp logstash:/usr/share/logstash/config /data/elk/logstash/
docker cp logstash:/usr/share/logstash/data /data/elk/logstash/
docker cp logstash:/usr/share/logstash/pipeline /data/elk/logstash/
// 授予權限
chmod 777 -R /data/elk/logstash

修改組態檔中的elasticsearch地址

 vi /data/elk/logstash/config/logstash.yml

在這里插入圖片描述

重新啟動logstash

docker rm -f logstash

docker run \
--name logstash \
--restart=always \
-p 5044:5044 \
-p 9600:9600 \
-v /data/elk/logstash/config:/usr/share/logstash/config \
-v /data/elk/logstash/data:/usr/share/logstash/data \
-v /data/elk/logstash/pipeline:/usr/share/logstash/pipeline \
-d logstash:7.7.0

3. docker 安裝部署filebeat

安裝在應用服務器直接讀取log檔案

下載鏡像

	docker pull elastic/filebeat:7.7.0

臨時啟動

	docker run -d --name=filebeat elastic/filebeat:7.7.0

拷貝資料檔案

	docker cp filebeat:/usr/share/filebeat /data/
	chmod 777 -R /data/filebeat
	chmod go-w /data/filebeat/filebeat.yml

編輯組態檔(輸出到logstash)

vi /data/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages/*.log

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~

output.logstash:
  hosts: '192.168.247.181:5044'
  indices:
    - index: "filebeat-%{+yyyy.MM.dd}"

重新啟動filebeat

docker rm -f filebeat

/data/logs為專案日志存放目錄

docker run \
  --name=filebeat \
  --restart=always \
  -v /data/filebeat:/usr/share/filebeat \
  -v /data/logs:/var/log/messages \
  -d elastic/filebeat:7.7.0

在logstash中添加filebeat組態檔

日志格式決議可根據自己專案來變更，我這里把請求頭日志放在request索引下，請求程序日志放在backend索引下

// logback日志輸出格式 與grok 決議格式對應
<pattern>[%X{appName:-admin}] [%X{nodeName:-127.0.0.1}] [%d{DEFAULT}] [%p] [%t] [traceId:%X{traceId:-0}] [user:%X{userName:-root}] [ip:%X{ip:-127.0.0.1}] [%c:%L] %m%n</pattern>

vi /data/elk/logstash/config/conf.d/beat.conf

input {
    beats {
        port => 5044
    }
}

filter {

    mutate {
        copy => { "message" => "logmessage" }
    }

    grok { 
        match =>{
           "message" => [
                "\A\[%{NOTSPACE:moduleName}\]\s+\[%{NOTSPACE:nodeName}\]\s+\[%{DATA:datetime}]\s+\[%{NOTSPACE:level}]\s+\[%{NOTSPACE:thread}\]\s+\[traceId:%{NOTSPACE:traceId}\]\s+\[user:%{NOTSPACE:user}\]\s+\[ip:%{NOTSPACE:ip}\]\s+\[%{NOTSPACE:localhost}\]\s+%{GREEDYDATA:message}[\n|\t](?<exception>(?<exceptionType>^.*[Exception|Error]):\s+(?<exceptionMessage>(.)*?)[\n|\t](?<stacktrace>(?m)(.*)))",
                "\A\[%{NOTSPACE:moduleName}\]\s+\[%{NOTSPACE:nodeName}\]\s+\[%{DATA:datetime}]\s+\[%{NOTSPACE:level}]\s+\[%{NOTSPACE:thread}\]\s+\[traceId:%{NOTSPACE:traceId}\]\s+\[user:%{NOTSPACE:user}\]\s+\[ip:%{NOTSPACE:ip}\]\s+\[%{NOTSPACE:localhost}\]\s+(?<message>(?m).*)" 
          ]
        }
        overwrite => [ "message" ]
    }

    date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
        remove_field => ["timestamp"]
    }

    ruby {
        code => 'event.set("document_type", "request") if event.get("message").include?"requestUri"'
    }

    if [document_type] == 'request' {
      json {
        source => "message"
        skip_on_invalid_json => true
        remove_field => ["message"]
      }
    }

}

output {
      if [document_type] == 'request' {
        elasticsearch {
          hosts => "192.168.247.181:9200"
          index => "request-%{+YYYY.MM.dd}"
          sniffing => true
          manage_template => false
          template_overwrite => true
        }
      } else {
        elasticsearch {
          hosts => "192.168.247.181:9200"
          index => "backend-%{+YYYY.MM.dd}"
          sniffing => true
          manage_template => false
          template_overwrite => true
        }
      }
}

指定filebeats組態檔路徑
vi /data/elk/logstash/config/logstash.yml
// docker容器內部路徑
path.config: /usr/share/logstash/config/conf.d/*.conf
path.logs: /usr/share/logstash/logs
在這里插入圖片描述
重啟logstash

docker restart logstash

4. docker 安裝kibana查看日志

下載鏡像

docker pull kibana:7.7.0

組態檔

注意替換自己的elasticsearch ip

mkdir -p /data/elk/kibana/config/
vi /data/elk/kibana/config/kibana.yml
內容如下
#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://192.168.247.181:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true

啟動kibana

docker run -d \
  --name=kibana \
  --restart=always \
  -p 5601:5601 \
  -v /data/elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \
  kibana:7.7.0

訪問http://ip:5601

request*索引
在這里插入圖片描述

backend*索引

在這里插入圖片描述

二、整合Springboot可視化查詢

1.pom.xml中添加依賴

<dependency>
     <groupId>org.elasticsearch.client</groupId>
     <artifactId>elasticsearch-rest-high-level-client</artifactId>
     <version>7.1.0</version>
 </dependency>
 <dependency>
     <groupId>org.elasticsearch</groupId>
     <artifactId>elasticsearch</artifactId>
     <version>7.1.0</version>
 </dependency>

2.ES配置類

@Configuration
public class ElasticsearchConfiguration {

    private String host = "192.168.247.181";

    private int port = 9200;

    private int connTimeout = 3000;

    private int socketTimeout = 5000;

    private int connectionRequestTimeout = 500;

    @Bean(destroyMethod = "close", name = "client")
    public RestHighLevelClient initRestClient() {
        RestClientBuilder builder = RestClient.builder(new HttpHost(host, port))
                .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
                .setConnectTimeout(connTimeout)
                .setSocketTimeout(socketTimeout)
                .setConnectionRequestTimeout(connectionRequestTimeout));
        return new RestHighLevelClient(builder);
    }
}

3.資料查詢

	@PostMapping("searchRequest")
    public Map<String, Object> searchRequest(@RequestBody RequestLogForm param) throws Exception {
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        if (StringUtils.isNotBlank(param.getModuleName())) {
            boolQueryBuilder.must(QueryBuilders.termQuery("moduleName.keyword", param.getModuleName()));
        }
        if (StringUtils.isNotBlank(param.getRequestUri())) {
            boolQueryBuilder.must(QueryBuilders.matchPhraseQuery("requestUri", param.getRequestUri()));
        }
        if (StringUtils.isNotBlank(param.getTraceId())) {
            boolQueryBuilder.must(QueryBuilders.termQuery("traceId.keyword", param.getTraceId()));
        }
        if (StringUtils.isNotBlank(param.getUser())) {
            boolQueryBuilder.must(QueryBuilders.termQuery("user.keyword", param.getUser()));
        }
        if (param.getBeginTime() != null || param.getEndTime() != null) {
            RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("@timestamp");
            if (param.getBeginTime() != null) {
                rangeQueryBuilder.gte(new java.util.Date(param.getBeginTime().getTime() + 3600 * 1000));
            }
            if (param.getEndTime() != null) {
                rangeQueryBuilder.lte(new java.util.Date(param.getEndTime().getTime() + 3600 * 1000));
            }
            boolQueryBuilder.must(rangeQueryBuilder);
        }

        sourceBuilder.query(boolQueryBuilder);
        sourceBuilder.from((param.getPage() - 1) * param.getLimit());
        sourceBuilder.size(param.getLimit());

        sourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
        sourceBuilder.sort("@timestamp", SortOrder.DESC);

        SearchRequest searchRequest = new SearchRequest();
        searchRequest.source(sourceBuilder);
        // 該索引包含的日志每個請求一條記錄，所以篩選的記錄可以代表請求的呼叫情況
        searchRequest.indices("request-*");

        log.info("ES Query: \n{}", sourceBuilder.toString());
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        log.info("ES Response: \n{}", JSON.toJSONString(searchResponse));
        List<RequestLog> list = new ArrayList<>();
        for (SearchHit documentFields : searchResponse.getHits().getHits()) {
            RequestLog requestLog = JSON.parseObject(documentFields.getSourceAsString(), RequestLog.class);
            list.add(requestLog);
        }
        Map<String, Object> result = new HashMap<>(4);
        result.put("code", 0);
        result.put("message", "SUCCESS");
        result.put("data", list);
        result.put("count", searchResponse.getHits().getTotalHits().value);
        return result;
    }

	@GetMapping("searchBackend/{traceId}")
    public String searchBackend(@PathVariable String traceId) throws Exception {
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        boolQueryBuilder.must(QueryBuilders.termQuery("traceId.keyword", traceId));
        sourceBuilder.query(boolQueryBuilder);
        sourceBuilder.from(0);
        // ElasticSearch的最大查詢結果樹，跟ElasticSearch的配置保持一致
        sourceBuilder.size(10000);
        sourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
        sourceBuilder.sort("datetime.keyword", SortOrder.ASC);
        sourceBuilder.sort("log.offset", SortOrder.ASC);

        SearchRequest searchRequest = new SearchRequest();
        searchRequest.source(sourceBuilder);
        // 該索引包含的日志是每個請求執行程序中寫的日志，一個請求會有多條記錄
        searchRequest.indices("backend-*");

        log.info("ES Query: \n" + sourceBuilder.toString());
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        for (ShardSearchFailure ssf : searchResponse.getShardFailures()) {
            System.err.println(ssf.reason());
        }

        StringBuffer concatMessage = new StringBuffer();
        for (SearchHit documentFields : searchResponse.getHits().getHits()) {
            BackendLog backendLog = JSON.parseObject(documentFields.getSourceAsString(), BackendLog.class);
            concatMessage.append(backendLog.getDatetime() + " " + backendLog.getLocalhost() + " " + backendLog.getMessage() + "<br/>");
            concatMessage.append(" <br/>");
        }
        return concatMessage.toString();
    }

轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/297400.html

標籤：其他

上一篇：如何用MATLAB為小姐姐跳舞視頻增添另一個小姐姐跳舞進度條

下一篇：一篇帶你們認識tomcat！東西全在這里了

ELK+slf4j+springboot日志可視化查詢

Springboot+ELK實作日志查詢

文章目錄

一、elasticsearch+logstash+filebeat+kibana安裝部署 （統一版本:7.7.0）

1. docker 安裝部署elasticsearch

下載鏡像

創建掛載的目錄

創建容器并啟動

訪問http://ip:9200如下圖

2. docker 安裝部署logstash

下載鏡像

啟動logstash(拷貝資料到持久化目錄)

修改組態檔中的elasticsearch地址

重新啟動logstash

3. docker 安裝部署filebeat

下載鏡像

臨時啟動

拷貝資料檔案

編輯組態檔(輸出到logstash)

重新啟動filebeat

在logstash中添加filebeat組態檔

4. docker 安裝kibana查看日志

下載鏡像

組態檔

啟動kibana

訪問http://ip:5601

二、整合Springboot可視化查詢

1.pom.xml中添加依賴

2.ES配置類

3.資料查詢

一、elasticsearch+logstash+filebeat+kibana安裝部署（統一版本:7.7.0）