上篇的博客完成了Es的單臺服務器架設

https://blog.csdn.net/lengyer/article/details/110919426

一、安裝logstash

下載logstash-7.3.2的安裝包，版本要和Es的一樣

https://artifacts.elastic.co/downloads/logstash/logstash-7.3.2.tar.gz

解壓：

tar zxvf logstash-7.3.2.tar.gz

mv logstash-7.3.2 /etc/

配置logstash：

vim /etc/logstash-7.3.2/config/logstash-nginx.conf

input {
    file {
        path => "/var/log/nginx/access.log"
        type => "access.log"
        start_position => "beginning"
        stat_interval => "3"
    }

    file {
        path => "/var/log/nginx/access.log.1"
        type => "access.log"
        start_position => "beginning"
        stat_interval => "3"
    }

}

filter {
    if [type] == "access.log" {
    grok {
            patterns_dir => "/etc/logstash-7.3.2/patterns/nginx"
            match => { "message" => "%{NGINXACCESS}"}
         }

   date {
            match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss" ]
            target => "@timestamp"
        }

   geoip {
       source => "client_ip"
       target => "geoip"
       database => "/etc/logstash-7.3.2/db/GeoLite2-City.mmdb"
       add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
       add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
         }

   useragent {
        source => "user_agent"
        target => "use_agent"
             }

   urldecode {
        all_fields => true
             }

        mutate {
            gsub => ["user_agent","[\"]",""]
            convert => [ "response","integer" ]
            convert => [ "body_bytes_sent","integer" ]
            convert => [ "bytes_sent","integer" ]
            convert => [ "upstream_response_time","float" ]
            convert => [ "upstream_status","integer" ]
            convert => [ "request_time","float" ]
            convert => [ "port","integer" ]
            convert => [ "[geoip][coordinates]", "float"]
              }
    }
}

output {

    stdout{
    codec=>rubydebug
          }

    if [type] == "access.log" {
        elasticsearch {
           action => "index"
            hosts => ["ES的ip:9200"]
            index => "logstash-nginx-%{+YYYY.MM.dd.HH}"
        }
    }
}

mkdir /etc/logstash-7.3.2/db

mkdir /etc/logstash-7.3.2/patterns

下載GeoLite2-City.mmdb

谷歌資料庫，可以通過ip獲取到經緯度，

https://download.csdn.net/download/lengyer/13618925

解壓：

tar zxvf GeoLite2-City_20201208.tar.gz

放到指定檔案夾下：

cd GeoLite2-City_20200410
mv GeoLite2-City.mmdb /etc/logstash-7.3.2/db

Nginx的日志正則運算式

后面要通過kibana生成各種各樣的圖形，要先定好正則運算式，logstash才可以決議對應的欄位生成對應的資料，

vim /etc/nginx/nginx.conf

在http里加入：

        log_format my_log '$remote_addr [$time_local] "$request_time" '
                          ' "$http_host" "$request" "$http_referer" '
                          '"$status" "$bytes_sent" $request_body'
                          '"$http_user_agent" "$http_x_forwarded_for"';

cd /etc/logstash-7.3.2/patterns

放入nginx檔案

nginx檔案下載連接：

https://download.csdn.net/download/lengyer/13619314

Output里的內容

    stdout{
    codec=>rubydebug
          }

這段是logstash推送到elasticsearch會顯示出來推送了哪些欄位和內容

如果不需要可以刪掉

二、運行Logstash

cd /etc/logstash-7.3.2/

bin/logstash -f config/logstash-nginx.conf

三、檢查是不是運行成功

在logstash這會又推送界面

{
          "host" => "nginx-1",
    "@timestamp" => 2020-12-09T11:25:03.108Z,
      "@version" => "1",
       "message" => "24 +0800] \"0.278\"  \"-\" \"\\x03\\x00\\x00/*\\xE0\\x00\\x00\\x00\\x00\\x00Cookie: mstshash=Administr\" \"-\" \"400\" \"343\" -\"-\" \"-\"",
          "type" => "access.log",
          "tags" => [
        [0] "_grokparsefailure",
        [1] "_geoip_lookup_failure"
    ],
          "path" => "/var/log/nginx/access.log.1"
}

然后再elasticsearch-head里查看有沒有 logstash-nginx-當前時間的索引創建，