我正在使用 ELK 堆疊將 nginx 訪問日志保存到 elasticsearch。具體來說，我使用 filebeat 來收集它們并使用 logstash 來決議它們。我正在使用以下配置：
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
- /var/log/spring/geo/*.log
output.logstash:
enabled: true
hosts: ["logstash:5035"]
input {
beats {
port => 5035
}
}
filter {
grok {
match => [ "message" , "%{COMBINEDAPACHELOG} %{GREEDYDATA:http_x_forwarded_for}"]
}
grok {
match => [ "http_x_forwarded_for" , "%{IP:real_client_ip}"]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "nginx-geoip" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
useragent {
source => "message"
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
index => "weblogs-%{ YYYY.MM.dd}"
document_type => "nginx_logs"
user => "elastic"
password => "changeme"
}
stdout { codec => rubydebug }
}
但是，我注意到由于某種原因，并非所有日志都傳遞給 elasticsearch。例如，假設我有以下日志：
172.20.0.1 - - [17/Oct/2022:08:25:22 0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "111.111.111.111"
112.111.0.1 - - [17/Oct/2022:12:43:22 0000] "GET /favicon.ico HTTP/1.1" 404 150 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "-"
111.111.0.1 - - [17/Oct/2022:12:44:44 0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "111.111.111.111"
172.19.0.1 - - [17/Oct/2022:12:45:29 0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "78.87.79.206, 188.114.103.233"
172.18.0.1 - - [17/Oct/2022:12:46:29 0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "78.87.79.206, 188.114.103.233"
索引已創建，但112.111.0.1 - - [17/Oct/2022:12:43:22 0000] "GET /favicon.ico HTTP/1.1" 404 150 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "-"如果我通過開發工具查詢索引，則不會出現日志。知道是什么導致了錯誤嗎？
編輯：我正在使用的查詢如下：
GET weblogs-2022.10.17/_search
{
"size" : 100,
"query": {
"match_all" : {}
},
"sort" : [{"@timestamp":{"order": "desc"}}]
}
結果包括 4 個日志而不是 5 個，我得到的部分內容如下（我不能包括所有回傳，因為它非常大）：
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
}
uj5u.com熱心網友回復：
它不是索引，因為當前的 grok 模式與以下日志不匹配：
112.111.0.1 - - [17/Oct/2022:12:43:22 0000] "GET /favicon.ico HTTP/1.1" 404 150 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "-"
為什么不匹配？
因為它在啟動時在IP地址之后包含額外的空間。所有其他日志有 1 個空間，以上日志有 2 個空間。
您可以使用以下配置更新 logstash 中的第一個 grok 過濾器，它也會索引該日志。
grok {
match => [ "message" , "%{COMBINEDAPACHELOG} %{GREEDYDATA:http_x_forwarded_for}", "%{IPORHOST:clientip}%{SPACE}%{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{GREEDYDATA:http_x_forwarded_for}"]
}
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/517315.html