本文主要介紹 Filebeat 的相關概念及基本使用,相關的環境及軟體資訊如下:CentOS 7.9、FileBeat 8.2.2、Logstash 8.2.2、Elasticsearch 8.2.2,
1、Filebeat 簡介
1.1、Filebeat 總覽
Filebeat 是用于轉發和集中日志資料的輕量級傳送程式, 作為服務器上的代理安裝,Filebeat 監視你指定的日志檔案或位置,收集日志事件,并將它們轉發到 Elasticsearch 或 Logstash 以進行索引,
Filebeat 具有如下的一些特性:
- 正確處理日志旋轉:針對每隔一個時間段生產一個新的日志的案例,Filebeat 可以幫我們正確地處理新生產的日志,并重新啟動對新生成日志的處理
- 背壓敏感:如果日志生成的速度過快,從而導致 Filebeat 生產的速度超過 Elasticsearch 處理的速度,那么 Filebeat 可以自動調節處理的速度,以達到 Elasticsearch 可以處理的范圍內
- “至少一次”保證:每個日志生成的事件至少被處理一次
- 結構化日志:可以處理結構化的日志資料資料
- 多行事件:如果一個日志有多行資訊,也可以被正確處理,比如錯誤資訊往往是多行資料
- 條件過濾:可以有條件地過濾一些事件
Filebeat 的作業方式如下:啟動 Filebeat 時,它將啟動一個或多個輸入,這些輸入將在為日志資料指定的位置中查找, 對于 Filebeat 所找到的每個日志,Filebeat 都會啟動收集器(havester), 每個收集器都讀取一個日志以獲取新內容,并將新日志資料發送到 libbeat,libbeat 會匯總事件,并將匯總的資料發送到為 Filebeat 配置的輸出,

從上面有可以看出來在 spooler 里有一些快取,這個可以用于重新發送以確保至少一次的事件消費,同時也可以用于背壓敏感,一旦 Filebeat 生成的事件的速度超過 Elasticsearch 能夠處理的極限,這個快取可以用于存盤一些事件,每個 Filebeat 可以配置多個 input,并且每個 input 可以配置來采集一個或多個檔案路徑的檔案, 就像上面的圖顯示的那樣,Filebeat 支持多種輸入方式,Filbeat 支持如下的一些輸出:
- Elasticsearch
- Logstash
- Kafka
- Redis
- File
- Console
- Cloud
1.2、Filebeat 模塊
一個 Filebeat 模塊通常由如下的部分組成:

Filebeat 模塊簡化了常見日志格式的收集,決議和可視化,檔案集包含以下內容:
- Filebeat 輸入配置,其中包含在其中查找日志檔案的默認路徑, 這些默認路徑取決于作業系統, Filebeat 配置還負責在需要時將多行事件縫合在一起,
- Elasticsearch Ingest Node 管道定義,用于決議日志行,
- 欄位定義,用于為每個欄位配置具有正確的 Elasticsearch 型別,它們還包含每個欄位的簡短說明,
- 示例 Kibana 儀表板(如果有)可用于可視化日志檔案,
Filebeat 會根據你的環境自動調整這些配置,并將它們加載到相應的 Elastic Stack 組件中,
一個典型的模塊(例如,對于 Nginx 日志)由一個或多個檔案集(對于 Nginx,訪問和錯誤日志)組成,比如,Nginx 模塊決議 NGINX HTTP 服務器創建的訪問和錯誤日志,它在幕后執行如下的一些任務:
- 設定日志檔案的默認路徑(你可以更改)
- 確保每個多行日志事件都作為單個事件發送
- 使用 ingest node 來決議和處理日志行
- 將資料塑造成適合在 Kibana 中進行可視化的結構
- 部署儀表板以可視化日志資料
針對其它的 Beats 模塊來說,基本和 Filebeat 一樣,目前針對 Elasticsearch 所提供的模塊來說,有非常多的模塊可以供使用:

關于 Filbeat 的模塊資訊,可查看官網檔案:https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html,
2、Filebeat 安裝
根據環境下載對應的安裝包:https://www.elastic.co/cn/downloads/beats/filebeat,這里選擇 Linux x86_64 版本;下載完成后在服務器上解壓即可:
tar zxvf filebeat-8.2.2-linux-x86_64.tar.gz
3、Filebeat 使用
這里使用 Filebeat 來收集 Nginx 訪問日志并發送到 Logstash 和 Elasticsearch 中,
3.1、按照普通日志檔案方式收集 Nginx 訪問日志
3.1.1、配置
修改組態檔 filebeat.yml,配置輸入、輸出,開始可以先把輸出配成 console,除錯沒問題后再配置輸出到 Logstash 和 Elasticsearch,
filebeat.inputs: - type: filestream id: my-filestream-id enabled: true paths: - /home/hadoop/app/nginx-1.8.0/logs/access.log output.console: pretty: true enable: true
3.1.2、啟動
./filebeat -e -c filebeat.yml
3.1.3、驗證
訪問 Nginx,日志檔案 /home/hadoop/app/nginx-1.8.0/logs/access.log 會輸出日志:
10.49.196.1 - - [16/Sep/2022:15:35:20 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
filebeat 的控制臺列印采集的日志資訊如下:
{ "@timestamp": "2022-09-16T07:35:25.797Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "8.2.2", "pipeline": "filebeat-8.2.2-nginx-access-pipeline" }, "event": { "timezone": "+08:00", "module": "nginx", "dataset": "nginx.access" }, "fileset": { "name": "access" }, "ecs": { "version": "1.12.0" }, "agent": { "type": "filebeat", "version": "8.2.2", "ephemeral_id": "5f0443a4-e5c6-43c0-88d2-4ad7b946c4a7", "id": "197bfd49-e03a-416e-b53f-4ac143b94fa5", "name": "pxc2" }, "message": "10.49.196.1 - - [16/Sep/2022:15:35:20 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"", "input": { "type": "log" }, "host": { "id": "a91a3eb89492470398ed77cd24a29299", "containerized": false, "ip": [ "10.49.196.11", "fe80::2011:ba1b:b32b:b63a" ], "mac": [ "00:0c:29:97:29:ce" ], "hostname": "pxc2", "name": "pxc2", "architecture": "x86_64", "os": { "version": "7 (Core)", "family": "redhat", "name": "CentOS Linux", "kernel": "3.10.0-957.el7.x86_64", "codename": "Core", "type": "linux", "platform": "centos" } }, "log": { "offset": 16308, "file": { "path": "/home/hadoop/app/nginx-1.8.0/logs/access.log" } }, "service": { "type": "nginx" } }View Code
3.1.4、處理器
默認情況下配置了如下處理器:
processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
如果覺得輸出的日志過多,可以把這些處理器去掉并添加去除欄位的處理器:
processors: #- add_host_metadata: # when.not.contains.tags: forwarded #- add_cloud_metadata: ~ #- add_docker_metadata: ~ #- add_kubernetes_metadata: ~ - drop_fields: fields: ["agent.id", "log"]
3.1.5、配置輸出到 Logstash
output.logstash: hosts: ["localhost:5044"]
Logstash 接受到的日志資訊如下:
{ "tags" => [ [0] "beats_input_codec_plain_applied" ], "event" => { "timezone" => "+08:00", "original" => "10.49.196.1 - - [16/Sep/2022:15:51:38 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"", "dataset" => "nginx.access", "module" => "nginx" }, "log" => { "offset" => 16866, "file" => { "path" => "/home/hadoop/app/nginx-1.8.0/logs/access.log" } }, "@version" => "1", "fileset" => { "name" => "access" }, "message" => "10.49.196.1 - - [16/Sep/2022:15:51:38 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"", "@timestamp" => 2022-09-16T07:51:45.515Z, "agent" => { "name" => "pxc2", "type" => "filebeat", "ephemeral_id" => "9a6a130b-04c6-4bbf-a228-60e1f237f5a9", "version" => "8.2.2", "id" => "197bfd49-e03a-416e-b53f-4ac143b94fa5" }, "ecs" => { "version" => "1.12.0" }, "host" => { "name" => "pxc2" }, "service" => { "type" => "nginx" }, "input" => { "type" => "log" } }View Code
3.1.6、配置輸出到 Elasticsearch
output.elasticsearch: hosts: ["localhost:9200"]
Elasticsearch 會自動創建名為 filebeat-8.2.2 的資料流,資料流中保存的日志資訊如下:
{ "_index": ".ds-filebeat-8.2.2-2022.09.16-000001", "_id": "8qxKRYMBPnCOyxVi1GuP", "_version": 1, "_score": 1, "_source": { "agent": { "name": "pxc2", "id": "197bfd49-e03a-416e-b53f-4ac143b94fa5", "ephemeral_id": "a6b1d9a5-8485-4391-92be-6a6ae530a5cd", "type": "filebeat", "version": "8.2.2" }, "nginx": { "access": { "remote_ip_list": [ "10.49.196.1" ] } }, "log": { "file": { "path": "/home/hadoop/app/nginx-1.8.0/logs/access.log" }, "offset": 17052 }, "source": { "address": "10.49.196.1", "ip": "10.49.196.1" }, "fileset": { "name": "access" }, "url": { "path": "/", "original": "/" }, "input": { "type": "log" }, "@timestamp": "2022-09-16T07:54:24.000Z", "ecs": { "version": "1.12.0" }, "_tmp": {}, "related": { "ip": [ "10.49.196.1" ] }, "service": { "type": "nginx" }, "host": { "name": "pxc2" }, "http": { "request": { "method": "GET" }, "response": { "status_code": 304, "body": { "bytes": 0 } }, "version": "1.1" }, "event": { "ingested": "2022-09-16T07:54:34.439241175Z", "original": "10.49.196.1 - - [16/Sep/2022:15:54:24 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"", "timezone": "+08:00", "created": "2022-09-16T07:54:33.213Z", "kind": "event", "module": "nginx", "category": [ "web" ], "type": [ "access" ], "dataset": "nginx.access", "outcome": "success" }, "user_agent": { "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "name": "Chrome", "device": { "name": "Other" }, "version": "105.0.0.0" } }, "fields": { "event.category": [ "web" ], "user_agent.os.full": [ "Windows 10" ], "user_agent.original.text": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" ], "url.original.text": [ "/" ], "source.address": [ "10.49.196.1" ], "user_agent.os.name.text": [ "Windows" ], "user_agent.os.version": [ "10" ], "user_agent.os.name": [ "Windows" ], "traefik.access.user_agent.name": [ "Chrome" ], "service.type": [ "nginx" ], "agent.type": [ "filebeat" ], "event.module": [ "nginx" ], "http.request.method": [ "GET" ], "related.ip": [ "10.49.196.1" ], "traefik.access.user_agent.original": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" ], "source.ip": [ "10.49.196.1" ], "agent.name": [ "pxc2" ], "host.name": [ "pxc2" ], "user_agent.version": [ "105.0.0.0" ], "http.response.status_code": [ 304 ], "http.version": [ "1.1" ], "event.kind": [ "event" ], "event.timezone": [ "+08:00" ], "event.outcome": [ "success" ], "user_agent.original": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" ], "event.original": [ "10.49.196.1 - - [16/Sep/2022:15:54:24 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"" ], "fileset.name": [ "access" ], "nginx.access.remote_ip_list": [ "10.49.196.1" ], "input.type": [ "log" ], "log.offset": [ 17052 ], "user_agent.name": [ "Chrome" ], "agent.hostname": [ "pxc2" ], "http.response.body.bytes": [ 0 ], "traefik.access.user_agent.os_name": [ "Windows" ], "user_agent.os.full.text": [ "Windows 10" ], "event.ingested": [ "2022-09-16T07:54:34.439Z" ], "url.original": [ "/" ], "@timestamp": [ "2022-09-16T07:54:24.000Z" ], "url.path": [ "/" ], "agent.id": [ "197bfd49-e03a-416e-b53f-4ac143b94fa5" ], "ecs.version": [ "1.12.0" ], "event.type": [ "access" ], "log.file.path": [ "/home/hadoop/app/nginx-1.8.0/logs/access.log" ], "event.created": [ "2022-09-16T07:54:33.213Z" ], "agent.ephemeral_id": [ "a6b1d9a5-8485-4391-92be-6a6ae530a5cd" ], "agent.version": [ "8.2.2" ], "user_agent.device.name": [ "Other" ], "event.dataset": [ "nginx.access" ] } }View Code
3.2、使用 Nginx 模塊收集 Nginx 訪問日志
Filebeat 中 Nginx 模塊可以針對 Nginx 日志進行決議,簡化了我們處理日志的程序,
3.2.1、配置
A、filebeat.yml 檔案配置
filebeat.inputs: - type: filestream id: my-filestream-id enabled: false paths: - /home/hadoop/app/nginx-1.8.0/logs/access.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false output.elasticsearch: hosts: ["localhost:9200"]
B、modules.d/nginx.yml 檔案配置
- module: nginx access: enabled: true var.paths: ["/home/hadoop/app/nginx-1.8.0/logs/access.log"] error: enabled: false ingress_controller: enabled: false
3.2.2、啟用 Nginx 模塊并啟動 Filebeat
./filebeat modules enable nginx
./filebeat -e -c filebeat.yml
3.2.3、驗證
訪問 Nginx 后,Elasticsearch 會自動創建名為 filebeat-8.2.2 的資料流,資料流中保存的日志資訊如下:
{ "_index": ".ds-filebeat-8.2.2-2022.09.16-000001", "_id": "-qxgRYMBPnCOyxVikGvO", "_version": 1, "_score": 1, "_source": { "agent": { "name": "pxc2", "id": "197bfd49-e03a-416e-b53f-4ac143b94fa5", "type": "filebeat", "ephemeral_id": "c7505853-b6de-46b0-abb0-7727dfb37d4b", "version": "8.2.2" }, "nginx": { "access": { "remote_ip_list": [ "10.49.196.1" ] } }, "log": { "file": { "path": "/home/hadoop/app/nginx-1.8.0/logs/access.log" }, "offset": 17424 }, "source": { "address": "10.49.196.1", "ip": "10.49.196.1" }, "fileset": { "name": "access" }, "url": { "path": "/", "original": "/" }, "input": { "type": "log" }, "@timestamp": "2022-09-16T08:18:09.000Z", "ecs": { "version": "1.12.0" }, "_tmp": {}, "related": { "ip": [ "10.49.196.1" ] }, "service": { "type": "nginx" }, "host": { "name": "pxc2" }, "http": { "request": { "method": "GET" }, "response": { "status_code": 304, "body": { "bytes": 0 } }, "version": "1.1" }, "event": { "ingested": "2022-09-16T08:18:19.574911507Z", "original": "10.49.196.1 - - [16/Sep/2022:16:18:09 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"", "timezone": "+08:00", "created": "2022-09-16T08:18:18.539Z", "kind": "event", "module": "nginx", "category": [ "web" ], "type": [ "access" ], "dataset": "nginx.access", "outcome": "success" }, "user_agent": { "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "name": "Chrome", "device": { "name": "Other" }, "version": "105.0.0.0" } }, "fields": { "event.category": [ "web" ], "user_agent.os.full": [ "Windows 10" ], "user_agent.original.text": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" ], "url.original.text": [ "/" ], "source.address": [ "10.49.196.1" ], "user_agent.os.name.text": [ "Windows" ], "user_agent.os.version": [ "10" ], "user_agent.os.name": [ "Windows" ], "traefik.access.user_agent.name": [ "Chrome" ], "service.type": [ "nginx" ], "agent.type": [ "filebeat" ], "event.module": [ "nginx" ], "http.request.method": [ "GET" ], "related.ip": [ "10.49.196.1" ], "traefik.access.user_agent.original": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" ], "source.ip": [ "10.49.196.1" ], "agent.name": [ "pxc2" ], "host.name": [ "pxc2" ], "user_agent.version": [ "105.0.0.0" ], "http.response.status_code": [ 304 ], "http.version": [ "1.1" ], "event.kind": [ "event" ], "event.timezone": [ "+08:00" ], "event.outcome": [ "success" ], "user_agent.original": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" ], "event.original": [ "10.49.196.1 - - [16/Sep/2022:16:18:09 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\"" ], "fileset.name": [ "access" ], "nginx.access.remote_ip_list": [ "10.49.196.1" ], "input.type": [ "log" ], "log.offset": [ 17424 ], "user_agent.name": [ "Chrome" ], "agent.hostname": [ "pxc2" ], "http.response.body.bytes": [ 0 ], "traefik.access.user_agent.os_name": [ "Windows" ], "user_agent.os.full.text": [ "Windows 10" ], "event.ingested": [ "2022-09-16T08:18:19.574Z" ], "url.original": [ "/" ], "@timestamp": [ "2022-09-16T08:18:09.000Z" ], "url.path": [ "/" ], "agent.id": [ "197bfd49-e03a-416e-b53f-4ac143b94fa5" ], "ecs.version": [ "1.12.0" ], "event.type": [ "access" ], "log.file.path": [ "/home/hadoop/app/nginx-1.8.0/logs/access.log" ], "event.created": [ "2022-09-16T08:18:18.539Z" ], "agent.ephemeral_id": [ "c7505853-b6de-46b0-abb0-7727dfb37d4b" ], "agent.version": [ "8.2.2" ], "user_agent.device.name": [ "Other" ], "event.dataset": [ "nginx.access" ] } }View Code
可以看到日志資訊已經被決議出各個單獨的有用欄位,如:source.ip,url.path,user_agent.name 等等,
參考:https://blog.csdn.net/UbuntuTouch/article/details/104432643
轉載請註明出處,本文鏈接:https://www.uj5u.com/houduan/514266.html
標籤:其他
上一篇:day46-JDBC和連接池02
下一篇:C語言小白刷題
