ELK日志分析系統
一.基礎環境配置：
1.ip
192.168.200.10 elasticsearch+kibana ELK-1
192.168.200.20 elasticsearch+logstash ELK-2
192.168.200.30 elasticsearch ELK-3
[root@localhost ~]# hostnamectl set-hostname elk-1
[root@localhost ~]# hostnamectl set-hostname elk-2
[root@localhost ~]# hostnamectl set-hostname elk-3
2.配置hosts[三臺]
[root@elk-1 ~]# vi /etc/hosts
[root@elk-1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.10 elk-1
192.168.200.20 elk-2
192.168.200.30 elk-3
3.安裝jdk[三臺]
[root@elk-1 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
……
[root@elk-1 ~]# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
二.elasticserach安裝
（1）安裝es
[root@elk-1 ~]# rpm -ivh elasticsearch-6.0.0.rpm
[root@elk-2 ~]# rpm -ivh elasticsearch-6.0.0.rpm
[root@elk-3 ~]# rpm -ivh elasticsearch-6.0.0.rpm
#i表示安裝，v表示顯示安裝程序，h表示顯示進度
（2）配置es
配置elasticsearch的組態檔，組態檔在/etc/elasticsearch/elasticsearch.yml
Elk-1節點：增加以下 //字樣
root@elk-1 ~]# vi /etc/elasticsearch/elasticsearch.yml
[root@elk-1 ~]# cat /etc/elasticsearch/elasticsearch.yml
```handlebars
# ======= Elasticsearch Configuration ===========
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ------------------Cluster --------------------
# Use a descriptive name for your cluster:
cluster.name: ELK //在這里插入代碼片配置es的集群名稱，默認是elasticsearch，es會自動發現在同一網段下的es，如果在同一網段下有多個集群，就可以用這個屬性來區分不同的集群，
# ------------------------Node -----------------
# Use a descriptive name for the node:
node.name: elk-1 //節點名，默認隨機指定一個name串列中名字，該串列在es的jar包中config檔案夾里name.txt檔案中，其中有很多作者添加的有趣名字，
node.master: true //指定該節點是否有資格被選舉成為node，默認是true，es是默認集群中的第一臺機器為master，如果這臺機掛了就會重新選舉master， 其他兩節點為false
node.data: false //指定該節點是否存盤索引資料，默認為true，其他兩節點為true
# ----------------- Paths ----------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /var/lib/elasticsearch //索引資料存盤位置(保持默認,不要開啟注釋)
# Path to log files:
path.logs: /var/log/elasticsearch //設定日志檔案的存盤路徑，默認是es根目錄下的logs檔案夾
# --------------- Network ------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
network.host: 192.168.200.10 //設定系結的ip地址，可以是ipv4或ipv6的，默認為0.0.0.0，
# Set a custom port for HTTP:
http.port: 9200 //啟動的es對外訪問的http埠，默認9200
# For more information, consult the network module documentation.
# --------------------Discovery ----------------
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["elk-1","elk-2","elk-3"] //設定集群中master節點的初始串列，可以通過這些節點來自動發現新加入集群的節點，
Elk-2節點：
[root@elk-2 ~]# vi /etc/elasticsearch/elasticsearch.yml
[root@elk-2 ~]# cat /etc/elasticsearch/elasticsearch.yml |grep -v ^# |grep -v ^$
cluster.name: ELK
node.name: elk-2
node.master: false
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.200.20
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elk-1","elk-2","elk-3"]
Elk-3節點：
[root@elk-2 ~]# vi /etc/elasticsearch/elasticsearch.yml
[root@elk-2 ~]# cat /etc/elasticsearch/elasticsearch.yml |grep -v ^# |grep -v ^$
cluster.name: ELK
node.name: elk-3
node.master: false
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.200.30
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elk-1","elk-2","elk-3"]
（3）啟動服務
通過命令啟動es服務，啟動后使用netstat命令查看是否埠啟動
[root@elk-1 ~]# systemctl start elasticsearch
[root@elk-2 ~]# systemctl start elasticsearch
[root@elk-3 ~]# systemctl start elasticsearch
[root@elk-1 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1446/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1994/master
tcp6 0 0 192.168.200.10:9200 :::* LISTEN 19280/java
tcp6 0 0 192.168.200.10:9300 :::* LISTEN 19280/java
（4）檢測集群狀態
通過curl ‘IP:9200/_cluster/health?pretty’ 來檢查集群狀態，命令如下
Elk-1節點：
[root@elk-1 ~]# curl '192.168.200.10:9200/_cluster/health?pretty'
{
"cluster_name" : "ELK",
"status" : "green", //為green則代表健康沒問題，yellow或者red 則是集群有問題
"timed_out" : false, //是否有超時
"number_of_nodes" : 3, //集群中的節點數量
"number_of_data_nodes" : 2, //集群中data節點的數量
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
三.部署kibana
（1）安裝kibana
[root@elk-1 ~]# rpm -ivh kibana-6.0.0-x86_64.rpm
（2）配置kibana
[root@elk-1 ~]# vi /etc/kibana/kibana.yml
[root@elk-1 ~]# cat /etc/kibana/kibana.yml |grep -v ^#
server.port: 5601
server.host: 192.168.200.10
elasticsearch.url: "http://192.168.200.10:9200"
（3）啟動kibana
[root@elk-1 ~]# systemctl start kibana
查看埠
[root@elk-1 ~]# netstat -lntp |grep node
tcp 0 0 192.168.200.10:5601 0.0.0.0:* LISTEN 19958/node
啟動成功后網頁訪問（192.168.200.10:5601），可以訪問到如下界面
四.Logstash部署：
（1）安裝logstash
[root@elk-2 ~]# rpm -ivh logstash-6.0.0.0.rpm
（2）配置logstash
配置/etc/logstash/logstash.yml，修改增加如下：
[root@elk-2 ~]# vi /etc/logstash/logstash.yml
http.host: "192.168.200.20"
配置logstash收集syslog日志
[root@elk-2 ~]# vi /etc/logstash/conf.d/syslog.conf
[root@elk-2 ~]# cat /etc/logstash/conf.d/syslog.conf
input { #定義日志源
file {
path => "/var/log/messages" #定義日志來源路徑 目錄要給644權限，不然無法讀取日志
type => "systemlog" #定義型別
start_position => "beginning"
stat_interval => "3"
}
}
output { #定義日志輸出
if [type] == "systemlog" {
elasticsearch {
hosts => ["192.168.200.10:9200"]
index => "system-log-%{+YYYY.MM.dd}"
}
}
}
賦權 /var/log/messages
[root@elk-2 ~]# chomd 644 /var/log/messages
#創建軟連接，方便使用logstash命令
[root@elk-2 ~]# ln -s /usr/share/logstash/bin/logstash /usr/bin`
#檢測組態檔是否錯誤：
[root@elk-2 ~]# logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK #為ok則代表沒問題
//--path.settings 用于指定logstash的組態檔所在的目錄
//-f 指定需要被檢測的組態檔的路徑
//--config.test_and_exit 指定檢測完之后就退出，不然就會直接啟動了
（3）啟動logstash
檢查組態檔沒有問題后啟動logstash服務，
[root@elk-2 ~]# vi /etc/rsyslog.conf
在#### RULES ####增加一行
*.* @@192.168.200.20:10514
[root@elk-2 ~]# systemctl start logstash
查看埠netstat -lntp
[root@elk-2 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1443/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2009/master
tcp6 0 0 192.168.200.20:9200 :::* LISTEN 19365/java
tcp6 0 0 :::10514 :::* LISTEN 21835/java
tcp6 0 0 192.168.200.20:9300 :::* LISTEN 19365/java
tcp6 0 0 :::22 :::* LISTEN 1443/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2009/master
tcp6 0 0 192.168.200.20:9600 :::* LISTEN 21835/java
啟動服務后，有行程但是沒有埠的問題解決：
[root@elk-2 ~]# cat /var/log/logstash/logstash-plain.log
//權限問題，因為之前我們以root的身份在終端啟動過logstash，所以產生的相關檔案的屬組屬主都是root
[root@elk-2 ~]# ll /var/lib/logstash/
total 4
drwxr-xr-x. 2 root root 6 Dec 6 15:45 dead_letter_queue
drwxr-xr-x. 2 root root 6 Dec 6 15:45 queue
-rw-r--r--. 1 root root 36 Dec 6 15:45 uuid
[root@elk-2 ~]# chown -R logstash /var/lib/logstash/
[root@elk-2 ~]# systemctl restart logstash #重啟服務后即可
啟動完畢后，讓syslog產生日志，用第三臺主機登錄elk-2機器，登錄后退出，
6.完善
（1）kibana上查看日志
之前部署kibana完成后，還沒有檢索日志，現在logstash部署完成，我們回到kibana服務器上查看日志索引，執行命令如下：
root@elk-1 ~]# curl '192.168.200.10:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open system-log-2019.12.06 UeKk3IY6TiebNu_OD04YZA 5 1 938 0 816kb 412.2kb
green open .kibana KL7WlNw_T7K36_HSbchBcw 1 1 1 0 7.3kb 3.6kb
[root@elk-1 ~]# curl -XGET/DELETE '192.168.200.10:9200/system-log-2021.10.31?pretty'
{
"system-log-20" : {
"aliases" : { },
"mappings" : {
"systemlog" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
（2）web界面配置
瀏覽器訪問192.168.200.10:5601,到kibana上配置 索引：
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/344160.html
標籤：其他