ZooKeeper監控資料采集方案——Telegraf Plugin

首發CSDN：徐同學呀，原創不易，轉載請注明源鏈接，我是徐同學，用心輸出高質量文章，希望對你有所幫助，

一、前言

1、五種官方監控方案

Zookeeper官方提供了五種監控資料采集方案：

1. Prometheus，運行Prometheus監控服務是獲取和記錄ZooKeeper指標的最簡單方法，需要在zoo.cfg中設定引數metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider、 metricsProvider.httpPort=7000（默認埠7000），Prometheus啟動后每10秒記錄一次ZooKeeper資料，
2. Grafana，Grafana有內置的Prometheus支持，只需添加一個Prometheus資料源，如果Grafana作為圖表展示終端，安裝Prometheus收集Zookeeper監控資料比較合適，
3. Apache Zookeeper Telegraf Plugin + InfluxDB，安裝啟動Telegraf Plugin插件，定時收集ZooKeeper監控資料，并匯入InfluxDB中，
4. JMX，使用JMX方式監控Zookeeper，需要對JMX有一定了解，
5. Four letter words，運行四字命令獲取Zookeeper運行資料，可以自己寫收集腳本，定時對某些Zookeeper集群執行四字命令（如mntr），將資料寫入MySql或者InfluxDB，

2、選擇Telegraf Plugin采集zk運行時資料

經過對比，最終選擇Telegraf Plugin作為zk運行時資料采集方案，選擇理由：

• 生產環境用的influxdb存盤資料，Telegraf Plugin完美契合，
• Telegraf是Go寫的腳本，這意味著它是一個編譯的獨立二進制檔案，可以在任何系統上運行，無需其他依賴，占用記憶體小，性能高，
• Telegraf是插件驅動的資料收集和輸出，易擴展，
• Telegraf 采集zk指標的原理是定時執行四字命令mntr，默認頻率是10s一次，
• Telegraf上手簡單，只需要配置輸入源zk，輸出源influxdb即可，

Telegraf 的缺點：

• 上手容易，過濾、聚合配置較復雜，有一定學習成本，
• 默認收集mntr，暫未找到可以配置四字命令的地方，
• 入到influxdb的表名默認是zookeeper，暫未找到可以自定義表名的地方，
• 暫未找到配置過濾指標的地方，比如某些指標不想入庫，

如果不想用Telegraf，也可以自行根據mntr等四字命令開發定時腳本，技術掌握和靈活性大，

二、四字命令mntr

Zookeeper的四字命令，可參考https://www.runoob.com/w3cnote/zookeeper-sc-4lw.html，

mntr可以輸出zk集群健康狀態資料，如果想獲取zk節點的配置資訊，用conf，

[xxx@stefan~]$ echo mntr | nc 127.0.0.1 2181
zk_version      3.4.6-1569965, built on 02/20/2014 09:09 GMT
zk_avg_latency  0
zk_max_latency  1078
zk_min_latency  0
zk_packets_received     3956911
zk_packets_sent 3957032
zk_num_alive_connections        20
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  4220
zk_watch_count  32
zk_ephemerals_count     42
zk_approximate_data_size        16452676
zk_open_file_descriptor_count   47
zk_max_file_descriptor_count    409600

[xxx@stefan ~]$ echo mntr | nc 127.0.0.1 2181
zk_version      3.4.6-1569965, built on 02/20/2014 09:09 GMT
# 平均延遲
zk_avg_latency  0
# 最大延遲
zk_max_latency  549
# 最小延遲
zk_min_latency  0
# 收包
zk_packets_received     3683733
# 發包
zk_packets_sent 3684169
# 連接數
zk_num_alive_connections        24
# 堆積請求數
zk_outstanding_requests 0
# 狀態（角色）
zk_server_state leader
# znode數量
zk_znode_count  4220
# watch數量
zk_watch_count  2790
# 臨時節點數量
zk_ephemerals_count     42
# 資料大小
zk_approximate_data_size        16452676
# 打開的檔案描述符數量
zk_open_file_descriptor_count   52
# 最大檔案描述符數量
zk_max_file_descriptor_count    409600
# leader 會多出如下3個資訊
# follower數量
zk_followers    2
# 已經同步的follower數量
zk_synced_followers     2
# 正在同步的數量
zk_pending_syncs        0

三、Telegraf Plugin安裝（CentOS）

參考官方鏈接：https://www.influxdata.com/get-influxdb/

cd /usr/local
mkdir telegraf
cd telegraf
wget https://dl.influxdata.com/telegraf/releases/telegraf-1.19.2-1.x86_64.rpm
sudo yum localinstall telegraf-1.19.2-1.x86_64.rpm

執行如上命令即可安裝成功，運行telegraf --version查看安裝版本：

[root@Thu Sep 09 /usr/local]# telegraf --version
Telegraf 1.19.2 (git: HEAD 3cb135b6)

四、Telegraf Plugin啟動與除錯

1、除錯

Telegraf一旦啟動并運行，它將開始收集指標，并每10秒將資料寫入本地的inflxdb，在啟動之前可以用如下命令測驗腳本是否正常：

# 運行下腳本，將收集的資訊列印在控制臺
telegraf --test
# 運行下腳本，將收集的cpu、mem資訊列印在控制臺（:分割）
telegraf --input-filter cpu:mem --test
# 只列印zookeeper收集資訊
telegraf --input-filter zookeeper --test

2、啟動與停止

sudo service telegraf start
sudo service telegraf stop
sudo service telegraf restart

如果直接輸入telegraf，也是運行腳本，但不是后臺運行，日志會同步列印在控制臺，前期測驗可以使用，

3、telegraf --help

Telegraf, The plugin-driven server agent for collecting and reporting metrics.

Usage:

  telegraf [commands|flags]

The commands & flags are:

  config              print out full sample configuration to stdout
  version             print the version to stdout
# 篩選要啟用的聚合器，分隔符為:
  --aggregator-filter <filter>   filter the aggregators to enable, separator is :
# 要加載的組態檔  
  --config <file>                configuration file to load
  --config-directory <directory> directory containing additional *.conf files
  --watch-config                 Telegraf will restart on local config changes. Monitor changes 
                                 using either fs notifications or polling.  Valid values: 'inotify' or 'poll'. 
                                 Monitoring is off by default.
  --plugin-directory             directory containing *.so files, this directory will be
                                 searched recursively. Any Plugin found will be loaded
                                 and namespaced.
# 打開除錯日志                                 
  --debug                        turn on debug logging
  --input-filter <filter>        filter the inputs to enable, separator is :
  --input-list                   print available input plugins.
  --output-filter <filter>       filter the outputs to enable, separator is :
  --output-list                  print available output plugins.
  --pidfile <file>               file to write our pid to
  --pprof-addr <address>         pprof address to listen on, don't activate pprof if empty
  --processor-filter <filter>    filter the processors to enable, separator is :
  --quiet                        run in quiet mode
  --section-filter               filter config sections to output, separator is :
                                 Valid values are 'agent', 'global_tags', 'outputs',
                                 'processors', 'aggregators' and 'inputs'
  --sample-config                print out full sample configuration
  --once                         enable once mode: gather metrics once, write them, and exit
  --test                         enable test mode: gather metrics once and print them
  --test-wait                    wait up to this many seconds for service
                                 inputs to complete in test or once mode
  --usage <plugin>               print usage for a plugin, ie, 'telegraf --usage mysql'
  --version                      display the version and exit

五、telegraf.conf

Telegraf Plugin安裝之后啟動配置默認為/etc/telegraf/telegraf.conf，其中可以配置腳本的基本屬性，如資料收集的時間間隔，輸出源influxdb配置，輸入源zookeeper配置等，如下僅截取關心的部分配置：

1、agent

[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## Maximum number of unwritten metrics per output.  Increasing this value
  ## allows for longer periods of output downtime without dropping metrics at the
  ## cost of higher maximum memory usage.
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "μs"), "ms", "s".
  precision = ""

  ## Log at debug level.
  # debug = false
  ## Log only error level messages.
  # quiet = false

  ## Log target controls the destination for logs and can be one of "file",
  ## "stderr" or, on Windows, "eventlog".  When set to "file", the output file
  ## is determined by the "logfile" setting.
  # logtarget = "file"

  ## Name of the file to be logged to when using the "file" logtarget.  If set to
  ## the empty string then logs are written to stderr.
  # logfile = ""

  ## The logfile will be rotated after the time interval specified.  When set
  ## to 0 no time based rotation is performed.  Logs are rotated only when
  ## written to, if there is no log activity rotation may be delayed.
  # logfile_rotation_interval = "0d"

  ## The logfile will be rotated when it becomes larger than the specified
  ## size.  When set to 0 no size based rotation is performed.
  # logfile_rotation_max_size = "0MB"

  ## Maximum number of rotated archives to keep, any older logs are deleted.
  ## If set to -1, no archives are removed.
  # logfile_rotation_max_archives = 5

  ## Pick a timezone to use when logging or type 'local' for local time.
  ## Example: America/Chicago
  # log_with_timezone = ""

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false

2、outputs.influxdb

# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]
  ## The full HTTP or UDP URL for your InfluxDB instance.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  # urls = ["unix:///var/run/influxdb.sock"]
  # urls = ["udp://127.0.0.1:8089"]
  # urls = ["http://127.0.0.1:8086"]

  ## The target database for metrics; will be created as needed.
  ## For UDP url endpoint database needs to be configured on server side.
  # database = "telegraf"

  ## The value of this tag will be used to determine the database.  If this
  ## tag is not set the 'database' option is used as the default.
  # database_tag = ""

  ## If true, the 'database_tag' will not be included in the written metric.
  # exclude_database_tag = false

  ## If true, no CREATE DATABASE queries will be sent.  Set to true when using
  ## Telegraf with a user without permissions to create databases or when the
  ## database already exists.
  # skip_database_creation = false

  ## Name of existing retention policy to write to.  Empty string writes to
  ## the default retention policy.  Only takes effect when using HTTP.
  # retention_policy = ""

  ## The value of this tag will be used to determine the retention policy.  If this
  ## tag is not set the 'retention_policy' option is used as the default.
  # retention_policy_tag = ""

  ## If true, the 'retention_policy_tag' will not be included in the written metric.
  # exclude_retention_policy_tag = false

  ## Write consistency (clusters only), can be: "any", "one", "quorum", "all".
  ## Only takes effect when using HTTP.
  # write_consistency = "any"

  ## Timeout for HTTP messages.
  # timeout = "5s"

  ## HTTP Basic Auth
  # username = "telegraf"
  # password = "metricsmetricsmetricsmetrics"

  ## HTTP User-Agent
  # user_agent = "telegraf"

  ## UDP payload size is the maximum packet size to send.
  # udp_payload = "512B"

  ## Optional TLS Config for use on HTTP connections.
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## HTTP Proxy override, if unset values the standard proxy environment
  ## variables are consulted to determine which proxy, if any, should be used.
  # http_proxy = "http://corporate.proxy:3128"

  ## Additional HTTP headers
  # http_headers = {"X-Special-Header" = "Special-Value"}

  ## HTTP Content-Encoding for write request body, can be set to "gzip" to
  ## compress body or "identity" to apply no encoding.
  # content_encoding = "gzip"

  ## When true, Telegraf will output unsigned integers as unsigned values,
  ## i.e.: "42u".  You will need a version of InfluxDB supporting unsigned
  ## integer values.  Enabling this option will result in field type errors if
  ## existing data has been written.
  # influx_uint_support = false

3、inputs.zookeeper

# # Reads 'mntr' stats from one or many zookeeper servers
# [[inputs.zookeeper]]
#   ## An array of address to gather stats about. Specify an ip or hostname
#   ## with port. ie localhost:2181, 10.0.0.1:2181, etc.
#
#   ## If no servers are specified, then localhost is used as the host.
#   ## If no port is specified, 2181 is used
#   servers = [":2181"]
#
#   ## Timeout for metric collections from all servers.  Minimum timeout is "1s".
#   # timeout = "5s"
#
#   ## Optional TLS Config
#   # enable_tls = true
#   # tls_ca = "/etc/telegraf/ca.pem"
#   # tls_cert = "/etc/telegraf/cert.pem"
#   # tls_key = "/etc/telegraf/key.pem"
#   ## If false, skip chain & host verification
#   # insecure_skip_verify = true

更多配置可參考：https://github.com/influxdata/telegraf

六、注意事項

1、四字命令不在白名單

[root@Thu Sep 09 ~]# echo mntr | nc 127.0.0.1 2192
mntr is not executed because it is not in the whitelist.

執行某些四字命令報了not in the whitelist.，需要在zk配置zoo.conf中加一行4lw.commands.whitelist=*，重啟zk即可，

2、zk指標會因為zk版本而不同

mntr在zk3.7.0指標會很多，而且未在Telegraf的配置中看到可以過濾指標的方式（Tag Limit Processor Plugin是過濾tag key的），默認measurement名稱為zookeeper，未看到重命名的地方，

3、不要安裝influxDB 2.0

influxDB 2.0太新了，和influxDB1.8有比較大的差別，influx命令變了，啟動配置路徑也變了，翻了半天檔案也沒找到默認配置路徑在哪里，，，，所以還是暫時老老實實安裝influxDB1.8，可參考influxdb基礎（一）——influxdb安裝與基本配置（centos），

4、Telegraf好像只能收集mntr

未在Telegraf配置中找到配置zk四字命令的地方，默認是mntr，

5、Telegraf學習成本

Telegraf上手成本低，除了輸入插件和輸出插件外，Telegraf還包括聚合器和處理器插件，它們用于聚合和處理通過Telegraf收集的指標，想對資料做復雜操作，有一定學習成本，

七、參考文獻

1. https://zookeeper.apache.org/doc/current/zookeeperMonitor.html
2. https://github.com/influxdata/telegraf
3. https://rootnroll.com/d/telegraf/
4. https://www.influxdata.com/integration/apache-zookeeper/
5. https://docs.influxdata.com/telegraf/v1.19/

如若文章有錯誤理解，歡迎批評指正，同時非常期待你的留言和點贊，如果覺得有用，不妨來個一鍵三連，讓更多人受益，