prometheus相關的服務都部署在monitoring這個namespace下
部署prometheus服務
-
創建namespace
$ cd /opt/k8s/prometheus $ cat>1-namespace.yml<<EOF apiVersion: v1 kind: Namespace metadata: name: monitoring EOF -
創建prometheus對應的組態檔,利用kubernetes的ConfigMap
$ cd /opt/k8s/prometheus $ cat>2-prom-cnfig.yml<<EOF apiVersion: v1 kind: ConfigMap metadata: name: prom-config namespace: monitoring data: prometheus.yml: | global: scrape_interval: 15s scrape_timeout: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] EOF -
創建用來存盤prometheus資料的pv、pvc(采用本地存盤、可以通過搭建NFS、GlusterFs等分布式檔案系統代替)
$ cd /opt/k8s/prometheus $ cat>3-prom-pv.yml<<EOF kind: PersistentVolume apiVersion: v1 metadata: namespace: monitoring name: prometheus labels: type: local app: prometheus spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: /opt/k8s/prometheus/data --- kind: PersistentVolumeClaim apiVersion: v1 metadata: namespace: monitoring name: prometheus-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi EOF -
創建啟動prometheus的檔案,以deployment形式部署,外部訪問通過NodePort型別的service
$ cd /opt/k8s/prometheus $ cat>4-prometheus.yml<<EOF apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitoring labels: app: prometheus spec: selector: matchLabels: app: prometheus replicas: 1 template: metadata: labels: app: prometheus spec: containers: - name: prometheus image: prom/prometheus:v2.16.0 args: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - "--storage.tsdb.retention=7d" - "--web.enable-lifecycle" ports: - containerPort: 9090 volumeMounts: - mountPath: "/prometheus" subPath: prometheus name: data - mountPath: "/etc/prometheus" name: config resources: requests: cpu: 500m memory: 2Gi limits: cpu: 500m memory: 2Gi volumes: - name: config configMap: name: prom-config - name: data persistentVolumeClaim: claimName: prometheus-claim --- apiVersion: v1 kind: Service metadata: namespace: monitoring name: prometheus spec: type: NodePort ports: - port: 9090 targetPort: 9090 nodePort: 9090 selector: app: prometheus EOF- 在Prometheus的啟動命令中,傳入了引數storage.tsdb.path、storage.tsdb.retention分別指定了Prometheus資料存盤路徑、存盤時間
- web.enable-lifecycle 當配置資訊變更后通過/-/reload重新加載新的配置內容,不用重啟服務
-
啟動Prometheus服務
$ cd /opt/k8s/prometheus $ mkdir data & chmod -R 777 data $ kubectl create -f 1-namespace.yml -f 2-prom-cnfig.yml -f 3-prom-pv.yml -f 4-prometheus.yml -
查看組件狀態,確保所有服務正常啟動
$ kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/prometheus-57cf64764d-xqnvl 1/1 Running 0 51s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/prometheus NodePort 10.254.209.164 <none> 9090:9090/TCP 51s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/prometheus 1/1 1 1 51s NAME DESIRED CURRENT READY AGE replicaset.apps/prometheus-57cf64764d 1 1 1 51s -
界面訪問

和kubernetes監控相關的技術
- cAdvisor google 開源的一款容器監控方案,收集容器自身的各種資源使用、性能相關資訊,通過resutful API的形式提供給外部,kubernetes把cAdvisor對應的功能集成到了kubelet中,所以不需要再單獨部署,直接從kubelet獲取,并且kubelet還對容器資訊進行summary,以pod為單位供外部呼叫,
- metrics-server 是kubernetes核心監控流程中的一個組件,是Heapster的替代方案,從kubelet指標介面中獲取資訊,主要是CPU、Memory,再通過API server暴露出去,主要供kubectl top、HPA等kubernetes組件使用,只在記憶體中存盤最后一次獲取到的指標資訊,不負責資料存盤
- kube-state-metrics 通過監聽 API Server 生成有關資源物件的狀態指標,比如 Deployment、replica sets等,只在記憶體中存盤最后一次獲取到的指標資訊,不負責資料存盤
- node-exporter Prometheus官方提供的,專門用來收集*NIX系統自身、以及對應硬體的指標資訊
- kube-prometheus 一站式的kubernetes監控方案,將node-exporter、prometheus、kube-state-metrics、Grafana、metrics-server等組件收集起來,提供了更加便捷的腳本供使用者快速搭建一個完整的監控平臺,
kubernetes監控內容
- 對集群自身狀態的監控 ,如節點自身的CPU、Memory、IO、Network等資訊
- kubernetes系統自身組件的監控如 kube-schedule-manager、kube-proxy、kubelet等
- 集群中運行容器的監控,如容器、Pod等為單元的CPU、Memory資訊
- 集群中編排組件對應的指標監控,如Deployment、Daemonset等
本文采用自己部署組件的形式,看如何一步一步搭建一個監控平臺,對于有快速搭建需求的,可以參考 kube-prometheus
部署node-exporter
因為要監控每一個節點,所以采用Daemonset控制器來部署node-exporter,在每個節點上都運行一個Pod
-
啟動檔案
$ cd /opt/k8s/prometheus $ cat>5-node-exporter.yml<<EOF apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: node-exporter name: node-exporter namespace: monitoring spec: selector: matchLabels: app: node-exporter template: metadata: labels: app: node-exporter spec: containers: - name: node-exporter image: 192.168.0.107/prometheus/node-exporter:v0.18.1 args: - --web.listen-address=:9100 - --path.procfs=/host/proc - --path.sysfs=/host/sys - --path.sysfs=/host/sys - --path.rootfs=/host/root - --no-collector.hwmon - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$ resources: limits: cpu: 250m memory: 180Mi requests: cpu: 102m memory: 180Mi ports: - containerPort: 9100 volumeMounts: - mountPath: /host/proc name: proc readOnly: false - mountPath: /host/sys name: sys readOnly: false - mountPath: /host/root mountPropagation: HostToContainer name: root readOnly: true hostNetwork: true hostPID: true nodeSelector: kubernetes.io/os: linux securityContext: runAsNonRoot: true runAsUser: 65534 tolerations: - operator: Exists volumes: - hostPath: path: /proc name: proc - hostPath: path: /sys name: sys - hostPath: path: / name: root EOF -
啟動node-exporter
$ cd /opt/k8s/prometheus $ kubectl create -f 5-node-exporter.yml $ kubectl -n monitoring get pod | grep node node-exporter-854vr 1/1 Running 6 50m node-exporter-lv9pv 1/1 Running 0 50m -
通過prometheus收集node-exporter的指標資訊
因為集群的節點之后可能會動態擴容和縮減,所以不便采用靜態配置的形式,Prometheus給我們提供了Kubernetes對應的服務發現功能,可以實作對Kubernetes的動態監控,其中對節點的監控利用node的服務發現方式,在prometheus的組態檔中追加如下配置(對應的2-prom-cnfig.yml也需要追加,否則Configmap重建后這些資訊就丟掉了)
$ kubectl -n monitoring edit configmaps prom-config- job_name: "kubernetes-nodes" kubernetes_sd_configs: - role: node relabel_configs: - source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace - action: labelmap regex: __meta_kubernetes_node_label_(.+)追加完成后執行如下命令,重新加載配置項,至于為何這樣配置,后面章節配置原理具體講解
$ curl -XPOST http://192.168.0.107:9090/-/reload此時,prometheus就會嘗試獲取集群node資訊,查看promethes的日志資訊,會發現如下錯誤提示
level=error ts=2020-03-22T10:37:13.856Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:333: Failed to list *v1.Node: nodes is forbidden: User \"system:serviceaccount:monitoring:default\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"意思是用默認的serviceaccount不能 list *v1.Node,因此需要我們為Prometheus重新創建serviceaccount,并賦予相應的權限
-
創建prometheus對應的serviceaccount,并賦予相應的權限
$ cd /opt/k8s/prometheus $ cat>6-prometheus-serivceaccount-role.yaml<<EOF apiVersion: v1 kind: ServiceAccount metadata: name: prometheus-k8s namespace: monitoring --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus-k8s rules: - apiGroups: [""] resources: - nodes/proxy - nodes - namespaces - endpoints - pods - services verbs: ["get","list","watch"] - apiGroups: [""] resources: - nodes/metrics verbs: ["get"] - nonResourceURLs: - /metrics verbs: ["get"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus-k8s roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus-k8s subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring EOF$ cd /opt/k8s/prometheus $ kubectl create -f 6-prometheus-serivceaccount-role.yaml修改Prometheus的啟動yaml,追加serivceaccount配置,重啟Prometheus
... spec: serviceAccountName: prometheus-k8s containers: - name: prometheus ...查看prometheus監控物件串列

kubernetes_sd_config 配置原理詳解
-
目標地址查找
當kubernetes_sd_config配置的role是node時,prometheus啟動后會呼叫kubernetes的 LIST Node API獲取node相關資訊,并從node物件中獲取IP和PORT來構成監控地址,
其中IP獲取按照如下順序InternalIP, ExternalIP, LegacyHostIP, HostName查找
Port值默認采用Kubelet的HTTP port,
通過如下命令可查看 list Node介面回傳的IP 和 Port資訊
$ kubectl get node -o=jsonpath='{range .items[*]}{.status.addresses}{"\t"}{.status.daemonEndpoints}{"\n"}{end}' [map[address:192.168.0.107 type:InternalIP] map[address:master type:Hostname]] map[kubeletEndpoint:map[Port:10250]] [map[address:192.168.0.114 type:InternalIP] map[address:slave type:Hostname]] map[kubeletEndpoint:map[Port:10250]]上述回傳結果表示集群中有兩個節點,構成的taget地址分別是
192.168.0.107:10250 192.168.0.114:10250 -
relabe_configs
Relabeling可以讓Prometheus在抓取資料之前動態的修改標簽的值,Prometheus有許多默認標簽,其中下面幾個和我們處理有關
__address__:初始化時會設定成目標地址對應的<host>:<port>instance:__address__標簽的值經過Relabel階段后,會設定給標簽instance, 即instance是__address__標簽經過Relabel后的值__scheme__:默認值 http__metrics_path__默認值 /metrics
Prometheus拉取指標資訊的目的地址是把這幾個標簽連接起來
__scheme__://instance/__metrics_path__ -
我們啟動node-expeorter后,在各個節點的
:9100/metrics上暴露了node指標資訊,然后在prometheus中追加了如下配置段- job_name: "kubernetes-nodes" kubernetes_sd_configs: - role: node relabel_configs: - source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace - action: labelmap regex: __meta_kubernetes_node_label_(.+)其中relabel_configs的第一個配置片段
- source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace- 通過regex從
__address__中匹配出IP地址 - replacement:對應的值設定成
${IP}:9100 - target_label: 將
__address__的值替換成replacement,即${IP}:9100
經過這些步驟后,拼接成的獲取指標地址為
[http://192.168.0.107:9100/metrics, http://192.168.0.114:9100/metrics],和我們的node-exporter暴露的指標地址匹配,就可以拉取Node的指標資訊了另外因為prometheus會將node對應的標簽變成
__meta_kubernetes_node_label_<labelname>,所以追加了一個labelmap的動作,再把這些標簽名字還原出來 - 通過regex從
-
完整配置例子參考prometheus-kubernetes
追加收集kubelete提供的指標
kubelet會收集一些api server 、etcd等服務的指標資訊,可以通過如下命令查看
$ kubectl get --raw https://192.168.0.107:10250/metrics
- 其中10250是kubelet的默認監聽埠
接下來通過在promehteus中追加配置,讓promehteus拉取這些資訊
- job_name: "kubernetes-kubelet"
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
追加收集cAdvisor指標,實作對集群容器的監控
kubelet默認集成了cAdvisor,用來對集群中容器資訊進行收集,Kubernetes 1.7.3以及之后的版本中,把cAdvisor收集的指標資訊(以container_開頭)從Kubelet對應的/metrics中移除了,所以需要額外配置一個收集cAdvisor的job,cAdvisor指標呼叫命令
$ kubectl get --raw https://192.168.0.107:6443/api/v1/nodes/master/proxy/metrics/cadvisor
- 其中10250是kubelet的默認監聽埠
接下來通過在promehteus中追加配置,讓promehteus拉取這些資訊
- job_name: "kubernetes-cadvisor"
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
添加完成后重新load 組態檔,查看prometheus監控物件串列

完整的組態檔如下
$ cd /opt/k8s/prometheus
$ cat 2-prom-cnfig.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: prom-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s
scrape_timeout: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: "kubernetes-nodes"
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__address__]
regex: '(.*):10250'
replacement: '${1}:9100'
target_label: __address__
action: replace
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- job_name: "kubernetes-kubelet"
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- job_name: "kubernetes-cadvisor"
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
部署grafana
-
創建用來存盤grafana資料的pv、pvc(采用本地存盤、可以通過搭建NFS、GlusterFs等分布式檔案系統代替)
$ cd /opt/k8s/prometheus $ cat>7-grafana-pv.yml<<EOF kind: PersistentVolume apiVersion: v1 metadata: namespace: monitoring name: grafana labels: type: local app: grafana spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: /opt/k8s/prometheus/grafana-pvc --- kind: PersistentVolumeClaim apiVersion: v1 metadata: namespace: monitoring name: grafana-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi EOF -
grafana部署檔案
$ cd /opt/k8s/prometheus $ cat>8-grafana.yml<<EOF apiVersion: apps/v1 kind: Deployment metadata: labels: app: grafana name: grafana namespace: monitoring spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: containers: - image: grafana/grafana:6.6.2 name: grafana ports: - containerPort: 3000 name: http readinessProbe: httpGet: path: /api/health port: http resources: limits: cpu: 200m memory: 400Mi requests: cpu: 100m memory: 200Mi volumeMounts: - mountPath: /var/lib/grafana name: grafana-pvc readOnly: false subPath: data - mountPath: /etc/grafana/provisioning/datasources name: grafana-pvc readOnly: false subPath: datasources - mountPath: /etc/grafana/provisioning/dashboards name: grafana-pvc readOnly: false subPath: dashboards-pro - mountPath: /grafana-dashboard-definitions/0 name: grafana-pvc readOnly: false subPath: dashboards nodeSelector: beta.kubernetes.io/os: linux securityContext: runAsNonRoot: true runAsUser: 65534 volumes: - name: grafana-pvc persistentVolumeClaim: claimName: grafana-claim --- apiVersion: v1 kind: Service metadata: namespace: monitoring name: grafana spec: type: NodePort ports: - port: 3000 targetPort: 3000 nodePort: 3000 selector: app: grafana EOF -
啟動grafana
-
創建grafana掛載目錄
$ cd /opt/k8s/prometheus $ mkdir -p grafana-pvc/data $ mkdir -p grafana-pvc/datasources $ mkdir -p grafana-pvc/dashboards-pro $ mkdir -p grafana-pvc/dashboards $ chmod -R 777 grafana-pvc- data目錄存放grafana的資料
- datasources存放預定義的資料源
- dashboards-pro存放dashboards管理檔案,其中配置的dashboard的檔案地址指向grafana-pvc/dashboards掛載到容器中的地址/grafana-dashboard-definitions/0
- dashboards存放真正的dashboards定義檔案(json)
-
創建默認的資料源檔案
$ cd /opt/k8s/prometheus/grafana-pvc/datasources $ cat > datasource.yaml<<EOF apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus.monitoring.svc:9090 EOF -
創建默認的dashboards管理檔案
$ cd /opt/k8s/prometheus/grafana-pvc/dashboards-pro $ cat >dashboards.yaml<<EOF apiVersion: 1 providers: - name: '0' orgId: 1 folder: '' type: file editable: true updateIntervalSeconds: 10 allowUiUpdates: false options: path: /grafana-dashboard-definitions/0 EOF -
創建默認的dashboard定義檔案
可以到 a collection of shared dashboards,找到自己需要的dashboard模版,下載對應的json檔案將對應檔案存放到/opt/k8s/prometheus/grafana-pvc/dashboards),這里作為示例,下載1 Node Exporter for Prometheus Dashboard CN v20191102,對應的ID是8919,
$ cd /opt/k8s/prometheus/grafana-pvc/dashboards $ wget https://grafana.com/api/dashboards/8919/revisions/11/download -o node-exporter-k8s.json因為模版中的資料源默認用的是
${DS_PROMETHEUS_111},從界面匯入時有配置項供替換,我們直接下載json檔案,所以通過直接修改檔案,把資料源改成我們在/opt/k8s/prometheus/grafana-pvc/datasources下配置的資料源$ cd /opt/k8s/prometheus/grafana-pvc/dashboards $ sed -i "s/\${DS_PROMETHEUS_111}/Prometheus/g" node-exporter-k8s.json修改title
... "timezone": "browser", "title": "k8s-node-monitoring", ... -
啟動
$ cd /opt/k8s/prometheus/ $ kubectl create -f 7-grafana-pv.yml 8-grafana.yml
-
-
通過界面查看,因為我們已經默認設定過資料源、dashboard等資訊,可以直接查看對應的dashboard

在部署grafana時,我們配置了默認的資料源、dashboard等資訊,主要是為了實作,系統部署后這些默認監控指標可以直接觀察,不需要實施人員現場配置,
其他的監控例如使用 Kube-state-metrics以及cAdvisor metrics實作對集群中Deployment、StatefulSet、容器、pod的監控也可以采用這種形式來實作,如可以利用1. Kubernetes Deployment Statefulset Daemonset metrics作為模版,稍微修改滿足我們的監控需要,這里就不再展示具體步驟,讀者可以自行嘗試,
轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/33930.html
標籤:其他
上一篇:大贏家上下分充值的游戲
