一步步學習k8s(三)

新一代容器監控Prometheus

Prometheus(普羅米修斯)是一個最初在SoundCloud上構建的監控系統，
在2012年成為社區開源專案，偶那個有非常活躍的開發人員和用戶社區，為強調開源和獨立維護，Promethues于2016年加入CNCF，成為繼kubernetes之后的這第二個托管專案，
https://Prometheus.io
https://github.com/promethues

Prometheus的特點：

• 多維資料模型:由度量名稱和鍵值對標識的時間序列資料
• 內置時間序列資料庫：TSDB
• promQL:一種靈活的查詢語言，可以利用多維資料完成復雜查詢
• 基于HTTP的pull(拉取)方式采集時間序列資料(exporter)
• 同時支持PushGateway組件收集資料
• 通過服務發現或靜態配置發現目標
• 多種圖形模式及儀表盤支持
• 支持做為資料源接入Grafana

Prometheus的架構：

1、部署kube-state-metrics（收集k8s基本資訊的軟體）

在運維主機10.4.7.200上

1.1、準備kebe-state-metrics鏡像

官方地址：https://quay.io/repository/coreos/kube-state-metrics

[root@hdss7-200 ~]# docker pull quay.io/coreos/kube-state-metrics:v1.5.0
[root@hdss7-200 ~]# docker images |grep kube-state
quay.io/coreos/kube-state-metrics          v1.5.0          91599517197a   2 years ago     31.8MB
[root@hdss7-200 ~]# docker tag 91599517197a harbor.od.com/public/kube-state-metrics:v1.5.0
[root@hdss7-200 ~]# docker push harbor.od.com/public/kube-state-metrics:v1.5.0

1.2、準備資源配置清單

rbac.yaml

[root@hdss7-200 k8s-yaml]# mkdir kube-state-metrics
[root@hdss7-200 k8s-yaml]# cd kube-state-metrics/
[root@hdss7-200 kube-state-metrics]# vim rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: clusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
rules:
- apiGroups:
  _ ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - list
  - watch
  apiGroups:
  - batch
   resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: clusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

dp.yaml

[root@hdss7-200 kube-state-metrics]# vim dp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  labels:
    grafanak8sapp: "true"
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      grafanak8sapp: "true"
      app: kube-state-metrics
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        grafanak8sapp: "true"
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: harbor.od.com/public/kube-state-metrics:v1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: http-metrics
          protocol: TCP
        readinessProbe:
          fallureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
      serviceAccountName: kube-state-metrics

1.3、應用資源配置清單

在任意一臺運算節點上：

1.4、檢查啟動情況

2、部署node-exporter（收集k8s運算節點上基礎設施資訊的軟體）

3、部署cadvisor（監控容器內部使用資源的重要工具）

4、部署blackbox-exporter

4.1、準備blackbox-exporter鏡像

4.2、準備資源配置清單

4.3、應用資源配置清單

4.4、瀏覽器訪問

5、部署Promethues

5.1、準備prometheus鏡像

5.2、準備資源配置清單

5.3、準備prometheus的組態檔

5.4、應用資源配置清單

5.5、決議域名

5.6、瀏覽器訪問

5.7、Prometheus監控內容

5.7.1、etcd

5.7.2、kubernetes-apiserver

5.7.3、kubernetes-kubelet

5.7.4、kubernetes-kube-state

5.7.5、traefik

5.7.6、blackbox*

5.7.7、kubernetes-pods*

5.8、修改traefix服務接入prometheus監控

5.9、修改dubbo-service服務接入prometheus

5.10、修改dubbo-consumer服務接入prometheus

6、部署Grafana