一 CSI存盤機制
1.1 CSI簡介
Kubernetes從1.9版本開始引入容器存盤介面Container Storage Interface（CSI）機制，用于在Kubernetes和外部存盤系統之間建立一套標準的存盤管理介面，通過該介面為容器提供存盤服務，1.2 CSI的設計背景
Kubernetes通過PV、PVC、Storageclass已經提供了一種強大的基于插件的存盤管理機制，但是各種存盤插件提供的存盤服務都是基于一種被稱為“in-true”（樹內）的方式提供的，這要求存盤插件的代碼必須被放進Kubernetes的主干代碼庫中才能被Kubernetes呼叫，屬于緊耦合的開發模式，這種“in-tree”方式會帶來一些問題：- 存盤插件的代碼需要與Kubernetes的代碼放在同一代碼庫中，并與Kubernetes的二進制檔案共同發布；
- 存盤插件代碼的開發者必須遵循Kubernetes的代碼開發規范；
- 存盤插件代碼的開發者必須遵循Kubernetes的發布流程，包括添加對Kubernetes存盤系統的支持和錯誤修復；
- Kubernetes社區需要對存盤插件的代碼進行維護，包括審核、測驗等作業；
- 存盤插件代碼中的問題可能會影響Kubernetes組件的運行，并且很難排查問題；
- 存盤插件代碼與Kubernetes的核心組件（kubelet和kubecontroller-manager）享有相同的系統特權權限，可能存在可靠性和安全性問題，
Kubernetes已有的FlexVolume插件機制試圖通過為外部存盤暴露一個基于可執行程式（exec）的API來解決這些問題，盡管它允許第三方存盤提供商在Kubernetes核心代碼之外開發存盤驅動，但仍然有兩個問題沒有得到很好的解決：
- 部署第三方驅動的可執行檔案仍然需要宿主機的root權限，存在安全隱患；
- 存盤插件在執行mount、attach這些操作時，通常需要在宿主機上安裝一些第三方工具包和依賴庫，使得部署程序更加復雜，例如部署Ceph時需要安裝rbd庫，部署GlusterFS時需要安裝mount.glusterfs庫，等等，
基于以上這些問題和考慮，Kubernetes逐步推出與容器對接的存盤介面標準，存盤提供方只需要基于標準介面進行存盤插件的實作，就能使用Kubernetes的原生存盤機制為容器提供存盤服務，這套標準被稱為CSI（容器存盤介面）， 在CSI成為Kubernetes的存盤供應標準之后，存盤提供方的代碼就能和Kubernetes代碼徹底解耦，部署也與Kubernetes核心組件分離，顯然，存盤插件的開發由提供方自行維護，就能為Kubernetes用戶提供更多的存盤功能，也更加安全可靠， 基于CSI的存盤插件機制也被稱為“out-of-tree”（樹外）的服務提供方式，是未來Kubernetes第三方存盤插件的標準方案，
二 CSI架構
2.1 CSI存盤組件/部署架構
KubernetesCSI存盤插件的關鍵組件和推薦的容器化部署架構：
其中主要包括兩種組件：CSI Controller和CSI Node，
2.2 CSI Controller
CSI Controller的主要功能是提供存盤服務視角對存盤資源和存盤卷進行管理和操作，在Kubernetes中建議將其部署為單實體Pod，可以使用StatefulSet或Deployment控制器進行部署，設定副本數量為1，保證為一種存盤插件只運行一個控制器實體， 在這個Pod內部署兩個容器：- 與Master（kube-controller-manager）通信的輔助sidecar容器，在sidecar容器內又可以包含external-attacher和external-provisioner兩個容器，它們的功能分別如下，
- external-attacher：監控VolumeAttachment資源物件的變更，觸發針對CSI端點的ControllerPublish和ControllerUnpublish操作，
- external-provisioner：監控PersistentVolumeClaim資源物件的變更，觸發針對CSI端點的CreateVolume和DeleteVolume操作，
- CSI Driver存盤驅動容器，由第三方存盤提供商提供，需要實作上述介面，
2.3 CSI Node
CSI Node的主要功能是對主機（Node）上的Volume進行管理和操作，在Kubernetes中建議將其部署為DaemonSet，在每個Node上都運行一個Pod， 在這個Pod中部署以下兩個容器：- 與kubelet通信的輔助sidecar容器node-driver-registrar，主要功能是將存盤驅動注冊到kubelet中；
- CSI Driver存盤驅動容器，由第三方存盤提供商提供，主要功能是接收kubelet的呼叫，需要實作一系列與Node相關的CSI介面，例如NodePublishVolume介面（用于將Volume掛載到容器內的目標路徑）、NodeUnpublishVolume介面（用于從容器中卸載Volume），等等，
三 CSI插件使用實踐
3.1 實驗說明
以csi-hostpath插件為例，演示部署CSI插件、用戶使用CSI插件提供的存盤資源，3.2 開啟特性
設定Kubernetes服務啟動引數，為kube-apiserver、kubecontroller-manager和kubelet服務的啟動引數添加， [root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-apiserver.yaml……
- --allow-privileged=true
- --feature-gates=CSIPersistentVolume=true
- --runtime-config=storage.k8s.io/v1alpha1=true
……
[root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-controller-manager.yaml
……
- --feature-gates=CSIPersistentVolume=true
……
[root@k8smaster01 ~]# vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --feature-gates=CSIPersistentVolume=true" ……[root@k8smaster01 ~]# systemctl daemon-reload [root@k8smaster01 ~]# systemctl restart kubelet.service
3.3 創建CRD資源物件
創建CSINodeInfo和CSIDriverRegistry CRD資源物件： [root@k8smaster01 ~]# vi csidriver.yamlapiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: csidrivers.csi.storage.k8s.io labels: addonmanager.kubernetes.io/mode: Reconcile spec: group: csi.storage.k8s.io names: kind: CSIDriver plural: csidrivers scope: Cluster validation: openAPIV3Schema: properties: spec: description: Specification of the CSI Driver. properties: attachRequired: description: Indicates this CSI volume driver requires an attach operation,and that Kubernetes should call attach and wait for any attach operationto complete before proceeding to mount. type: boolean podInfoOnMountVersion: description: Indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations. type: string version: v1alpha1[root@k8smaster01 ~]# vi csinodeinfo.yaml
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: csinodeinfos.csi.storage.k8s.io labels: addonmanager.kubernetes.io/mode: Reconcile spec: group: csi.storage.k8s.io names: kind: CSINodeInfo plural: csinodeinfos scope: Cluster validation: openAPIV3Schema: properties: spec: description: Specification of CSINodeInfo properties: drivers: description: List of CSI drivers running on the node and their specs. type: array items: properties: name: description: The CSI driver that this object refers to. type: string nodeID: description: The node from the driver point of view. type: string topologyKeys: description: List of keys supported by the driver. items: type: string type: array status: description: Status of CSINodeInfo properties: drivers: description: List of CSI drivers running on the node and their statuses. type: array items: properties: name: description: The CSI driver that this object refers to. type: string available: description: Whether the CSI driver is installed. type: boolean volumePluginMechanism: description: Indicates to external components the required mechanism to use for any in-tree plugins replaced by this driver. pattern: in-tree|csi type: string version: v1alpha1[root@k8smaster01 ~]# kubectl apply -f csidriver.yaml [root@k8smaster01 ~]# kubectl apply -f csinodeinfo.yaml
3.4 創建相應RBAC
[root@k8smaster01 ~]# git clone https://github.com/kubernetes-csi/drivers [root@k8smaster01 ~]# cd drivers/deploy/hostpath/ [root@k8smaster01 hostpath]# vi csi-hostpath-attacher-rbac.yaml--- apiVersion: v1 kind: ServiceAccount metadata: name: csi-attacher # replace with non-default namespace name namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: external-attacher-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "update"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] - apiGroups: ["csi.storage.k8s.io"] resources: ["csinodeinfos"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "update"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-attacher-role subjects: - kind: ServiceAccount name: csi-attacher # replace with non-default namespace name namespace: default roleRef: kind: ClusterRole name: external-attacher-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: # replace with non-default namespace name namespace: default name: external-attacher-cfg rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-attacher-role-cfg # replace with non-default namespace name namespace: default subjects: - kind: ServiceAccount name: csi-attacher # replace with non-default namespace name namespace: default roleRef: kind: Role name: external-attacher-cfg apiGroup: rbac.authorization.k8s.io[root@k8smaster01 hostpath]# vi csi-hostpath-provisioner-rbac.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: csi-provisioner # replace with non-default namespace name namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: external-provisioner-runner rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots"] verbs: ["get", "list"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["get", "list"] - apiGroups: ["csi.storage.k8s.io"] resources: ["csinodeinfos"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-provisioner-role subjects: - kind: ServiceAccount name: csi-provisioner # replace with non-default namespace name namespace: default roleRef: kind: ClusterRole name: external-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: # replace with non-default namespace name namespace: default name: external-provisioner-cfg rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-provisioner-role-cfg # replace with non-default namespace name namespace: default subjects: - kind: ServiceAccount name: csi-provisioner # replace with non-default namespace name namespace: default roleRef: kind: Role name: external-provisioner-cfg apiGroup: rbac.authorization.k8s.io[root@k8smaster01 hostpath]# vi csi-hostpathplugin-rbac.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: csi-driver-registrar # replace with non-default namespace name namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: driver-registrar-runner rules: - apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch", "create", "update", "patch"] # The following permissions are only needed when running # driver-registrar without the --kubelet-registration-path # parameter, i.e. when using driver-registrar instead of # kubelet to update the csi.volume.kubernetes.io/nodeid # annotation. That mode of operation is going to be deprecated # and should not be used anymore, but is needed on older # Kubernetes versions. # - apiGroups: [""] # resources: ["nodes"] # verbs: ["get", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-driver-registrar-role subjects: - kind: ServiceAccount name: csi-driver-registrar # replace with non-default namespace name namespace: default roleRef: kind: ClusterRole name: driver-registrar-runner apiGroup: rbac.authorization.k8s.io[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher-rbac.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner-rbac.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin-rbac.yaml
3.5 正式部署
[root@k8smaster01 ~]# cd drivers/deploy/hostpath/ [root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin.yaml 提示：如上相應yaml建議修改鏡像源為國內： gcr.io ----> gcr.azk8s.cn （國內） quay.io ----> quay.azk8s.cn （國內）四 測驗使用
4.1 確認驗證
[root@k8smaster01 ~]# kubectl get pods
4.2 創建StorageClass
[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-storageclass.yamlapiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-hostpath-sc
provisioner: csi-hostpath
reclaimPolicy: Delete
volumeBindingMode: Immediate
[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-storageclass.yaml
4.3 創建PVC
[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-pvc.yamlapiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: csi-hostpath-sc
[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-pvc.yaml
[root@k8smaster01 ~]# kubectl get pvc
[root@k8smaster01 ~]# kubectl get pv
4.4 創建應用
[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-app.yamlkind: Pod apiVersion: v1 metadata: name: my-csi-app spec: containers: - name: my-frontend image: busybox volumeMounts: - mountPath: "/data" name: my-csi-volume command: [ "sleep", "1000000" ] volumes: - name: my-csi-volume persistentVolumeClaim: claimName: csi-pvc[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-app.yaml [root@k8smaster01 ~]# kubectl get pods
提示：更多CSI插件示例參考：https://feisky.gitbooks.io/kubernetes/plugins/csi.html， CSI官方檔案：https://kubernetes-csi.github.io/docs/
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/115515.html
標籤：Linux
上一篇：shell腳本