作者:申紅磊,QingCloud 容器解決方案架構師,開源專案愛好者,KubeSphere Member
在正式閱讀本文之前,先友情提醒一下:不建議您在生產環境中使用 NFS 存盤(特別是 Kubernetes 1.20 或以上版本),原因如下:
- selfLink was empty 在 K8s 集群 v1.20 之前都存在,在 v1.20 之后被洗掉問題,
- 還有可能引起 failed to obtain lock 和 input/output error 等問題,從而導致 Pod CrashLoopBackOff,此外,部分應用不兼容 NFS,例如 Prometheus 等,
安裝 NFS Server
#安裝 NFS 服務器端
$ sudo apt-get update #執行以下命令確保使用最新軟體包
$ sudo apt-get install nfs-kernel-server
#安裝 NFS 客戶端
$ sudo apt-get install nfs-common
# yum
$ yum install -y nfs-utils
創建共享目錄
先查看組態檔 /etc/exports:
$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
創建共享目標并賦權:
#目錄ksha3.2
$ mkdir /ksha3.2
$ chmod 777 /ksha3.2
#目錄demo
$ mkdir /demo
$ chmod 777 /demo
#目錄/home/ubuntu/nfs/ks3.1
$ mkdir /home/ubuntu/nfs/ks3.1
$ chmod 777 /home/ubuntu/nfs/ks3.1
添加到組態檔中 /etc/exports:
$ vi /etc/exports
....
/home/ubuntu/nfs/ks3.1 *(rw,sync,no_subtree_check)
/mount/ksha3.2 *(rw,sync,no_subtree_check)
/mount/demo *(rw,sync,no_subtree_check)
#/mnt/ks3.2 139.198.186.39(insecure,rw,sync,anonuid=500,anongid=500)
#/mnt/demo 139.198.167.103(rw,sync,no_subtree_check)
#此檔案的配置格式為:<輸出目錄> [客戶端1 選項(訪問權限,用戶映射,其他)] [客戶端2 選項(訪問權限,用戶映射,其他)]
注意:如果共享目錄創建無效或者遺漏創建,使用時會報例外如下:
$ mount.nfs: access denied by server while mounting 139.198.168.114:/mnt/demo1
#客戶端掛載 nfs 共享目錄的時候提示 mount.nfs: access denied by server while mounting,
#問題原因:服務器端的共享目錄沒有設定允許其他人訪問的權限,或者客戶端掛載的目錄沒有權限,
#解決辦法:在服務器端修改共享目錄的權限,成功連接,
驗證
#更新組態檔,重新加載 /etc/exports 的配置:
$ exportfs -rv
#在 nfs server 上測驗 查看本機共享的目錄:
$ showmount -e 127.0.0.1
$ showmount -e localhost
$ showmount -e 127.0.0.1
Export list for 127.0.0.1:
/mount/demo *
/mount/ksha3.2 *
/home/ubuntu/nfs/ks3.1 *
使用
在其它網路通的機器上,使用 NFS 共享目錄,
安裝客戶端
如果不安裝客戶端會報:bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program.
#apt-get
$ apt-get install nfs-common
#yum
$ yum install nfs-utils
在用戶機器上進行測驗:
#本機創建掛載目錄
#目錄ksha3.2
$ mkdir /root/aa
$ chmod 777 /root/aa
#使用 nfs client 上測驗,將 /root/aa 目錄掛載到遠端 139.198.168.114 的共享存盤目錄/mount/ksha3.2上
$ mount -t nfs 139.198.168.114:/mount/ksha3.2 /root/aa
查看:
#使用 df -h 查看
$ df -h | grep aa
139.198.168.114:/mount/ksha3.2 146G 5.3G 140G 4% /root/aa
解綁:
$ umount -t nfs 139.198.168.114:/mnt/ksha3.2 /root/aa
#如果卸載的時候提示:umount:/mnt:device is busy;解決方法:需要退出掛載目錄再進行卸載,或者是否NFS server宕機了
#需要強制卸載:mount –lf /mnt
#此命令也可以:fuser –km /mnt 不建議用
KubeSphere 對接 NFS 動態分配器
可安裝 NFS 客戶端程式,為了方便用戶對接 NFS 服務端,KubeSphere 可安裝 NFS 動態分配器,支持動態分配存盤卷,分配和回收存盤卷程序簡便,可對接一個或者多個 NFS 服務端,
當然也可以使用 Kubernetes 官方方法對接 NFS 服務端,這是一種靜態分配存盤卷方法,分配和回收存盤卷程序復雜,可對接多個 NFS 服務端,
前提條件
用戶對接 NFS 服務端時應確保 KubeSphere 各節點有權限掛載 NFS 服務端檔案夾,
操作步驟
以下步驟示例中,NFS 服務端 IP 為 139.198.168.114,NFS 共享檔案夾為 /mnt/ksha3.2,
安裝 NFS 動態分配器
首先請在這里下載 rbac.yaml,或者直接執行命令:
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/rbac.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
然后下載官方的 nfs provisoner 用途的 deployment,deployment 檔案中有幾處,請根據自己的情況做修改:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs/provisioner-229
- name: NFS_SERVER
value: 139.198.168.114
- name: NFS_PATH
value: /mount/ksha3.2
volumes:
- name: nfs-client-root
nfs:
server: 139.198.168.114
path: /mount/ksha3.2
執行創建:
$ kubectl apply -f deployment.yaml
deployment.apps/nfs-client-provisioner created
接下來下載官方的 class.yaml,然后創建 StorageClass,需要根據實際情況修改引數:
#請根據上方deployment部署時候的provisioner_name做對應的修改,或者沒有修改,就不用動
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
annotations:
"storageclass.kubernetes.io/is-default-class": "false"
provisioner: nfs/provisioner-229 # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
執行創建:kubectl apply -f storageclass.yaml,
如果想讓這個 NFS 作為默認的 Provisioner, 那么就添加如下的 annotation:
annotations:
"storageclass.kubernetes.io/is-default-class": "true"
或者標記一個 StorageClass 為默認的 StorageClass, 你需要添加 / 設定注解 storageclass.kubernetes.io/is-default-class=true,
$ kubectl patch storageclass <your-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
# 檢查集群中是否已存在 default Storage Class
$ kubectl get sc
NAME PROVISIONER AGE
glusterfs (default) kubernetes.io/glusterfs 3d4h
驗證安裝結果
執行以下命令,查看 NFS 動態分配器容器組是否正常運行,
$ kubectl get po -A | grep nfs-client
default nfs-client-provisioner-7d69b9f45f-ks94m 1/1 Running 0 9m3s
查看 NFS 存盤型別
$ kubectl get sc managed-nfs-storage
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage nfs/provisioner-229 Delete Immediate false 6m28s
創建和掛載 NFS 存盤卷
現在可以通過動態創建 NFS 存盤卷和作業負載掛載 NFS 存盤卷了,
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: demo4nfs
namespace: ddddd
annotations:
kubesphere.io/creator: admin
volume.beta.kubernetes.io/storage-provisioner: nfs/provisioner-229
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: managed-nfs-storage
volumeMode: Filesystem
unexpected error getting claim reference: selfLink was empty, can’t make reference
問題現象
使用 NFS 創建 PV 時,PVC 一直是處于 Pending 狀態,
查看 PVC:
$ kubectl get pvc -n ddddd
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
demo4nfs Bound pvc-a561ce85-fc0d-42af-948e-6894ac000264 10Gi RWO managed-nfs-storage 32m
查看詳細資訊:
#查看當前pvc的狀態資訊,發現是在等待volume的創建
$ kubectl get pvc -n ddddd
查看 nfs-client-provisioner 的日志,是 seltlink was empty 的問題,selfLink was empty 在 K8s 集群 v1.20 之前都存在,在 v1.20 之后被洗掉,需要在 /etc/kubernetes/manifests/kube-apiserver.yaml 中添加引數,
$ kubectl get pod -n default
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-7d69b9f45f-ks94m 1/1 Running 0 27m
$ kubectl logs -f nfs-client-provisioner-7d69b9f45f-ks94m
I0622 09:41:33.606000 1 leaderelection.go:185] attempting to acquire leader lease default/nfs-provisioner-229...
E0622 09:41:33.612745 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nfs-provisioner-229", GenerateName:"", Namespace:"default", SelfLink:"", UID:"e8f19e28-f17f-4b22-9bb8-d4cbe20c796b", ResourceVersion:"23803580", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63791487693, loc:(*time.Location)(0x1956800)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"nfs-client-provisioner-7d69b9f45f-ks94m_8081b417-f20f-11ec-bff3-f64a9f402fda\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2022-06-22T09:41:33Z\",\"renewTime\":\"2022-06-22T09:41:33Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'nfs-client-provisioner-7d69b9f45f-ks94m_8081b417-f20f-11ec-bff3-f64a9f402fda became leader'
I0622 09:41:33.612829 1 leaderelection.go:194] successfully acquired lease default/nfs-provisioner-229
I0622 09:41:33.612973 1 controller.go:631] Starting provisioner controller nfs/provisioner-229_nfs-client-provisioner-7d69b9f45f-ks94m_8081b417-f20f-11ec-bff3-f64a9f402fda!
I0622 09:41:33.713170 1 controller.go:680] Started provisioner controller nfs/provisioner-229_nfs-client-provisioner-7d69b9f45f-ks94m_8081b417-f20f-11ec-bff3-f64a9f402fda!
I0622 09:53:33.461902 1 controller.go:987] provision "ddddd/demo4nfs" class "managed-nfs-storage": started
E0622 09:53:33.464213 1 controller.go:1004] provision "ddddd/demo4nfs" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I0622 09:56:33.623717 1 controller.go:987] provision "ddddd/demo4nfs" class "managed-nfs-storage": started
E0622 09:56:33.625852 1 controller.go:1004] provision "ddddd/demo4nfs" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
解決辦法
在 kube-apiserver.yaml 檔案中添加引數 - --feature-gates=RemoveSelfLink=false,

#使用命令查找一下 kube-apiserver.yaml的位置
$ find / -name kube-apiserver.yaml
/data/kubernetes/manifests/kube-apiserver.yaml
#在檔案中添加 - --feature-gates=RemoveSelfLink=false ,如下圖
$ cat /data/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.100.25:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --feature-gates=RemoveSelfLink=false
- --advertise-address=0.0.0.0
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
#添加之后使用kubeadm部署的集群會自動加載部署pod
#kubeadm安裝的apiserver是Static Pod,它的組態檔被修改后,立即生效,
#Kubelet 會監聽該檔案的變化,當您修改了 /etc/kubenetes/manifest/kube-apiserver.yaml 檔案之后,kubelet 將自動終止原有的 #kube-apiserver-{nodename} 的 Pod,并自動創建一個使用了新配置引數的 Pod 作為替代,
#如果您有多個 Kubernetes Master 節點,您需要在每一個 Master 節點上都修改該檔案,并使各節點上的引數保持一致,
#這里需注意如果api-server啟動失敗 需重新在執行一遍
$ kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
GitHub 官方 ISSUES
- unexpected error getting claim reference: selfLink was empty, can’t make reference
本文由博客一文多發平臺 OpenWrite 發布!
轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/518878.html
標籤:其他
