文章目錄
- 一、部署 K8S
- 1. 環境準備
- 2. 安裝 docker
- 3. 配置阿里云 K8S repo 源（三個節點）
- 4. 安裝 kubeadm，kubelet，kubectl（三個節點）
- 5. 部署 kubernetes Master 節點（master 節點上執行）
- 6. k8s-node 節點加入 master 節點（兩個 node 執行）
- 7. 安裝 Pod 網路插件（CNI 插件，master 節點）
- 二、部署 KubeSphere
- 1. kubesphere 簡介
- 2. 環境需求
- 3. 安裝 helm 和 tiller
- 3.1 安裝helm
- 3.2 創建 rbac 權限檔案
- 3.3 安裝 tiller
- 3.4 檢查
- 4. 安裝 OpenEBS
- 4.1 去除 master上污點
- 4.2 安裝 OpenEBS
- 5. 部署 KubeSphere
- 5.1 執行以下命令以開始安裝
- 5.2 檢查安裝日志
- 5.3 查看運行狀態
- 5.4 瀏覽器訪問
- 5.5 查看集群管理界面
- 5.6 重新給 master 打上污點
- 6. 可插拔安裝插件
一、部署 K8S
1. 環境準備
三臺服務器
192.168.8.12 master
192.168.8.13 node01
192.168.8.14 node02
#關閉防火墻，selinux，swap
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
swapoff -a
#修改主機名，并寫入三臺服務器的 host 中
hostnamectl set-hostname master && su
hostnamectl set-hostname node01 && su
hostnamectl set-hostname node02 && su
cat >> /etc/hosts << EOF
192.168.8.12 master
192.168.8.13 node01
192.168.8.14 node02
EOF
#將橋接的 IPV4 流量傳遞到 iptables 鏈
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
#時間同步
yum -y install ntpdate
ntpdate time.windows.com
2. 安裝 docker
三臺服務器上操作
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum install -y docker-ce docker-ce-cli containerd.io
yum -y install epel-release && yum clean all && yum makecache #如果無法安裝docker再執行
systemctl start docker && systemctl enable docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
systemctl restart docker
3. 配置阿里云 K8S repo 源（三個節點）
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum clean all && yum makecache
4. 安裝 kubeadm，kubelet，kubectl（三個節點）
yum install -y kubelet-1.17.3 kubeadm-1.17.3 kubectl-1.17.3 --nogpgcheck
systemctl enable kubelet && systemctl start kubelet
5. 部署 kubernetes Master 節點（master 節點上執行）
初始化 kubeadm
kubeadm init \
--apiserver-advertise-address=192.168.8.12 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.17.3 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16
引數說明
kubeadm init \
--apiserver-advertise-address=10.0.0.116 \ #指定master監聽的地址，修改為自己的master地址
--image-repository registry.aliyuncs.com/google_containers \ #指定為aliyun的下載源，最好用國內的
--kubernetes-version v1.17.3 \
--service-cidr=10.96.0.0/12 \ #設定集群內部的網路
--pod-network-cidr=10.244.0.0/16 #設定pod的網路
# service-cidr 和 pod-network-cidr 最好就用這個，不然需要修改后面的 kube-flannel.yaml 檔案
出現 Your Kubernetes control-plane has initialized successfully! 為初始化成功，初始化失敗先排查原因
#如果忘了token，可以執行下面的命令進行查看
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
node 節點加入集群需要生成的 token，token 有效期為 24 小時，過期需要重新創建，創建命令為 kubeadm token create --print-join-command
執行以下命令可使用 kubectl 管理工具
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@master ~]#kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 4m18s v1.17.3
[root@master ~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.17.3 ae853e93800d 22 months ago 116MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.17.3 b0f1517c1f4b 22 months ago 161MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.17.3 90d27391b780 22 months ago 171MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.17.3 d109c0821a2b 22 months ago 94.4MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 2 years ago 41.6MB
registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 2 years ago 288MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 4 years ago 742kB
6. k8s-node 節點加入 master 節點（兩個 node 執行）
node01
[root@node01 ~]# kubeadm join 192.168.8.12:6443 --token l4kasn.nsm4gmn7yyvfzj1s \
--discovery-token-ca-cert-hash sha256:59e3b2efafd295b638844633e8e18ab294bb853fca0712e6795cd8f6a5f3f635
node02
[root@node02 ~]# kubeadm join 192.168.8.12:6443 --token l4kasn.nsm4gmn7yyvfzj1s \
--discovery-token-ca-cert-hash sha256:59e3b2efafd295b638844633e8e18ab294bb853fca0712e6795cd8f6a5f3f635
master 查看
[root@master ~]#kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 10m v1.17.3
node01 NotReady <none> 48s v1.17.3
node02 NotReady <none> 27s v1.17.3
可以看到節點顯示 NotReady 狀態，需要安裝網路插件
7. 安裝 Pod 網路插件（CNI 插件，master 節點）
下載插件 yaml 檔案
#國外網站
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
#國內網站
wget http://120.78.77.38/file/kube-flannel.yaml
執行安裝
#需要時間安裝
kubectl apply -f kube-flannel.yaml
執行 kubectl get pods -n kube-system 查看出錯是因為 quay.io 網站目前國內無法訪問，資源下載不成功，解決方案參考：https://blog.csdn.net/K_520_W/article/details/116566733
這里提供一個方便的解決方法
#修改 flannel 插件檔案
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yaml
kubectl apply -f kube-flannel.yaml
kubectl get pods -n kube-system
kubectl get node #部署好網路插件，node 準備就緒
查看
[root@master ~]#kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-j52kj 0/1 Pending 0 6m40s
coredns-7ff77c879f-sm7t4 0/1 Pending 0 6m40s
etcd-master 1/1 Running 0 6m50s
kube-apiserver-master 1/1 Running 0 6m50s
kube-controller-manager-master 1/1 Running 0 6m50s
kube-flannel-ds-amd64-2z7sl 0/1 Init:0/1 0 16s
kube-flannel-ds-amd64-65n5x 0/1 Init:0/1 0 16s
kube-flannel-ds-amd64-s5kf5 0/1 Init:0/1 0 16s
kube-proxy-dvkff 1/1 Running 0 3m20s
kube-proxy-jl84d 1/1 Running 0 4m22s
kube-proxy-kvktr 1/1 Running 0 6m40s
kube-scheduler-master 1/1 Running 0 6m50s
[root@master ~]#kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 8m21s v1.18.0
node01 Ready <none> 5m44s v1.18.0
node02 Ready <none> 4m42s v1.18.0
測驗 k8s 集群，在集群中創建一個 pod，驗證是否能正常運行
[root@master ~]# kubectl get pod #默認命名空間現在是沒有 pod 的
No resources found in default namespace.
[root@master ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master ~]# kubectl expose deployment nginx --port=80 --type=NodePort #暴露埠供外網訪問
service/nginx exposed
[root@master ~]#kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-f89759699-g76rt 1/1 Running 0 6m8s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 15m
service/nginx NodePort 10.104.42.45 <none> 80:30547/TCP 6m
# nginx 暴露了 32141 埠，http://nodeIP:port 訪問，任意節點 IP 都可以
[root@master ~]# curl 192.168.8.12:30547
二、部署 KubeSphere
1. kubesphere 簡介
KubeSphere 是一款面向云原生設計的開源專案，在目前主流容器調度平臺 Kubernetes 之上構建的分布式多租戶容器管理平臺，提供簡單易用的操作界面以及向導式操作方式，在降低用戶使用容器調度平臺學習成本的同時，極大降低開發、測驗、運維的日常作業的復雜度，
官網地址
3.0 版本安裝檔案 和 2.1 版本安裝檔案
2. 環境需求
3. 安裝 helm 和 tiller
Helm 是 Kubernetes 的包管理器，包管理器類似于我們在 Ubuntu 中使用的 apt、Centos中使用的 yum 或者 Python 中的 pip 一樣，能快速查找、下載和安裝軟體包，
Helm 由客 戶端組件 helm 和服務端組件 Tiller 組成，能夠將一組 K8S 資源打包統一管理，是查找、共享和使用為 Kubernetes 構建的軟體的最佳方式
3.1 安裝helm
#上傳安裝包到目錄下
[root@master ~]#ll helm-v2.17.0-linux-amd64.tar.gz
-rw-r--r-- 1 root root 25097357 12月 31 10:39 helm-v2.17.0-linux-amd64.tar.gz
[root@master ~]#tar zxvf helm-v2.17.0-linux-amd64.tar.gz
[root@master ~]#ls
helm-v2.17.0-linux-amd64.tar.gz kube-flannel.yaml linux-amd64
[root@master ~]#cp linux-amd64/helm /usr/local/bin/
[root@master ~]#cp linux-amd64/tiller /usr/local/bin/
[root@master ~]#helm version
Client: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
Error: could not find tiller
3.2 創建 rbac 權限檔案
cat > helm-rbac.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
EOF
[root@master ~]#kubectl apply -f helm-rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
3.3 安裝 tiller
helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.17.0 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
[root@master ~]#kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-86c57db685-gxnmf 1/1 Running 0 87m
kube-system coredns-9d85f5447-s8f77 1/1 Running 0 109m
kube-system coredns-9d85f5447-t6fd4 1/1 Running 0 109m
kube-system etcd-master 1/1 Running 0 109m
kube-system kube-apiserver-master 1/1 Running 0 109m
kube-system kube-controller-manager-master 1/1 Running 0 109m
kube-system kube-flannel-ds-amd64-c9mst 1/1 Running 0 89m
kube-system kube-flannel-ds-amd64-hfmgw 1/1 Running 0 97m
kube-system kube-flannel-ds-amd64-nsnc7 1/1 Running 0 97m
kube-system kube-proxy-kcpcj 1/1 Running 0 99m
kube-system kube-proxy-kjsg8 1/1 Running 0 99m
kube-system kube-proxy-ms6rg 1/1 Running 0 109m
kube-system kube-scheduler-master 1/1 Running 0 109m
kube-system tiller-deploy-59665c97b6-h9bd8 1/1 Running 0 2m53s
3.4 檢查
[root@master ~]#tiller
[main] 2021/12/31 21:57:16 Starting Tiller v2.17.0 (tls=false)
[main] 2021/12/31 21:57:16 GRPC listening on :44134
[main] 2021/12/31 21:57:16 Probes listening on :44135
[main] 2021/12/31 21:57:16 Storage driver is ConfigMap
[main] 2021/12/31 21:57:16 Max history per release is 0
[root@master ~]#helm version
Client: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
4. 安裝 OpenEBS
官方檔案供參考
4.1 去除 master上污點
[root@master ~]#kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready master 114m v1.17.3 192.168.8.12 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.12
node01 Ready <none> 103m v1.17.3 192.168.8.13 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.12
node02 Ready <none> 103m v1.17.3 192.168.8.14 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.12
[root@master ~]#kubectl describe node master | grep Taint
Taints: node-role.kubernetes.io/master:NoSchedule
[root@master ~]#kubectl taint nodes master node-role.kubernetes.io/master:NoSchedule-
node/master untainted
[root@master ~]#kubectl describe node master | grep Taint
Taints: <none>
4.2 安裝 OpenEBS
使用阿里的源會導致在后續安裝openebs時,資源庫沒有1.5的鏡像資源,錯誤如下
[root@k8s-node1 ~]# helm install --namespace openebs --name openebs stable/openebs --version 1.5.0
Error: failed to download "stable/openebs" (hint: running `helm repo update` may help)
解決方法如下：
- 先下載要用到的docker鏡像如下
docker pull openebs/m-apiserver:1.5.0
docker pull openebs/openebs-k8s-provisioner:1.5.0
docker pull openebs/snapshot-controller:1.5.0
docker pull openebs/snapshot-provisioner:1.5.0
docker pull openebs/node-disk-manager-amd64:v0.4.5
docker pull openebs/node-disk-operator-amd64:v0.4.5
docker pull openebs/admission-server:1.5.0
docker pull openebs/provisioner-localpv:1.5.0
- 下載修改后的 yaml 檔案如下
# This manifest deploys the OpenEBS control plane components, with associated CRs & RBAC rules
# NOTE: On GKE, deploy the openebs-operator.yaml in admin context
# Create the OpenEBS namespace
apiVersion: v1
kind: Namespace
metadata:
name: openebs
---
# Create Maya Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
name: openebs-maya-operator
namespace: openebs
---
# Define Role that allows operations on K8s pods/deployments
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: openebs-maya-operator
rules:
- apiGroups: ["*"]
resources: ["nodes", "nodes/proxy"]
verbs: ["*"]
- apiGroups: ["*"]
resources: ["namespaces", "services", "pods", "pods/exec", "deployments", "deployments/finalizers", "replicationcontrollers", "replicasets", "events", "endpoints", "configmaps", "secrets", "jobs", "cronjobs"]
verbs: ["*"]
- apiGroups: ["*"]
resources: ["statefulsets", "daemonsets"]
verbs: ["*"]
- apiGroups: ["*"]
resources: ["resourcequotas", "limitranges"]
verbs: ["list", "watch"]
- apiGroups: ["*"]
resources: ["ingresses", "horizontalpodautoscalers", "verticalpodautoscalers", "poddisruptionbudgets", "certificatesigningrequests"]
verbs: ["list", "watch"]
- apiGroups: ["*"]
resources: ["storageclasses", "persistentvolumeclaims", "persistentvolumes"]
verbs: ["*"]
- apiGroups: ["volumesnapshot.external-storage.k8s.io"]
resources: ["volumesnapshots", "volumesnapshotdatas"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: [ "get", "list", "create", "update", "delete", "patch"]
- apiGroups: ["*"]
resources: [ "disks", "blockdevices", "blockdeviceclaims"]
verbs: ["*" ]
- apiGroups: ["*"]
resources: [ "cstorpoolclusters", "storagepoolclaims", "storagepoolclaims/finalizers", "cstorpoolclusters/finalizers", "storagepools"]
verbs: ["*" ]
- apiGroups: ["*"]
resources: [ "castemplates", "runtasks"]
verbs: ["*" ]
- apiGroups: ["*"]
resources: [ "cstorpools", "cstorpools/finalizers", "cstorvolumereplicas", "cstorvolumes", "cstorvolumeclaims"]
verbs: ["*" ]
- apiGroups: ["*"]
resources: [ "cstorpoolinstances", "cstorpoolinstances/finalizers"]
verbs: ["*" ]
- apiGroups: ["*"]
resources: [ "cstorbackups", "cstorrestores", "cstorcompletedbackups"]
verbs: ["*" ]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
verbs: ["get", "create", "list", "delete", "update", "patch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
- apiGroups: ["*"]
resources: [ "upgradetasks"]
verbs: ["*" ]
---
# Bind the Service Account with the Role Privileges.
# TODO: Check if default account also needs to be there
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: openebs-maya-operator
subjects:
- kind: ServiceAccount
name: openebs-maya-operator
namespace: openebs
roleRef:
kind: ClusterRole
name: openebs-maya-operator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: maya-apiserver
namespace: openebs
labels:
name: maya-apiserver
openebs.io/component-name: maya-apiserver
openebs.io/version: 1.5.0
spec:
selector:
matchLabels:
name: maya-apiserver
openebs.io/component-name: maya-apiserver
replicas: 1
strategy:
type: Recreate
rollingUpdate: null
template:
metadata:
labels:
name: maya-apiserver
openebs.io/component-name: maya-apiserver
openebs.io/version: 1.5.0
spec:
serviceAccountName: openebs-maya-operator
containers:
- name: maya-apiserver
imagePullPolicy: IfNotPresent
image: openebs/m-apiserver:1.5.0
ports:
- containerPort: 5656
env:
# OPENEBS_IO_KUBE_CONFIG enables maya api service to connect to K8s
# based on this config. This is ignored if empty.
# This is supported for maya api server version 0.5.2 onwards
#- name: OPENEBS_IO_KUBE_CONFIG
# value: "/home/ubuntu/.kube/config"
# OPENEBS_IO_K8S_MASTER enables maya api service to connect to K8s
# based on this address. This is ignored if empty.
# This is supported for maya api server version 0.5.2 onwards
#- name: OPENEBS_IO_K8S_MASTER
# value: "http://172.28.128.3:8080"
# OPENEBS_NAMESPACE provides the namespace of this deployment as an
# environment variable
- name: OPENEBS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# OPENEBS_SERVICE_ACCOUNT provides the service account of this pod as
# environment variable
- name: OPENEBS_SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
# OPENEBS_MAYA_POD_NAME provides the name of this pod as
# environment variable
- name: OPENEBS_MAYA_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
# If OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG is false then OpenEBS default
# storageclass and storagepool will not be created.
- name: OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG
value: "true"
# OPENEBS_IO_INSTALL_DEFAULT_CSTOR_SPARSE_POOL decides whether default cstor sparse pool should be
# configured as a part of openebs installation.
# If "true" a default cstor sparse pool will be configured, if "false" it will not be configured.
# This value takes effect only if OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG
# is set to true
- name: OPENEBS_IO_INSTALL_DEFAULT_CSTOR_SPARSE_POOL
value: "false"
# OPENEBS_IO_CSTOR_TARGET_DIR can be used to specify the hostpath
# to be used for saving the shared content between the side cars
# of cstor volume pod.
# The default path used is /var/openebs/sparse
#- name: OPENEBS_IO_CSTOR_TARGET_DIR
# value: "/var/openebs/sparse"
# OPENEBS_IO_CSTOR_POOL_SPARSE_DIR can be used to specify the hostpath
# to be used for saving the shared content between the side cars
# of cstor pool pod. This ENV is also used to indicate the location
# of the sparse devices.
# The default path used is /var/openebs/sparse
#- name: OPENEBS_IO_CSTOR_POOL_SPARSE_DIR
# value: "/var/openebs/sparse"
# OPENEBS_IO_JIVA_POOL_DIR can be used to specify the hostpath
# to be used for default Jiva StoragePool loaded by OpenEBS
# The default path used is /var/openebs
# This value takes effect only if OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG
# is set to true
#- name: OPENEBS_IO_JIVA_POOL_DIR
# value: "/var/openebs"
# OPENEBS_IO_LOCALPV_HOSTPATH_DIR can be used to specify the hostpath
# to be used for default openebs-hostpath storageclass loaded by OpenEBS
# The default path used is /var/openebs/local
# This value takes effect only if OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG
# is set to true
#- name: OPENEBS_IO_LOCALPV_HOSTPATH_DIR
# value: "/var/openebs/local"
- name: OPENEBS_IO_JIVA_CONTROLLER_IMAGE
value: "openebs/jiva:1.5.0"
- name: OPENEBS_IO_JIVA_REPLICA_IMAGE
value: "openebs/jiva:1.5.0"
- name: OPENEBS_IO_JIVA_REPLICA_COUNT
value: "3"
- name: OPENEBS_IO_CSTOR_TARGET_IMAGE
value: "openebs/cstor-istgt:1.5.0"
- name: OPENEBS_IO_CSTOR_POOL_IMAGE
value: "openebs/cstor-pool:1.5.0"
- name: OPENEBS_IO_CSTOR_POOL_MGMT_IMAGE
value: "openebs/cstor-pool-mgmt:1.5.0"
- name: OPENEBS_IO_CSTOR_VOLUME_MGMT_IMAGE
value: "openebs/cstor-volume-mgmt:1.5.0"
- name: OPENEBS_IO_VOLUME_MONITOR_IMAGE
value: "openebs/m-exporter:1.5.0"
- name: OPENEBS_IO_CSTOR_POOL_EXPORTER_IMAGE
###################################################################################################################
value: "openebs/m-exporter:1.5.0"
- name: OPENEBS_IO_HELPER_IMAGE
value: "openebs/linux-utils:1.5.0"
# OPENEBS_IO_ENABLE_ANALYTICS if set to true sends anonymous usage
# events to Google Analytics
- name: OPENEBS_IO_ENABLE_ANALYTICS
value: "true"
- name: OPENEBS_IO_INSTALLER_TYPE
value: "openebs-operator"
# OPENEBS_IO_ANALYTICS_PING_INTERVAL can be used to specify the duration (in hours)
# for periodic ping events sent to Google Analytics.
# Default is 24h.
# Minimum is 1h. You can convert this to weekly by setting 168h
#- name: OPENEBS_IO_ANALYTICS_PING_INTERVAL
# value: "24h"
livenessProbe:
exec:
command:
- /usr/local/bin/mayactl
- version
initialDelaySeconds: 30
periodSeconds: 60
readinessProbe:
exec:
command:
- /usr/local/bin/mayactl
- version
initialDelaySeconds: 30
periodSeconds: 60
---
apiVersion: v1
kind: Service
metadata:
name: maya-apiserver-service
namespace: openebs
labels:
openebs.io/component-name: maya-apiserver-svc
spec:
ports:
- name: api
port: 5656
protocol: TCP
targetPort: 5656
selector:
name: maya-apiserver
sessionAffinity: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: openebs-provisioner
namespace: openebs
labels:
name: openebs-provisioner
openebs.io/component-name: openebs-provisioner
openebs.io/version: 1.5.0
spec:
selector:
matchLabels:
name: openebs-provisioner
openebs.io/component-name: openebs-provisioner
replicas: 1
strategy:
type: Recreate
rollingUpdate: null
template:
metadata:
labels:
name: openebs-provisioner
openebs.io/component-name: openebs-provisioner
openebs.io/version: 1.5.0
spec:
serviceAccountName: openebs-maya-operator
containers:
- name: openebs-provisioner
imagePullPolicy: IfNotPresent
image: openebs/openebs-k8s-provisioner:1.5.0
env:
# OPENEBS_IO_K8S_MASTER enables openebs provisioner to connect to K8s
# based on this address. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
#- name: OPENEBS_IO_K8S_MASTER
# value: "http://10.128.0.12:8080"
# OPENEBS_IO_KUBE_CONFIG enables openebs provisioner to connect to K8s
# based on this config. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
#- name: OPENEBS_IO_KUBE_CONFIG
# value: "/home/ubuntu/.kube/config"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: OPENEBS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# OPENEBS_MAYA_SERVICE_NAME provides the maya-apiserver K8s service name,
# that provisioner should forward the volume create/delete requests.
# If not present, "maya-apiserver-service" will be used for lookup.
# This is supported for openebs provisioner version 0.5.3-RC1 onwards
#- name: OPENEBS_MAYA_SERVICE_NAME
# value: "maya-apiserver-apiservice"
livenessProbe:
exec:
command:
- pgrep
- ".*openebs"
initialDelaySeconds: 30
periodSeconds: 60
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: openebs-snapshot-operator
namespace: openebs
labels:
name: openebs-snapshot-operator
openebs.io/component-name: openebs-snapshot-operator
openebs.io/version: 1.5.0
spec:
selector:
matchLabels:
name: openebs-snapshot-operator
openebs.io/component-name: openebs-snapshot-operator
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: openebs-snapshot-operator
openebs.io/component-name: openebs-snapshot-operator
openebs.io/version: 1.5.0
spec:
serviceAccountName: openebs-maya-operator
containers:
- name: snapshot-controller
image: openebs/snapshot-controller:1.5.0
imagePullPolicy: IfNotPresent
env:
- name: OPENEBS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
livenessProbe:
exec:
command:
- pgrep
- ".*controller"
initialDelaySeconds: 30
periodSeconds: 60
# OPENEBS_MAYA_SERVICE_NAME provides the maya-apiserver K8s service name,
# that snapshot controller should forward the snapshot create/delete requests.
# If not present, "maya-apiserver-service" will be used for lookup.
# This is supported for openebs provisioner version 0.5.3-RC1 onwards
#- name: OPENEBS_MAYA_SERVICE_NAME
# value: "maya-apiserver-apiservice"
- name: snapshot-provisioner
image: openebs/snapshot-provisioner:1.5.0
imagePullPolicy: IfNotPresent
env:
- name: OPENEBS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# OPENEBS_MAYA_SERVICE_NAME provides the maya-apiserver K8s service name,
# that snapshot provisioner should forward the clone create/delete requests.
# If not present, "maya-apiserver-service" will be used for lookup.
# This is supported for openebs provisioner version 0.5.3-RC1 onwards
#- name: OPENEBS_MAYA_SERVICE_NAME
# value: "maya-apiserver-apiservice"
livenessProbe:
exec:
command:
- pgrep
- ".*provisioner"
initialDelaySeconds: 30
periodSeconds: 60
---
# This is the node-disk-manager related config.
# It can be used to customize the disks probes and filters
apiVersion: v1
kind: ConfigMap
metadata:
name: openebs-ndm-config
namespace: openebs
labels:
openebs.io/component-name: ndm-config
data:
# udev-probe is default or primary probe which should be enabled to run ndm
# filterconfigs contails configs of filters - in their form fo include
# and exclude comma separated strings
node-disk-manager.config: |
probeconfigs:
- key: udev-probe
name: udev probe
state: true
- key: seachest-probe
name: seachest probe
state: false
- key: smart-probe
name: smart probe
state: true
filterconfigs:
- key: os-disk-exclude-filter
name: os disk exclude filter
state: true
exclude: "/,/etc/hosts,/boot"
- key: vendor-filter
name: vendor filter
state: true
include: ""
exclude: "CLOUDBYT,OpenEBS"
- key: path-filter
name: path filter
state: true
include: ""
exclude: "loop,/dev/fd0,/dev/sr0,/dev/ram,/dev/dm-,/dev/md"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: openebs-ndm
namespace: openebs
labels:
name: openebs-ndm
openebs.io/component-name: ndm
openebs.io/version: 1.5.0
spec:
selector:
matchLabels:
name: openebs-ndm
openebs.io/component-name: ndm
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
name: openebs-ndm
openebs.io/component-name: ndm
openebs.io/version: 1.5.0
spec:
# By default the node-disk-manager will be run on all kubernetes nodes
# If you would like to limit this to only some nodes, say the nodes
# that have storage attached, you could label those node and use
# nodeSelector.
#
# e.g. label the storage nodes with - "openebs.io/nodegroup"="storage-node"
# kubectl label node <node-name> "openebs.io/nodegroup"="storage-node"
#nodeSelector:
# "openebs.io/nodegroup": "storage-node"
serviceAccountName: openebs-maya-operator
hostNetwork: true
containers:
- name: node-disk-manager
image: openebs/node-disk-manager-amd64:v0.4.5
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /host/node-disk-manager.config
subPath: node-disk-manager.config
readOnly: true
- name: udev
mountPath: /run/udev
- name: procmount
mountPath: /host/proc
readOnly: true
- name: sparsepath
mountPath: /var/openebs/sparse
env:
# namespace in which NDM is installed will be passed to NDM Daemonset
# as environment variable
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# pass hostname as env variable using downward API to the NDM container
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# specify the directory where the sparse files need to be created.
# if not specified, then sparse files will not be created.
- name: SPARSE_FILE_DIR
value: "/var/openebs/sparse"
# Size(bytes) of the sparse file to be created.
- name: SPARSE_FILE_SIZE
value: "10737418240"
# Specify the number of sparse files to be created
- name: SPARSE_FILE_COUNT
value: "0"
livenessProbe:
exec:
command:
- pgrep
- ".*ndm"
initialDelaySeconds: 30
periodSeconds: 60
volumes:
- name: config
configMap:
name: openebs-ndm-config
- name: udev
hostPath:
path: /run/udev
type: Directory
# mount /proc (to access mount file of process 1 of host) inside container
# to read mount-point of disks and partitions
- name: procmount
hostPath:
path: /proc
type: Directory
- name: sparsepath
hostPath:
path: /var/openebs/sparse
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: openebs-ndm-operator
namespace: openebs
labels:
name: openebs-ndm-operator
openebs.io/component-name: ndm-operator
openebs.io/version: 1.5.0
spec:
selector:
matchLabels:
name: openebs-ndm-operator
openebs.io/component-name: ndm-operator
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: openebs-ndm-operator
openebs.io/component-name: ndm-operator
openebs.io/version: 1.5.0
spec:
serviceAccountName: openebs-maya-operator
containers:
- name: node-disk-operator
image: openebs/node-disk-operator-amd64:v0.4.5
imagePullPolicy: Always
readinessProbe:
exec:
command:
- stat
- /tmp/operator-sdk-ready
initialDelaySeconds: 4
periodSeconds: 10
failureThreshold: 1
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
# the service account of the ndm-operator pod
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: OPERATOR_NAME
value: "node-disk-operator"
- name: CLEANUP_JOB_IMAGE
value: "openebs/linux-utils:1.5.0"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: openebs-admission-server
namespace: openebs
labels:
app: admission-webhook
openebs.io/component-name: admission-webhook
openebs.io/version: 1.5.0
spec:
replicas: 1
strategy:
type: Recreate
rollingUpdate: null
selector:
matchLabels:
app: admission-webhook
template:
metadata:
labels:
app: admission-webhook
openebs.io/component-name: admission-webhook
openebs.io/version: 1.5.0
spec:
serviceAccountName: openebs-maya-operator
containers:
- name: admission-webhook
image: openebs/admission-server:1.5.0
imagePullPolicy: IfNotPresent
args:
- -alsologtostderr
- -v=2
- 2>&1
env:
- name: OPENEBS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADMISSION_WEBHOOK_NAME
value: "openebs-admission-server"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: openebs-localpv-provisioner
namespace: openebs
labels:
name: openebs-localpv-provisioner
openebs.io/component-name: openebs-localpv-provisioner
openebs.io/version: 1.5.0
spec:
selector:
matchLabels:
name: openebs-localpv-provisioner
openebs.io/component-name: openebs-localpv-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: openebs-localpv-provisioner
openebs.io/component-name: openebs-localpv-provisioner
openebs.io/version: 1.5.0
spec:
serviceAccountName: openebs-maya-operator
containers:
- name: openebs-provisioner-hostpath
imagePullPolicy: Always
image: openebs/provisioner-localpv:1.5.0
env:
# OPENEBS_IO_K8S_MASTER enables openebs provisioner to connect to K8s
# based on this address. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
#- name: OPENEBS_IO_K8S_MASTER
# value: "http://10.128.0.12:8080"
# OPENEBS_IO_KUBE_CONFIG enables openebs provisioner to connect to K8s
# based on this config. This is ignored if empty.
# This is supported for openebs provisioner version 0.5.2 onwards
#- name: OPENEBS_IO_KUBE_CONFIG
# value: "/home/ubuntu/.kube/config"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: OPENEBS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# OPENEBS_SERVICE_ACCOUNT provides the service account of this pod as
# environment variable
- name: OPENEBS_SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: OPENEBS_IO_ENABLE_ANALYTICS
value: "true"
- name: OPENEBS_IO_INSTALLER_TYPE
value: "openebs-operator"
- name: OPENEBS_IO_HELPER_IMAGE
value: "openebs/linux-utils:1.5.0"
livenessProbe:
exec:
command:
- pgrep
- ".*localpv"
initialDelaySeconds: 30
periodSeconds: 60
---
- 最后執行安裝這個yaml
kubectl apply -f openebs_demo.yaml
- 查看創建的 StorageClass
[root@master ~]#kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-device openebs.io/local Delete WaitForFirstConsumer false 16m
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 16m
openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 16m
openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 16m
- 設定默認storageclass
[root@master ~]#kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
[root@master ~]#kubectl get pod -n openebs
NAME READY STATUS RESTARTS AGE
maya-apiserver-85c5fdcd96-gqsg5 1/1 Running 0 15m
openebs-admission-server-84fbd958f7-b2frv 1/1 Running 0 15m
openebs-localpv-provisioner-bf57b7fb8-c9h2c 1/1 Running 0 15m
openebs-ndm-4xhkp 1/1 Running 0 15m
openebs-ndm-9g7cv 1/1 Running 0 15m
openebs-ndm-operator-6bc7d8f85b-dm7hg 1/1 Running 0 15m
openebs-ndm-rgbv5 1/1 Running 0 15m
openebs-provisioner-776bd55fdf-snmkf 1/1 Running 0 15m
openebs-snapshot-operator-6f57874cf7-ntg7b 2/2 Running 0 15m
[root@master ~]#kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-device openebs.io/local Delete WaitForFirstConsumer false 33m
openebs-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 33m
openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 33m
openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 33m
注意：此時不要給master加上污點，否者導致后面的pods安裝不上(openldap,redis),待kubesphere安裝完成后加上污點
5. 部署 KubeSphere
官方檔案
5.1 執行以下命令以開始安裝
kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/kubesphere-installer.yaml
kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/cluster-configuration.yaml
5.2 檢查安裝日志
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
5.3 查看運行狀態
使用 kubectl get pod --all-namespaces 查看所有 Pod 在 KubeSphere 相關的命名空間是否正常運行，如果是正常運行，請通過以下命令來檢查控制臺的埠（默認為 30880）
[root@master ~]#kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-86c57db685-gxnmf 1/1 Running 0 3h11m
kube-system coredns-9d85f5447-s8f77 1/1 Running 0 3h33m
kube-system coredns-9d85f5447-t6fd4 1/1 Running 0 3h33m
kube-system etcd-master 1/1 Running 0 3h33m
kube-system kube-apiserver-master 1/1 Running 0 3h33m
kube-system kube-controller-manager-master 1/1 Running 0 3h33m
kube-system kube-flannel-ds-amd64-c9mst 1/1 Running 0 3h13m
kube-system kube-flannel-ds-amd64-hfmgw 1/1 Running 0 3h21m
kube-system kube-flannel-ds-amd64-nsnc7 1/1 Running 0 3h21m
kube-system kube-proxy-kcpcj 1/1 Running 0 3h23m
kube-system kube-proxy-kjsg8 1/1 Running 0 3h23m
kube-system kube-proxy-ms6rg 1/1 Running 0 3h33m
kube-system kube-scheduler-master 1/1 Running 0 3h33m
kube-system snapshot-controller-0 1/1 Running 0 4m58s
kube-system tiller-deploy-59665c97b6-h9bd8 1/1 Running 0 106m
kubesphere-controls-system default-http-backend-5d464dd566-r495w 1/1 Running 0 4m43s
kubesphere-controls-system kubectl-admin-6c9bd5b454-zjns4 1/1 Running 0 87s
kubesphere-monitoring-system alertmanager-main-0 2/2 Running 0 3m47s
kubesphere-monitoring-system alertmanager-main-1 2/2 Running 0 3m47s
kubesphere-monitoring-system alertmanager-main-2 2/2 Running 0 3m47s
kubesphere-monitoring-system kube-state-metrics-5c466fc7b6-jw9dn 3/3 Running 0 4m8s
kubesphere-monitoring-system node-exporter-7tgmg 2/2 Running 0 4m9s
kubesphere-monitoring-system node-exporter-9vfgz 2/2 Running 0 4m9s
kubesphere-monitoring-system node-exporter-nblhm 2/2 Running 0 4m9s
kubesphere-monitoring-system notification-manager-deployment-7ff95b7544-nggdq 1/1 Running 0 2m4s
kubesphere-monitoring-system notification-manager-deployment-7ff95b7544-x4n9h 1/1 Running 0 2m4s
kubesphere-monitoring-system notification-manager-operator-5cbb58b756-hfhpc 2/2 Running 0 4m
kubesphere-monitoring-system prometheus-k8s-0 3/3 Running 1 3m37s
kubesphere-monitoring-system prometheus-k8s-1 3/3 Running 1 3m37s
kubesphere-monitoring-system prometheus-operator-78c5cdbc8f-jmv6p 2/2 Running 0 4m10s
kubesphere-system ks-apiserver-56d87c5f65-4h694 1/1 Running 0 3m45s
kubesphere-system ks-console-fb4c655cf-7f6r5 1/1 Running 0 4m36s
kubesphere-system ks-controller-manager-7547c5df5d-dm4dz 1/1 Running 0 3m44s
kubesphere-system ks-installer-85854b8c8-jp5xg 1/1 Running 0 7m9s
kubesphere-system openldap-0 1/1 Running 0 4m51s
kubesphere-system redis-6fd6c6d6f9-mjc7b 1/1 Running 0 4m54s
openebs maya-apiserver-85c5fdcd96-gqsg5 1/1 Running 0 28m
openebs openebs-admission-server-84fbd958f7-b2frv 1/1 Running 0 28m
openebs openebs-localpv-provisioner-bf57b7fb8-c9h2c 1/1 Running 0 28m
openebs openebs-ndm-4xhkp 1/1 Running 0 28m
openebs openebs-ndm-9g7cv 1/1 Running 0 28m
openebs openebs-ndm-operator-6bc7d8f85b-dm7hg 1/1 Running 0 28m
openebs openebs-ndm-rgbv5 1/1 Running 0 28m
openebs openebs-provisioner-776bd55fdf-snmkf 1/1 Running 0 28m
openebs openebs-snapshot-operator-6f57874cf7-ntg7b 2/2 Running 0 28m
[root@master ~]#kubectl get svc/ks-console -n kubesphere-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ks-console NodePort 10.101.95.222 <none> 80:30880/TCP 5m29s
5.4 瀏覽器訪問
訪問 http://192.168.8.12:30880
- 修改密碼：
5.5 查看集群管理界面
5.6 重新給 master 打上污點
[root@master ~]#kubectl taint nodes master node-role.kubernetes.io/master=:NoSchedule
node/master tainted
[root@master ~]#kubectl describe node master | grep Taint
Taints: node-role.kubernetes.io/master:NoSchedule
6. 可插拔安裝插件
官方檔案
可以在 KubeSphere 安裝之前或之后啟用可插拔組件，
請參考示例檔案 cluster-configuration.yaml 獲取更多詳細資訊，
請確保集群中有足夠的 CPU 和記憶體，
強烈建議安裝這些可插拔組件，以體驗 KubeSphere 提供的全堆疊功能，
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/401584.html
標籤：其他