主頁 >  其他 > Kubernetes — 在 OpenStack 上使用 kubeadm 部署高可用集群

Kubernetes — 在 OpenStack 上使用 kubeadm 部署高可用集群

2020-12-23 12:48:51 其他

目錄

文章目錄

  • 目錄
  • 高可用集群部署拓撲
  • 高可用集群網路拓撲
  • 網路代理配置
  • Load Balancer 環境準備
  • Kubernetes Cluster 環境準備
    • 安裝 Container Runtime
    • 安裝 kubeadm、kubelet 和 kubectl
  • 初始化 Master 主控制平面節點
    • kubeadm init 的作業流
    • 執行初始化
  • 清理或重新進行初始化
  • 添加 Master 冗余控制平面節點
  • 添加 Node 作業負載節點
  • 安裝 CNI 網路插件

高可用集群部署拓撲

官方檔案：https://kubernetes.io/zh/docs/setup/production-environment/

  • 基礎設施：OpenStack
  • 虛擬機集群：3 Master、2 Node、2 Load Balancer
  • 計算資源：x86-64 processor、2CPU、2GB RAM、20GB free disk space
  • 作業系統：CentOS 7.x+
  • 版本：Kubernetes 1.18.14
  • Container Runtime：Docker

在這里插入圖片描述

高可用集群網路拓撲

網路代理配置

因為要科學上網，所以需要對 HTTP/S Proxy 和 No Proxy 進行精心的配置，否則要么下不下來軟體，要么出現網路連通性的錯誤，

export https_proxy=http://{proxy_ip}:7890 http_proxy=http://{proxy_ip}:7890 all_proxy=socks5://{proxy_ip}:7890 no_proxy=localhost,127.0.0.1,{apiserver_endpoint_ip},{k8s_mgmt_network_ip_pool},{pod_network_ip_pool},{service_network_ip_pool}

Load Balancer 環境準備

基于 OpenStack Octavia LBaaS 來提供 HA Load Balancer，也可以手動的配置 keepalived and haproxy（https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing），

  • VIP 選擇 kube-mgmt-subnet
    在這里插入圖片描述

  • Listener 選擇 TCP :6443 Socket（kube-apiserver 的監聽埠）
    在這里插入圖片描述

  • Members 選擇 3 個 k8s-master
    在這里插入圖片描述

  • Monitor 同樣選擇 TCP :6443 Socket
    在這里插入圖片描述

注意：創建好 Load Balancer 之后，首先要測驗一下 TCP 反向代理運行正常，由于 apiserver 現在尚未運行，所以預期會出現一個連接拒絕錯誤，在我們初始化了第一個控制平面節點之后，要記得再次進行測驗，

# nc -v LOAD_BALANCER_IP PORT
nc -v 192.168.0.100 6443

Kubernetes Cluster 環境準備

注意：在所有節點上執行以下操作，

  • 科學上網，
  • 添加全節點的 Hostname 決議，
# vi /etc/hosts

192.168.0.100 kube-apiserver-endpoint
192.168.0.148 k8s-master-1
192.168.0.112 k8s-master-2
192.168.0.193 k8s-master-3
192.168.0.208 k8s-node-1
192.168.0.174 k8s-node-2
  • 開啟全節點之間的 SSH 免密登錄，
  • 禁用 Swap 交換磁區，為了保證 kubelet 正常作業，
  • 確保 iptables 工具不使用 nftables 后端，nftables 后端與當前的 kubeadm 軟體包不兼容，它會導致重復的防火墻規則并破壞 kube-proxy，
  • 確保節點之間的網路聯通性，
    在這里插入圖片描述
  • 關閉 SELinux，為了允許容器訪問主機的檔案系統，
# 將 SELinux 設定為 permissive 模式（相當于將其禁用）
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
  • 在 RHEL/CentOS 7 上為了保證 kube-proxy 控制的資料流量必須進過 iptables 的處理來進行本地路由，所以要確保 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 被設定為 1，
# 確保加載了 br_netfilter 模塊，
modprobe br_netfilter
lsmod | grep br_netfilter

# 確保 sysctl 配置，將 Bridge 的 IPv4 流量傳遞到 iptables 的 Chain（鏈）
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
  • 安裝基礎依賴軟體：
yum install ebtables ethtool ipvsadm -y

安裝 Container Runtime

注意：當 Linux 使用 systemd 時，會創建一個 cgroup，此時需要保證 Container Runtime、kubelet 和 systemd 使用的是同一個 cgroup，否則會出現不可預測的問題，為此，我們需要將 Container Runtime、kubelet 配置成使用 systemd 來作為 cgroup 驅動，以此使系統更為穩定，

對于 Docker 而言，設定 native.cgroupdriver=systemd 選項即可，

  • 安裝：
# 安裝依賴包
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

# 新增 Docker 倉庫
sudo yum-config-manager --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

# 安裝 Docker CE
sudo yum update -y && sudo yum install -y \
  containerd.io-1.2.13 \
  docker-ce-19.03.11 \
  docker-ce-cli-19.03.11
  • 配置：
# 創建 /etc/docker 目錄
sudo mkdir /etc/docker

# 設定 Docker daemon
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF
  • 重啟：
# Create /etc/systemd/system/docker.service.d
sudo mkdir -p /etc/systemd/system/docker.service.d

# 重啟 Docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker

sudo systemctl status docker

安裝 kubeadm、kubelet 和 kubectl

注意：kubeadm 是 Kubernetes Cluster 的部署工具，但 kubeadm 不能用于安裝、管理 kubelet 或 kubectl，所以我們需要收到安裝它們，并且確保三者的版本倉庫是一致的，

  • 更新 Kubernetes YUM 倉庫：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
  • 安裝：
# 查詢版本
$ yum list kubelet kubeadm kubectl --showduplicates | grep 1.18.14 | sort -r
kubelet.x86_64                       1.18.14-0                       kubernetes
kubectl.x86_64                       1.18.14-0                       kubernetes
kubeadm.x86_64                       1.18.14-0                       kubernetes

# 安裝指定版本
yum install -y kubelet-1.18.14 kubeadm-1.18.14 kubectl-1.18.14 --disableexcludes=kubernetes

# 確定版本一致
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean", BuildDate:"2020-12-18T12:08:45Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean", BuildDate:"2020-12-18T12:11:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

$ kubelet --version
Kubernetes v1.18.14
  • 配置：上面我們提到過，需要將 Container Runtime、kubelet 配置成使用 systemd 來作為 cgroup 驅動，以此使系統更為穩定，
# vi /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
  • 啟動：
$ systemctl daemon-reload
$ systemctl restart kubelet
$ systemctl enable --now kubelet
$ systemctl status kubelet

注意：kubelet.sercice 每隔幾秒就會重啟一次，回圈等待 kubeadm 的指令，

初始化 Master 主控制平面節點

kubeadm init 的作業流

kubeadm init 命令通過執行下列步驟來啟動一個 Kubernetes Master：

  1. 預檢測系統狀態：當出現 ERROR 時就退出 kubeadm，除非問題得到解決或者顯式指定了 --ignore-preflight-errors=<錯誤串列> 引數，此外，也會出現 WARNING，

  2. 生成一個自簽名的 CA 證書來為每個系統組件建立身份標識：可以顯式指定 --cert-dir CA 中心目錄（默認為 /etc/kubernetes/pki），在該目錄下方式 CA 證書、密鑰等檔案，API Server 證書將為任何 --apiserver-cert-extra-sans 引數值提供附加的 SAN 條目，必要時將其小寫，

  3. 將 kubeconfig 檔案寫入 /etc/kubernetes/ 目錄：以便 kubelet、Controller Manager 和 Scheduler 用來連接到 API Server，它們都有自己的身份標識，同時生成一個名為 admin.conf 的獨立的 kubeconfig 檔案，用于管理操作，

  4. 為 API Server、Controller Manager 和 Scheduler 生成 static Pod 的清單檔案：存放在 /etc/kubernetes/manifests 下，kubelet 會輪訓監視這個目錄，在啟動 Kubernetes 時用于創建系統組件的 Pod，假使沒有提供一個外部的 etcd 服務的話，也會為 etcd 生成一份額外的 static Pod 清單檔案，

待 Master 的 static Pods 都運行正常后，kubeadm init 的作業流程才會繼續往下執行，

  1. 對 Master 使用 Labels 和 Stain mark（污點標記）：以此隔離生產作業負載不會調度到 Master 上，

  2. 生成 Token：將來其他的 Node 可使用該 Token 向 Master 注冊自己，也可以顯式指定 --token 提供 Token String，

  3. 為了使 Node 能夠遵照啟動引導令牌（Bootstrap Tokens）和 TLS 啟動引導（TLS bootstrapping）這兩份檔案中描述的機制加入到 Cluster 中，kubeadm 會執行所有的必要配置：

    1. 創建一個 ConfigMap 提供添加 Node 到 Cluster 中所需的資訊，并為該 ConfigMap 設定相關的 RBAC 訪問規則，
    2. 允許啟動引導令牌訪問 CSR 簽名 API，
    3. 配置自動簽發新的 CSR 請求，

  4. 通過 API Server 安裝一個 DNS 服務器（CoreDNS）和 kube-proxy：注意，盡管現在已經部署了 DNS 服務器，但直到安裝 CNI 時才調度它，

執行初始化

注意 1：因為我們要部署高可用集群，所以必須使用選項 --control-plane-endpoint 指定 API Server 的 HA Endpoint，
注意 2：由于 kubeadm 默認從 k8s.grc.io 下載所需鏡像，因此可以通過 --image-repository 指定阿里云的鏡像倉庫，
注意 3：如果顯式指定 --upload-certs，則意味著在擴展冗余 Master 時，你必須要手動地將 CA 證書從主控制平面節點復制到將要加入的冗余控制平面節點上，推薦使用，

  • 初始化：
kubeadm init \
  --control-plane-endpoint "192.168.0.100" \
  --kubernetes-version "1.18.14" \
  --pod-network-cidr "10.0.0.0/8" \
  --service-cidr "172.16.0.0/16" \
  --token "abcdef.0123456789abcdef" \
  --token-ttl "0" \
  --image-repository registry.aliyuncs.com/google_containers \
  --upload-certs

W1221 00:02:43.240309   10942 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.14
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.16.0.1 192.168.0.148 192.168.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.0.148 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.0.148 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 00:02:47.773223   10942 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 00:02:47.774303   10942 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.117265 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
    --control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1
  • 查看 Pods：檢查 Master 的組件是否齊全，
# 配置 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ kubectl get pod -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-7ff77c879f-fh9vb               0/1     Pending   0          23m
coredns-7ff77c879f-qmk7z               0/1     Pending   0          23m
etcd-k8s-master-1                      1/1     Running   0          24m
kube-apiserver-k8s-master-1            1/1     Running   0          24m
kube-controller-manager-k8s-master-1   1/1     Running   0          24m
kube-proxy-7hx55                       1/1     Running   0          23m
kube-scheduler-k8s-master-1            1/1     Running   0          24m
  • 查看 Images：
$ docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy                v1.18.14            8e6bca1d4e68        2 days ago          117MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.18.14            f17e261f4c8a        2 days ago          173MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.18.14            b734a959c6fb        2 days ago          162MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.18.14            95660d582e82        2 days ago          95.3MB
registry.aliyuncs.com/google_containers/pause                     3.2                 80d28bedfe5d        10 months ago       683kB
registry.aliyuncs.com/google_containers/coredns                   1.6.7               67da37a9a360        10 months ago       43.8MB
registry.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        14 months ago       288MB
  • 查看 Containers：
$ docker ps -a
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
f9a068b890d7        8e6bca1d4e68                                        "/usr/local/bin/kube…"   2 minutes ago       Up 2 minutes                            k8s_kube-proxy_kube-proxy-7hx55_kube-system_aacb0da3-16ec-414c-b138-856e2b470bb9_0
3b6adfa0b1a5        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-proxy-7hx55_kube-system_aacb0da3-16ec-414c-b138-856e2b470bb9_0
dcc47de63e50        f17e261f4c8a                                        "kube-apiserver --ad…"   3 minutes ago       Up 3 minutes                            k8s_kube-apiserver_kube-apiserver-k8s-master-1_kube-system_c693bd1fadf036d8e2e4df0afd49f062_0
53afb7fbe8c0        b734a959c6fb                                        "kube-controller-man…"   3 minutes ago       Up 3 minutes                            k8s_kube-controller-manager_kube-controller-manager-k8s-master-1_kube-system_f75424d466cd7197fb8095b0f59ea8d9_0
a4101a231c1b        303ce5db0e90                                        "etcd --advertise-cl…"   3 minutes ago       Up 3 minutes                            k8s_etcd_etcd-k8s-master-1_kube-system_f85e02734d6479f3bb3e468eea87fd3a_0
197f510ff6c5        95660d582e82                                        "kube-scheduler --au…"   3 minutes ago       Up 3 minutes                            k8s_kube-scheduler_kube-scheduler-k8s-master-1_kube-system_0213a889f9350758ac9847629f75db19_0
3a4590590093        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-controller-manager-k8s-master-1_kube-system_f75424d466cd7197fb8095b0f59ea8d9_0
4bbdc99a7a68        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-apiserver-k8s-master-1_kube-system_c693bd1fadf036d8e2e4df0afd49f062_0
19488127c269        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_etcd-k8s-master-1_kube-system_f85e02734d6479f3bb3e468eea87fd3a_0
e67d2f7a27b0        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-scheduler-k8s-master-1_kube-system_0213a889f9350758ac9847629f75db19_0
  • 測驗 API Server LB 是否正常：
$ nc -v 192.168.0.100 6443
Connection to 192.168.0.100 port 6443 [tcp/sun-sr-https] succeeded!

注意：上述 Token 的過期時間是 24 小時，如果希望在 24 小時之后繼續添加不通的節點，則需要重新生產 Token：

# 新建 Token
kubeadm token create
# output: 5didvk.d09sbcov8ph2amjw

# 新建 --discovery-token-ca-cert-hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'
# output: 8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78

清理或重新進行初始化

要再次運行 kubeadm init，你必須首先卸載集群，可以在 Master 上觸發盡力而為的清理：

kubeadm reset

Reset 程序不會重置或清除 iptables 規則或 IPVS 表，如果你希望重置 iptables 或 IPVS，則必須手動進行：

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm -C

根據需求調整引數，重新進行初始化：

kubeadm init <args>

或許，徹底洗掉節點：

kubectl delete node <node name>

添加 Master 冗余控制平面節點

在第一個 Master 初始化完畢之后，我們就可以繼續添加冗余 Master 節點了，

  • 添加 k8s-master-2：
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
    --control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-2 localhost] and IPs [192.168.0.112 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-2 localhost] and IPs [192.168.0.112 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.16.0.1 192.168.0.112 192.168.0.100]
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
W1221 00:30:18.978564   27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 00:30:18.986650   27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 00:30:18.987613   27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
{"level":"warn","ts":"2020-12-21T00:30:34.018+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.0.112:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-master-2 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

	Run 'kubectl get nodes' to see this node join the cluster.
  • 添加 k8s-master-3：
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
    --control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544
  • 檢查 Master 節點數：
# 配置 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
k8s-master-1   NotReady   master   35m     v1.18.14
k8s-master-2   NotReady   master   8m14s   v1.18.14
k8s-master-3   NotReady   master   2m30s   v1.18.14

添加 Node 作業負載節點

部署完高可用的 Master 控制平面之后，我們就可以注冊任意個 Node 作業負載節點了，

  • 添加 Node：
kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1

W1221 00:39:36.256784   29495 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
  • 檢查 Node：
$ kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
k8s-master-1   NotReady   master   37m     v1.18.14
k8s-master-2   NotReady   master   10m     v1.18.14
k8s-master-3   NotReady   master   4m24s   v1.18.14
k8s-node-1     NotReady   <none>   51s     v1.18.14
k8s-node-2     NotReady   <none>   48s     v1.18.14

安裝 CNI 網路插件

我們選擇使用 Calico SDN 方案，官方檔案：https://docs.projectcalico.org/about/about-calico

注意：

  1. Pod 網路不得與任何主機網路重疊：所以我們在執行 kubeadm init 時顯式指定了 --pod-network-cidr 引數，
  2. 確保 CNI 網路插件支持 RBAC（基于角色的訪問控制），
  3. 確保 CNI 支持 IPv6 或 IPv4v6，當你需要使用的時候，
  • 安裝：

$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
  • 檢查 Calico Pods：
$ watch kubectl get pod --all-namespaces

Every 2.0s: kubectl get pod --all-namespaces                                                                                                                                        Mon Dec 21 13:12:30 2020

NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7dbc97f587-nqrxv   1/1     Running   0          9m34s
kube-system   calico-node-47xmr                          1/1     Running   0          9m34s
kube-system   calico-node-8zwbg                          1/1     Running   0          9m34s
kube-system   calico-node-dj4qt                          1/1     Running   0          9m34s
kube-system   calico-node-glqqj                          1/1     Running   0          9m34s
kube-system   calico-node-jb4t4                          1/1     Running   0          9m34s
kube-system   coredns-7ff77c879f-fh9vb                   1/1     Running   0          13h
kube-system   coredns-7ff77c879f-qmk7z                   1/1     Running   0          13h
kube-system   etcd-k8s-master-1                          1/1     Running   0          13h
kube-system   etcd-k8s-master-2                          1/1     Running   0          12h
kube-system   etcd-k8s-master-3                          1/1     Running   0          12h
kube-system   kube-apiserver-k8s-master-1                1/1     Running   0          13h
kube-system   kube-apiserver-k8s-master-2                1/1     Running   0          12h
kube-system   kube-apiserver-k8s-master-3                1/1     Running   0          12h
kube-system   kube-controller-manager-k8s-master-1	 1/1     Running   1          13h
kube-system   kube-controller-manager-k8s-master-2	 1/1     Running   0          12h
kube-system   kube-controller-manager-k8s-master-3	 1/1     Running   0          12h
kube-system   kube-proxy-7hx55                           1/1     Running   0          13h
kube-system   kube-proxy-8dmc4                           1/1     Running   0          12h
kube-system   kube-proxy-9clqs                           1/1     Running   0          12h
kube-system   kube-proxy-cq5tq                           1/1     Running   0          12h
kube-system   kube-proxy-pm79q                           1/1     Running   0          12h
kube-system   kube-scheduler-k8s-master-1                1/1     Running   1          13h
kube-system   kube-scheduler-k8s-master-2                1/1     Running   0          12h
kube-system   kube-scheduler-k8s-master-3                1/1     Running   0          12h
  • 檢查 Cluster 節點的狀態：安裝了 CNI 之后節點的狀態應該是 Ready 的，
$ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
k8s-master-1   Ready    master   13h   v1.18.14
k8s-master-2   Ready    master   12h   v1.18.14
k8s-master-3   Ready    master   12h   v1.18.14
k8s-node-1     Ready    <none>   12h   v1.18.14
k8s-node-2     Ready    <none>   12h   v1.18.14

轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/239156.html

標籤：AI

上一篇：LearnOpenGL從入門到入魔(2)：如何使用OpenGL？

下一篇：比SkLearn更好用的機器學習工具-PyCaret

標籤雲
其他(157675) Python(38076) JavaScript(25376) Java(17977) C(15215) 區塊鏈(8255) C＃(7972) AI(7469) 爪哇(7425) MySQL(7132) html(6777) 基礎類(6313) sql(6102) 熊猫(6058) PHP(5869) 数组(5741) R(5409) Linux(5327) 反应(5209) 腳本語言(PerlPython)(5129) 非技術區(4971) Android(4554) 数据框(4311) css(4259) 节点.js(4032) C語言(3288) json(3245) 列表(3129) 扑(3119) C++語言(3117) 安卓(2998) 打字稿(2995) VBA(2789) Java相關(2746) 疑難問題(2699) 细绳(2522) 單片機工控(2479) iOS(2429) ASP.NET(2402) MongoDB(2323) 麻木的(2285) 正则表达式(2254) 字典(2211) 循环(2198) 迅速(2185) 擅长(2169) 镖(2155) 功能(1967) .NET技术(1958) Web開發(1951) python-3.x(1918) HtmlCss(1915) 弹簧靴(1913) C++(1909) xml(1889) PostgreSQL(1872) .NETCore(1853) 谷歌表格(1846) Unity3D(1843) for循环(1842)

熱門瀏覽
  • 網閘典型架構簡述

    網閘架構一般分為兩種：三主機的三系統架構網閘和雙主機的2+1架構網閘。 三主機架構分別為內端機、外端機和仲裁機。三機無論從軟體和硬體上均各自獨立。首先從硬體上來看，三機都用各自獨立的主板、記憶體及存盤設備。從軟體上來看，三機有各自獨立的作業系統。這樣能達到完全的三機獨立。對于“2+1”系統，“2”分為 ......

    uj5u.com 2020-09-10 02:00:44 more
  • 如何從xshell上傳檔案到centos linux虛擬機里

    如何從xshell上傳檔案到centos linux虛擬機里及：虛擬機CentOs下執行 yum -y install lrzsz命令，出現錯誤：鏡像無法找到軟體包 前言 一、安裝lrzsz步驟 二、上傳檔案 三、遇到的問題及解決方案 總結 前言 提示：其實很簡單，往虛擬機上安裝一個上傳檔案的工具 ......

    uj5u.com 2020-09-10 02:00:47 more
  • 一、SQLMAP入門

    一、SQLMAP入門 1、判斷是否存在注入 sqlmap.py -u 網址/id=1 id=1不可缺少。當注入點后面的引數大于兩個時。需要加雙引號， sqlmap.py -u "網址/id=1&uid=1" 2、判斷文本中的請求是否存在注入 從文本中加載http請求，SQLMAP可以從一個文本檔案中 ......

    uj5u.com 2020-09-10 02:00:50 more
  • Metasploit 簡單使用教程

    metasploit 簡單使用教程 浩先生， 2020-08-28 16:18:25 分類專欄： kail 網路安全 linux 文章標簽： linux資訊安全 編輯 著作權 metasploit 使用教程 前言 一、Metasploit是什么？ 二、準備作業 三、具體步驟 前言 Msfconsole ......

    uj5u.com 2020-09-10 02:00:53 more
  • 游戲逆向之驅動層與用戶層通訊

    驅動層代碼： #pragma once #include <ntifs.h> #define add_code CTL_CODE(FILE_DEVICE_UNKNOWN,0x800,METHOD_BUFFERED,FILE_ANY_ACCESS) /* 更多游戲逆向視頻www.yxfzedu.com ......

    uj5u.com 2020-09-10 02:00:56 more
  • 北斗電力時鐘（北斗授時服務器）讓網路資料更精準

    北斗電力時鐘（北斗授時服務器）讓網路資料更精準 北斗電力時鐘（北斗授時服務器）讓網路資料更精準 京準電子科技官微——ahjzsz 近幾年，資訊技術的得了快速發展，互聯網在逐漸普及，其在人們生活和生產中都得到了廣泛應用，并且取得了不錯的應用效果。計算機網路資訊在電力系統中的應用，一方面使電力系統的運行 ......

    uj5u.com 2020-09-10 02:01:03 more
  • 【CTF】CTFHub 技能樹 彩蛋 writeup

    ?碎碎念 CTFHub：https://www.ctfhub.com/ 筆者入門CTF時時剛開始刷的是bugku的舊平臺，后來才有了CTFHub。 感覺不論是網頁UI設計，還是題目質量，賽事跟蹤，工具軟體都做得很不錯。 而且因為獨到的金幣制度的確讓人有一種想去刷題賺金幣的感覺。 個人還是非常喜歡這個 ......

    uj5u.com 2020-09-10 02:04:05 more
  • 02windows基礎操作

    我學到了一下幾點 Windows系統目錄結構與滲透的作用 常見Windows的服務詳解 Windows埠詳解 常用的Windows注冊表詳解 hacker DOS命令詳解（net user / type /md /rd/ dir /cd /net use copy、批處理 等） 利用dos命令制作 ......

    uj5u.com 2020-09-10 02:04:18 more
  • 03.Linux基礎操作

    我學到了以下幾點 01Linux系統介紹02系統安裝，密碼啊破解03Linux常用命令04LAMP 01LINUX windows： win03 8 12 16 19 配置不繁瑣 Linux：redhat,centos(紅帽社區版)，Ubuntu server,suse unix:金融機構，證券，銀 ......

    uj5u.com 2020-09-10 02:04:30 more
  • 05HTML

    01HTML介紹 02頭部標簽講解03基礎標簽講解04表單標簽講解 HTML前段語言 js1.了解代碼2.根據代碼 懂得挖掘漏洞 （POST注入/XSS漏洞上傳）3.黑帽seo 白帽seo 客戶網站被黑帽植入劫持代碼如何處理4.熟悉html表單 <html><head><title>TDK標題，描述 ......

    uj5u.com 2020-09-10 02:04:36 more
最新发布
  • 2023年最新微信小程式抓包教程

    01 開門見山 隔一個月發一篇文章，不過分。 首先回顧一下《微信系結手機號資料庫被脫庫事件》，我也是第一時間得知了這個訊息，然后跟蹤了整件事情的經過。下面是這起事件的相關截圖以及近日流出的一萬條資料樣本： 個人認為這件事也沒什么，還不如關注一下之前45億快遞資料查詢渠道疑似在近日復活的訊息。 訊息是 ......

    uj5u.com 2023-04-20 08:48:24 more
  • web3 產品介紹：metamask 錢包 使用最多的瀏覽器插件錢包

    Metamask錢包是一種基于區塊鏈技術的數字貨幣錢包，它允許用戶在安全、便捷的環境下管理自己的加密資產。Metamask錢包是以太坊生態系統中最流行的錢包之一，它具有易于使用、安全性高和功能強大等優點。 本文將詳細介紹Metamask錢包的功能和使用方法。 一、 Metamask錢包的功能 數字資 ......

    uj5u.com 2023-04-20 08:47:46 more
  • vulnhub_Earth

    前言 靶機地址->>>vulnhub_Earth 攻擊機ip：192.168.20.121 靶機ip：192.168.20.122 參考文章 https://www.cnblogs.com/Jing-X/archive/2022/04/03/16097695.html https://www.cnb ......

    uj5u.com 2023-04-20 07:46:20 more
  • 從4k到42k，軟體測驗工程師的漲薪史，給我看哭了

    清明節一過，盲猜大家已經無心上班，在數著日子準備過五一，但一想到銀行卡里的余額……瞬間心情就不美麗了。最近，2023年高校畢業生就業調查顯示，本科畢業月平均起薪為5825元。調查一出，便有很多同學表示自己又被平均了。看著這一資料，不免讓人想到前不久中國青年報的一項調查：近六成大學生認為畢業10年內會 ......

    uj5u.com 2023-04-20 07:44:00 more
  • 最新版本 Stable Diffusion 開源 AI 繪畫工具之中文自動提詞篇

    🎈 標簽生成器 由于輸入正向提示詞 prompt 和反向提示詞 negative prompt 都是使用英文，所以對學習母語的我們非常不友好 使用網址：https://tinygeeker.github.io/p/ai-prompt-generator 這個網址是為了讓大家在使用 AI 繪畫的時候 ......

    uj5u.com 2023-04-20 07:43:36 more
  • 漫談前端自動化測驗演進之路及測驗工具分析

    隨著前端技術的不斷發展和應用程式的日益復雜，前端自動化測驗也在不斷演進。隨著 Web 應用程式變得越來越復雜，自動化測驗的需求也越來越高。如今，自動化測驗已經成為 Web 應用程式開發程序中不可或缺的一部分，它們可以幫助開發人員更快地發現和修復錯誤，提高應用程式的性能和可靠性。 ......

    uj5u.com 2023-04-20 07:43:16 more
  • CANN開發實踐：4個DVPP記憶體問題的典型案例解讀

    摘要：由于DVPP媒體資料處理功能對存放輸入、輸出資料的記憶體有更高的要求（例如，記憶體首地址128位元組對齊），因此需呼叫專用的記憶體申請介面，那么本期就分享幾個關于DVPP記憶體問題的典型案例，并給出原因分析及解決方法。 本文分享自華為云社區《FAQ_DVPP記憶體問題案例》，作者：昇騰CANN。 DVPP ......

    uj5u.com 2023-04-20 07:43:03 more
  • msf學習

    msf學習 以kali自帶的msf為例 一、msf核心模塊與功能 msf模塊都放在/usr/share/metasploit-framework/modules目錄下 1、auxiliary 輔助模塊，輔助滲透（埠掃描、登錄密碼爆破、漏洞驗證等） 2、encoders 編碼器模塊，主要包含各種編碼 ......

    uj5u.com 2023-04-20 07:42:59 more
  • Halcon軟體安裝與界面簡介

    1. 下載Halcon17版本到到本地 2. 雙擊安裝包后 3. 步驟如下 1.2 Halcon軟體安裝 界面分為四大塊 1. Halcon的五個助手 1) 影像采集助手：與相機連接，設定相機引數，采集影像 2) 標定助手：九點標定或是其它的標定，生成標定檔案及內參外參，可以將像素單位轉換為長度單位 ......

    uj5u.com 2023-04-20 07:42:17 more
  • 在MacOS下使用Unity3D開發游戲

    第一次發博客，先發一下我的游戲開發環境吧。 去年2月份買了一臺MacBookPro2021 M1pro(以下簡稱mbp)，這一年來一直在用mbp開發游戲。我大致分享一下我的開發工具以及使用體驗。 1、Unity 官網鏈接： https://unity.cn/releases 我一般使用的Apple ......

    uj5u.com 2023-04-20 07:40:19 more

Copyright © 2010-2020 有解無憂