二進制方式部署Kubernetes高可用集群
文章目錄
- 二進制方式部署Kubernetes高可用集群
- 1.環境準備
- 1.1.Kubernetes高可用集群部署方式
- 1.2.Kubernetes集群棄用docker容器
- 1.3.Kubernetes集群所需的證書
- 1.4.環境準備
- 1.5.安裝cfssl證書生成工具
- 2.作業系統初始化配置
- 3.部署Etcd集群
- 3.1.使用cfssl證書工具生成etcd證書
- 3.2.部署etcd集群
- 4.部署Docker服務
- 4.1.安裝docker
- 4.2.為docker創建systemctl啟動腳本
- 5.部署kubernetes master節點
- 5.1.使用cfssl生成apiserver的證書檔案
- 5.2.解壓二進制檔案復制相關組件程式
- 5.3.部署kube-apiserver組件
- 5.3.1.創建kube-apiserver組態檔
- 5.3.2.創建TLS Bootstrapping檔案
- 5.3.4.創建systemctl腳本管理apiserver
- 5.3.5.啟動kube-apiserver組件
- 5.4.部署kube-controller-manage組件
- 5.4.1.創建kube-controller-manage組態檔
- 5.4.2.生成kubeconfig檔案
- 5.4.3.創建systemctl腳本管理服務
- 5.4.4.啟動kube-controller-manage組件
- 5.5.部署kube-scheduler組件
- 5.5.1.創建kube-scheduler組態檔
- 5.5.2.生成kubeconfig檔案
- 5.5.3.創建systemctl腳本管理服務
- 5.5.4.啟動kube-scheduler組件
- 5.6.準備kubectl所需的kubeconfig檔案連接集群
- 5.6.1.生成證書檔案
- 5.6.2.生成kubeconfig檔案
- 5.6.3.使用kubectl查看集群連接資訊
- 6.在master節點部署node節點相關組件
- 6.1.在集群授權kubelet-bootstrap用戶允許請求證書
- 6.2.在master節點部署kubelet組件
- 6.2.1.將kubelet和kube-proxy的二進制檔案拷貝至對應目錄
- 6.2.2.創建kubelet組態檔
- 6.2.3.創建kubelet-config.yaml引陣列態檔
- 6.2.4.創建bootstrap-kubeconfig檔案
- 6.2.5.創建systemctl腳本并啟動服務
- 6.2.6.將master節點作為node加入集群內部
- 6.3.在master節點部署kube-proxy
- 6.3.1.創建kube-proxy組態檔
- 6.3.2.創建kube-proxy引陣列態檔
- 6.3.3.生成kubeconfig檔案
- 6.3.4.創建systemctl腳本管理服務
- 6.3.4.啟動kube-proxy組件
- 6.4.授權apiserver訪問kubelet
- 7.部署kubernetes calico網路組件
- 8.部署kubernetes node節點
- 8.1.解壓二進制檔案復制相關組件程式
- 8.2.部署kubelet組件
- 8.2.1.創建kubelet組態檔
- 8.2.2.創建kubelet引陣列態檔
- 8.2.3.創建bootstrap-kubeconfig檔案
- 8.2.4.創建systemctl腳本并啟動服務
- 8.2.5.master節點授權同意node節點加入集群
- 8.3.部署kube-proxy組件
- 8.3.1.創建kube-proxy組態檔
- 8.3.2.創建kube-proxy引陣列態檔
- 8.3.3.生成kube-config檔案
- 8.3.4.創建systemctl腳本管理服務
- 8.3.5.啟動kube-proxy組件
- 8.4.快速增加新的node節點
- 8.4.1.將kubelet和kube-proxy目錄拷貝至新的node節點
- 8.4.2.配置并啟動kubelet組件
- 8.4.3.master節點授權新node節點的請求
- 8.4.4.配置并啟動kube-proxy組件
- 9.為集群部署coredns組件
- 9.1.部署coredns組件
- 9.2.運行一個busybox容器測驗dns
- 10.擴容master節點組建kubernetes高可用集群
- 10.1.kubernetes高可用架構概念
- 10.2.在集群中新增一個etcd節點
- 10.2.1.首先新增加一臺單點的etcd
- 10.2.2.在現有etcd集群任意一個節點上增加新etcd節點
- 10.2.3.配置新增的etcd節點加入集群
- 10.2.4.配置kube-apiserver增加新的etcd節點
- 10.3.部署master-2節點
- 10.3.1.部署docker
- 10.3.2.部署kubernetes各個組件
- 10.3.3.授權master2節點加入集群
- 10.4.部署Nginx+Keepalived實作kubernetes高可用集群
- 10.4.1.部署Nginx負載均衡
- 10.4.2.部署keepalived雙機熱備
- 10.4.3.使用VIP訪問kubernetes服務
- 10.4.4.測驗keepalived高可用
- 10.5.切換kubernetes集群為高可用模式
- 11.測驗kubernetes高可用集群
- 12.在kubernetes集群運行一套服務驗證集群的可用性
- 12.1.創建資源yaml檔案
- 12.2.創建資源并進行測驗
- 13.部署kubernetes dashboard
- 13.1.部署dashboard
- 13.2.訪問dashboard
1.環境準備
1.1.Kubernetes高可用集群部署方式
目前生產環境部署Kubernetes建主要有兩種方式：
kubeadm：提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群，kubeadm安裝的k8s集群，所有的k8s組件都是以pod形式運行，
二進制包：從github上下載發行版的二進制包，手動部署每個組件，組成kubernetes集群，
Kubeadm降低部署成本，從而屏蔽了很多細節，遇到問題很難排查，如果想更容易可控，推薦使用二進制包部署Kubernetes集群，雖然手動部署麻煩點，期間可以學習很多作業原理，也利于后期維護，
1.2.Kubernetes集群棄用docker容器
在k8s平臺中，為了解決與容器運行時，比如docker的集成問題，在早期社區推出CRI介面，以支持更多的容器，當我們使用Docker作為容器運行時，首先kubelet呼叫dockershim的CRI容器介面連接docker行程，最后由docker啟動容器，
在k8s1.23版本中，k8s計劃棄用kubelet中的dockershim介面，dockershim介面一旦棄用，kubelet去呼叫CRL時就沒有可以與docker建立連接的一個介面，從而導致k8s棄用docker容器，
1.3.Kubernetes集群所需的證書
k8s所有組件均采用https加密通信，這些組件一般由兩套根證書生成：一個用于k8s apiserver一個用于etcd資料庫，
按照角色來分，證書分為管理節點和作業節點，
- 管理節點：指controller-manager和scheduler連接apiserver所需的客戶端證書，
- 作業節點：值kubelet和kube-proxy連接apiserver所需要的客戶端證書，而一般都會啟用Bootstrap TLS機制，所以kubelet的證書初次啟動會向apiserver申請頒發證書，由controller-manager組件自動頒發，
- 圖中紅線是k8s各個組件通過攜帶k8s自建證書頒發機構生成的客戶端證書訪問apiserver，圖中藍線是k8sapiserver組件通過etcd頒發的客戶端證書與etcd建立連接，
1.4.環境準備
| 角色 | IP | 組件 |
|---|---|---|
| binary-k8s-master1 | 192.168.20.10 | kube-apiserver、kube-controller-manage、kube-scheduler、kubelet、kube-proxy、docker、etcd、nginx、keepalived |
| binary-k8s-master2 | 192.168.20.11 | kube-apiserver、kube-controller-manage、kube-scheduler、kubelet、kube-proxy、docker、nginx、keepalived、etcd（擴容節點） |
| binary-k8s-node1 | 192.168.20.12 | kubelet、kube-proxy、docker、etcd |
| binary-k8s-node2 | 192.168.20.13 | kubelet、kube-proxy、docker、etcd |
| 負載均衡器IP | 192.168.20.9 | （作用于kube-apiserver的地址） |
首先部署一套單master節點的kubernetes集群，然后在增加一臺master節點，形成高可用集群，
單master節點的kubernetes集群服務器規劃，
| 角色 | IP | 組件 |
|---|---|---|
| binary-k8s-master1 | 192.168.20.10 | kube-apiserver、kube-controller-manage、kube-schedule、etcd |
| binary-k8s-node1 | 192.168.20.12 | kubelet、kube-proxy、docker、etcd |
| binary-k8s-node2 | 192.168.20.13 | kubelet、kube-proxy、docker、etcd |
1.5.安裝cfssl證書生成工具
cfssl是一個開源的證書管理工具，使用json檔案生成證書，相比openssl更方便使用，
[root@binary-k8s-master1 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@binary-k8s-master1 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@binary-k8s-master1 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[root@binary-k8s-master1 ~]# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
[root@binary-k8s-master1 ~]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[root@binary-k8s-master1 ~]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
[root@binary-k8s-master1 ~]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
2.作業系統初始化配置
1.關閉防火墻
systemctl stop firewalld
systemctl disable firewalld
2.關閉selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
3.關閉交換磁區
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
4.配置hosts
cat >> /etc/hosts << EOF
192.168.20.10 binary-k8s-master1
192.168.20.12 binary-k8s-node1
192.168.20.13 binary-k8s-node2
EOF
5.優化內核引數
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
3.部署Etcd集群
etcd是一個分布式鍵值存盤系統，kubernetes使用etcd進行資料存盤，為解決etcd單點故障，采用集群方式部署，3臺組組建集群，可以壞1臺，如果有5臺可以壞2臺，
| 節點名稱 | IP |
|---|---|
| etcd-1 | 192.168.20.10 |
| etcd-2 | 192.168.20.12 |
| etcd-3 | 192.168.20.13 |
3.1.使用cfssl證書工具生成etcd證書
1.生成CA自簽頒發機構證書
[root@binary-k8s-master1 ~/TLS/etcd]# vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
[root@binary-k8s-master1 ~/TLS/etcd]# vim ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
[root@binary-k8s-master1 ~/TLS/etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/08/27 17:16:49 [INFO] generating a new CA key and certificate from CSR
2021/08/27 17:16:49 [INFO] generate received request
2021/08/27 17:16:49 [INFO] received CSR
2021/08/27 17:16:49 [INFO] generating key: rsa-2048
2021/08/27 17:16:49 [INFO] encoded CSR
2021/08/27 17:16:49 [INFO] signed certificate with serial number 595276170535764345591605360849177409156623041535
2.使用自簽CA簽發Etcd HTTPS證書
申請證書的json檔案中有一個hosts欄位，這個欄位的值就是etcd集群的IP地址，可以多寫幾個IP，作為預留IP，方便擴容etcd集群，
1.創建證書申請檔案
[root@binary-k8s-master1 ~/TLS/etcd]# vim server-csr.json
{
"CN": "etcd",
"hosts": [
"192.168.20.10",
"192.168.20.11", #預留ip
"192.168.20.12",
"192.168.20.13"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
2.生成證書
[root@binary-k8s-master1 ~/TLS/etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/08/27 17:17:08 [INFO] generate received request
2021/08/27 17:17:08 [INFO] received CSR
2021/08/27 17:17:08 [INFO] generating key: rsa-2048
2021/08/27 17:17:08 [INFO] encoded CSR
2021/08/27 17:17:08 [INFO] signed certificate with serial number 390637014214409356442509482537912246480465374076
2021/08/27 17:17:08 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
3.查看生產的證書檔案
[root@binary-k8s-master1 ~/TLS/etcd]# ll
總用量 36
-rw-r--r--. 1 root root 288 8月 27 17:16 ca-config.json
-rw-r--r--. 1 root root 956 8月 27 17:16 ca.csr
-rw-r--r--. 1 root root 210 8月 27 17:16 ca-csr.json
-rw-------. 1 root root 1675 8月 27 17:16 ca-key.pem
-rw-r--r--. 1 root root 1265 8月 27 17:16 ca.pem
-rw-r--r--. 1 root root 1021 8月 27 17:17 server.csr
-rw-r--r--. 1 root root 311 8月 27 17:17 server-csr.json
-rw-------. 1 root root 1679 8月 27 17:17 server-key.pem
-rw-r--r--. 1 root root 1346 8月 27 17:17 server.pem
3.2.部署etcd集群
1.下載etcd二進制檔案
下載地址：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
部署二進制的程式集群最簡單的方式就是在其中一臺上面部署，然后將所有的檔案scp到其他機器上修改配置，一套集群也就完成了，
將下載好的檔案上傳至所有etcd節點，
etcd組態檔解釋
#[Member]
ETCD_NAME="etcd-1" #節點名稱
ETCD_DATA_DIR="/data/etcd/data" #資料目錄
ETCD_LISTEN_PEER_URLS="https://192.168.20.10:2380" #集群通信地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.10:2379,http://127.0.0.1:2379" #客戶端訪問的監聽地址，在這里加一個http://127.0.0.1:2379，在當前節點查集群資訊時就不需要指定證書去查詢了
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.10:2380" #集群通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.10:2379,http://127.0.0.1:2379" #客戶端通告地址，，在這里加一個http://127.0.0.1:2379，在當前節點查集群資訊時就不需要指定證書去查詢了
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.10:2380,etcd-2=https://192.168.20.12:2380,etcd-3=https://192.168.20.13:2380" #集群節點地址
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #集群的唯一標識
ETCD_INITIAL_CLUSTER_STATE="new" #加入集群的狀態，new為新集群，existing表示加入現有集群
2.部署etcd-1節點
1.創建程式目錄
[root@binary-k8s-master1 ~]# mkdir /data/etcd/{bin,conf,ssl,data} -p
2.解壓二進制檔案
[root@binary-k8s-master1 ~]# tar xf etcd-v3.4.9-linux-amd64.tar.gz
3.將二進制命令移動到制定出程式目錄
[root@binary-k8s-master1 ~]# mv etcd-v3.4.9-linux-amd64/etcd* /data/etcd/bin/
4.編輯組態檔
[root@binary-k8s-master1 ~]# vim /data/etcd/conf/etcd.conf
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.20.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.10:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.10:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.10:2379,http://127.0.0.1:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.10:2380,etcd-2=https://192.168.20.12:2380,etcd-3=https://192.168.20.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
5.撰寫systemctl控制腳本
[root@binary-k8s-master1 ~]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/data/etcd/conf/etcd.conf
ExecStart=/data/etcd/bin/etcd \
--cert-file=/data/etcd/ssl/server.pem \
--key-file=/data/etcd/ssl/server-key.pem \
--peer-cert-file=/data/etcd/ssl/server.pem \
--peer-key-file=/data/etcd/ssl/server-key.pem \
--trusted-ca-file=/data/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
6.復制證書檔案
[root@binary-k8s-master1 ~]# cp TLS/etcd/*.pem /data/etcd/ssl/
7.啟動etcd-1節點
[root@binary-k8s-master1 ~]# systemctl daemon-reload
[root@binary-k8s-master1 ~]# systemctl start etcd
[root@binary-k8s-master1 ~]# systemctl enable etcd
#第一個節點啟動會一直處于其中中的狀態，只有當第二個節點也啟動了，第一個節點才能啟動成功，因為集群版的etcd至少需要2個節點才能成功運行
3.配置etcd-2節點和etcd-3節點
部署完一個節點，可以直接將目錄拷貝至其他節點，省去安裝的一些步驟，
1.推送etcd目錄
[root@binary-k8s-master1 ~]# scp -rp /data/etcd root@192.168.20.12:/data
[root@binary-k8s-master1 ~]# scp -rp /data/etcd root@192.168.20.13:/data
2.推送systemctl啟動檔案
[root@binary-k8s-master1 ~]# scp -rp /usr/lib/systemd/system/etcd.service root@192.168.20.12:/usr/lib/systemd/system/
[root@binary-k8s-master1 ~]# scp -rp /usr/lib/systemd/system/etcd.service root@192.168.20.13:/usr/lib/systemd/system/
3.修改etcd-2組態檔
[root@binary-k8s-node1 ~]# vim /data/etcd/conf/etcd.conf
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.20.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.12:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.12:2379,http://127.0.0.1:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.10:2380,etcd-2=https://192.168.20.12:2380,etcd-3=https://192.168.20.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
4.修改etcd-3組態檔
[root@binary-k8s-node2 ~]# vim /data/etcd/conf/etcd.conf
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.20.13:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.13:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.13:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.13:2379,http://127.0.0.1:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.10:2380,etcd-2=https://192.168.20.12:2380,etcd-3=https://192.168.20.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
5.啟動etcd-1和etcd-2
[root@binary-k8s-node1 ~]# systemctl daemon-reload
[root@binary-k8s-node1 ~]# systemctl start etcd
[root@binary-k8s-node1 ~]# systemctl enable etcd
------------
[root@binary-k8s-node2 ~]# systemctl daemon-reload
[root@binary-k8s-node2 ~]# systemctl start etcd
[root@binary-k8s-node2 ~]# systemctl enable etcd
4.查看集群狀態
etcd-1啟動時會一直處于等待狀態，當etcd-2執行啟動命令時會立即啟動成功，并且etcd-1也會立刻啟動成功，
查看etcd的日志可以使用這個命令：[root@binary-k8s-master1 ~]# journalctl -u etcd -f
1.查看服務埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep etcd
tcp 0 0 192.168.20.10:2379 0.0.0.0:* LISTEN 9625/etcd
tcp 0 0 192.168.20.10:2380 0.0.0.0:* LISTEN 9625/etcd
2.查看集群狀態
#如果組態檔中2379埠沒有加一個127.0.0.1則這樣查看集群狀態
[root@binary-k8s-master1 ~]# ETCDCTL_API=3 /data/etcd/bin/etcdctl --cacert=/data/etcd/ssl/ca.pem --cert=/data/etcd/ssl/server.pem --key=/data/etcd/ssl/server-key.pem --endpoints="https://192.168.20.10:2379,https://192.168.20.12:2379,https://192.168.20.13:2379" endpoint health --write-out=table
+----------------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.20.10:2379 | true | 32.322714ms | |
| https://192.168.20.12:2379 | true | 31.524079ms | |
| https://192.168.20.13:2379 | true | 38.985949ms | |
+----------------------------+--------+-------------+-------+
#如果組態檔匯總2379埠加了一個127.0.0.1則可以使用如下方式查看集群資訊無需指定證書
[root@binary-k8s-master1 /data/etcd/conf]# /data/etcd/bin/etcdctl member list --write-out=table
+------------------+---------+--------+----------------------------+---------------------------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+--------+----------------------------+---------------------------------------------------+------------+
| 12446003b2a53d43 | started | etcd-2 | https://192.168.20.12:2380 | https://127.0.0.1:2379,https://192.168.20.12:2379 | false |
| 51ae3f86f3783687 | started | etcd-1 | https://192.168.20.10:2380 | http://127.0.0.1:2379,https://192.168.20.10:2379 | false |
| 667c9c7ba890c3f7 | started | etcd-3 | https://192.168.20.13:2380 | http://127.0.0.1:2379,https://192.168.20.13:2379 | false |
+------------------+---------+--------+----------------------------+---------------------------------------------------+------------+
組態檔狀態
etcd啟動成功的日志
4.部署Docker服務
所有kubernetes節點都需要安裝docker服務，包括master和node節點，
docker二進制檔案下載地址：https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
4.1.安裝docker
1.解壓二進制包
tar zxf docker-19.03.9.tgz
2.將可執行命令移動到系統路徑
mv docker/* /usr/bin
3.創建組態檔
mkdir /etc/docker
vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://9wn5tbfh.mirror.aliyuncs.com"]
}
4.2.為docker創建systemctl啟動腳本
1.撰寫啟動腳本
vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
2.啟動docker
systemctl daemon-reload
systemctl start docker
systemctl enable docker
5.部署kubernetes master節點
部署二進制的kubernetes組件大致可分為如下幾個步驟：
- 1.解壓二進制檔案
- 2.復制二進制程式到指定目錄
- 3.創建組件組態檔
- 4.生成組件的kubeconfig檔案
- 5.創建systemctl腳本管理服務
- 6.啟動組件
kubernetes集群的master節點和node節點的二進制檔案都從github上下載，master和node相關的所有組件都在一個程式包中，
下載地址： https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md
5.1.使用cfssl生成apiserver的證書檔案
1.生成CA自簽頒發機構證書
1.準備CA組態檔
[root@binary-k8s-master1 ~/TLS/k8s]# vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
[root@binary-k8s-master1 ~/TLS/k8s]# vim ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
2.生成證書檔案
[root@binary-k8s-master1 ~/TLS/k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/09/01 16:20:42 [INFO] generating a new CA key and certificate from CSR
2021/09/01 16:20:42 [INFO] generate received request
2021/09/01 16:20:42 [INFO] received CSR
2021/09/01 16:20:42 [INFO] generating key: rsa-2048
2021/09/01 16:20:43 [INFO] encoded CSR
2021/09/01 16:20:43 [INFO] signed certificate with serial number 90951268335404710707183639990677546638148434604
2.使用自簽CA簽發apiserver HTTPS證書
簽發的客戶端證書組態檔中的hosts欄位要包含所有Master/LB/VIP的IP地址，Node節點的地址可寫可不寫，
1.準備客戶端組態檔
[root@binary-k8s-master1 ~/TLS/k8s]# vim kube-apiserver-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.20.10",
"192.168.20.11",
"192.168.20.12",
"192.168.20.13",
"192.168.20.9",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2.生成證書檔案
[root@binary-k8s-master1 ~/TLS/k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
2021/09/01 16:30:24 [INFO] generate received request
2021/09/01 16:30:24 [INFO] received CSR
2021/09/01 16:30:24 [INFO] generating key: rsa-2048
2021/09/01 16:30:25 [INFO] encoded CSR
2021/09/01 16:30:25 [INFO] signed certificate with serial number 714472722509814799589567099679496298525490716083
2021/09/01 16:30:25 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
3.查看生產的證書檔案
[root@binary-k8s-master1 ~/TLS/k8s]# ll
總用量 36
-rw-r--r--. 1 root root 294 9月 1 16:20 ca-config.json
-rw-r--r--. 1 root root 1001 9月 1 16:20 ca.csr
-rw-r--r--. 1 root root 264 9月 1 16:20 ca-csr.json
-rw-------. 1 root root 1679 9月 1 16:20 ca-key.pem
-rw-r--r--. 1 root root 1359 9月 1 16:20 ca.pem
-rw-r--r--. 1 root root 1277 9月 1 16:30 kube-apiserver.csr
-rw-r--r--. 1 root root 602 9月 1 16:30 kube-apiserver-csr.json
-rw-------. 1 root root 1679 9月 1 16:30 kube-apiserver-key.pem
-rw-r--r--. 1 root root 1643 9月 1 16:30 kube-apiserver.pem
5.2.解壓二進制檔案復制相關組件程式
[root@binary-k8s-master1 ~]# mkdir /data/kubernetes/{bin,config,ssl,logs} -p
[root@binary-k8s-master1 ~]# tar xf kubernetes-server-linux-amd64.tar.gz
[root@binary-k8s-master1 ~]# cd kubernetes/server/bin/
[root@binary-k8s-master1 ~/kubernetes/server/bin]# cp kube-apiserver kube-scheduler kube-controller-manager /data/kubernetes/bin/
[root@binary-k8s-master1 ~/kubernetes/server/bin]# cp kubectl /usr/bin/
5.3.部署kube-apiserver組件
5.3.1.創建kube-apiserver組態檔
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--etcd-servers=https://192.168.20.10:2379,https://192.168.20.12:2379,https://192.168.20.13:2379 \
--bind-address=192.168.20.10 \
--secure-port=6443 \
--advertise-address=192.168.20.10 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/data/kubernetes/config/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/data/kubernetes/ssl/kube-apiserver.pem \
--kubelet-client-key=/data/kubernetes/ssl/kube-apiserver-key.pem \
--tls-cert-file=/data/kubernetes/ssl/kube-apiserver.pem \
--tls-private-key-file=/data/kubernetes/ssl/kube-apiserver-key.pem \
--client-ca-file=/data/kubernetes/ssl/ca.pem \
--service-account-key-file=/data/kubernetes/ssl/ca-key.pem \
--service-account-issuer=api \
--service-account-signing-key-file=/data/kubernetes/ssl/kube-apiserver-key.pem \
--etcd-cafile=/data/etcd/ssl/ca.pem \
--etcd-certfile=/data/etcd/ssl/server.pem \
--etcd-keyfile=/data/etcd/ssl/server-key.pem \
--requestheader-client-ca-file=/data/kubernetes/ssl/ca.pem \
--proxy-client-cert-file=/data/kubernetes/ssl/kube-apiserver.pem \
--proxy-client-key-file=/data/kubernetes/ssl/kube-apiserver-key.pem \
--requestheader-allowed-names=kubernetes \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/data/kubernetes/logs/k8s-audit.log"
組態檔各引數含義
| 配置引數 | 含義 |
|---|---|
| –logtostderr | 是否開啟日志 |
| –v | 日志的等級，等級越高內容越詳細 |
| –log-dir | 日志存放路徑 |
| –etcd-servers | etcd集群地址 |
| –bind-address | 監聽地址，也就是本機 |
| –secure-port | https安全埠 |
| –advertise-address | 集群通告地址 |
| –allow-privileged | 企業授權 |
| –service-cluster-ip-range | service資源IP地址段 |
| –enable-admission-plugins | 準入控制模塊 |
| –authorization-mode | 認證授權，啟用RBAC授權和節點自管理 |
| –enable-bootstrap-token-auth | 啟用TLS bootstrap機制，啟用之后kubelet可以自動給node節頒發證書 |
| –token-auth-file | bootstrap token檔案路徑 |
| –service-node-port-range | Service nodeport型別默認分配埠范圍 |
| –kubelet-client-certificate | apiserver訪問kubelet的客戶端證書檔案 |
| –kubelet-client-key | apiserver訪問kubelet的客戶端私鑰檔案 |
| –tls-cert-file | apiserver https證書 |
| –tls-private-key-file | apiserver https證書 |
| –client-ca-file | ca證書路徑 |
| –service-account-key-file | ca私鑰路徑 |
| –service-account-issuer | sa賬號授權過期時間的一個配置，1.20以后才有的特性 |
| –service-account-signing-key-file | 證書檔案路徑 |
| –etcd-cafile | etcd ca證書檔案路徑 |
| –etcd-certfile | etcd 客戶端證書檔案路徑 |
| –etcd-keyfile | etcd 客戶端私鑰檔案路徑 |
| –requestheader-client-ca-file | 聚合層相關配置 |
| –proxy-client-cert-file | 聚合層相關配置 |
| –proxy-client-key-file | 聚合層相關配置 |
| –requestheader-allowed-names | 聚合層相關配置 |
| –requestheader-extra-headers-prefix | 聚合層相關配置 |
| –enable-aggregator-routing | 聚合層相關配置 |
5.3.2.創建TLS Bootstrapping檔案
TLS Bootstraping：Master apiserver啟用TLS認證后，Node節點kubelet和kube-proxy要與kube-apiserver進行通信，必須使用CA簽發的有效證書才可以，當Node節點很多時，這種客戶端證書頒發需要大量作業，同樣也會增加集群擴展復雜度，為了簡化流程，Kubernetes引入了TLS bootstraping機制來自動頒發客戶端證書，kubelet會以一個低權限用戶自動向apiserver申請證書，kubelet的證書由apiserver動態簽署，所以強烈建議在Node上使用這種方式，目前主要用于kubelet，kube-proxy還是由我們統一頒發一個證書，
TLS bootstraping 作業流程：
kubelet首先取查找bootstraping組態檔，然后去連接apiserver，開始驗證bootstrap token檔案，再驗證證書檔案，最后頒發證書啟動成功，否則就會啟動失敗，
1.生成一個token值
[root@binary-k8s-master1 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
d7f96b0d86c574d0f64a713608db092
2.創建token檔案
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/token.csv
d7f96b0d86c574d0f64a713608db0922,kubelet-bootstrap,10001,"system:node-bootstrapper"
#格式：token，用戶名，UID，用戶組
5.3.4.創建systemctl腳本管理apiserver
[root@binary-k8s-master1 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/config/kube-apiserver.conf
ExecStart=/data/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
5.3.5.啟動kube-apiserver組件
1.拷貝我們需要的證書檔案
[root@binary-k8s-master1 ~]# cp TLS/k8s/*.pem /data/kubernetes/ssl/
2.啟動kube-apiserver
[root@binary-k8s-master1 ~]# systemctl daemon-reload
[root@binary-k8s-master1 ~]# systemctl start kube-apiserver
[root@binary-k8s-master1 ~]# systemctl enable kube-apiserver
3.查看埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep kube
tcp 0 0 192.168.20.10:6443 0.0.0.0:* LISTEN 28546/kube-apiserve
5.4.部署kube-controller-manage組件
5.4.1.創建kube-controller-manage組態檔
組態檔含義
–kubeconfig：指定用于連接apiserver的kubeconfig組態檔
–leader-elect：用于高可用集群，自動選舉
–cluster-signing-cert-file：指定CA證書檔案，為kubelet自動頒發證書
–cluster-signing-key-file：指定CA私鑰檔案，為kubelet自動頒發證書
–cluster-signing-duration：證書過期時間
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--leader-elect=true \
--kubeconfig=/data/kubernetes/config/kube-controller-manager.kubeconfig \
--bind-address=192.168.20.10 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/data/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/data/kubernetes/ssl/ca-key.pem \
--root-ca-file=/data/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/data/kubernetes/ssl/ca-key.pem \
--cluster-signing-duration=87600h0m0s"
5.4.2.生成kubeconfig檔案
kube-controller-manage利用kubeconfig組態檔連接apiserver，
kubeconfig檔案中包括集群apiserver地址、證書檔案、用戶，
1.由于kubeconfig需要證書檔案的支持，因此要生成一個證書
[root@binary-k8s-master1 ~/TLS/k8s]# vim kube-controller-manager-csr.json
{
"CN": "system:kube-controller-manager",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
[root@binary-k8s-master1 ~/TLS/k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2021/09/01 16:36:18 [INFO] generate received request
2021/09/01 16:36:18 [INFO] received CSR
2021/09/01 16:36:18 [INFO] generating key: rsa-2048
l2021/09/01 16:36:19 [INFO] encoded CSR
2021/09/01 16:36:19 [INFO] signed certificate with serial number 719101376219834763931271155238486242405063666906
2021/09/01 16:36:19 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@binary-k8s-master1 ~/TLS/k8s]# cp kube-controller-manager*pem /data/kubernetes/ssl/
2.生成kubeconfig檔案
#在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-master1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/data/kubernetes/config/kube-controller-manager.kubeconfig
#在kubeconfig檔案中增加證書檔案資訊
[root@binary-k8s-master1 ~]# kubectl config set-credentials kube-controller-manager \
--client-certificate=/data/kubernetes/ssl/kube-controller-manager.pem \
--client-key=/data/kubernetes/ssl/kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/data/kubernetes/config/kube-controller-manager.kubeconfig
#在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-master1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-controller-manager \
--kubeconfig=/data/kubernetes/config/kube-controller-manager.kubeconfig
3.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-master1 ~]# kubectl config use-context default --kubeconfig=/data/kubernetes/config/kube-controller-manager.kubeconfig
5.4.3.創建systemctl腳本管理服務
[root@binary-k8s-master1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/config/kube-controller-manager.conf
ExecStart=/data/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
5.4.4.啟動kube-controller-manage組件
1.啟動服務
[root@binary-k8s-master1 ~]# systemctl daemon-reload
[root@binary-k8s-master1 ~]# systemctl start kube-controller-manager
[root@binary-k8s-master1 ~]# systemctl enable kube-controller-manager
2.查看埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep kube
tcp 0 0 192.168.20.10:6443 0.0.0.0:* LISTEN 28546/kube-apiserve
tcp 0 0 192.168.20.10:10257 0.0.0.0:* LISTEN 28941/kube-controll
tcp6 0 0 :::10252 :::* LISTEN 28941/kube-controll
5.5.部署kube-scheduler組件
5.5.1.創建kube-scheduler組態檔
組態檔解釋
–kubeconfig：指定kubeconfig檔案
–leader-elect：選舉
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--leader-elect \
--kubeconfig=/data/kubernetes/config/kube-scheduler.kubeconfig \
--bind-address=192.168.20.10"
5.5.2.生成kubeconfig檔案
生成kubeconfig連接集群apiserver，
kube-schedule利用kubeconfig組態檔連接apiserver，
kubeconfig檔案中包括集群apiserver地址、證書檔案、用戶，
1.創建證書組態檔
[root@binary-k8s-master1 ~/TLS/k8s]# vim kube-scheduler-csr.json
{
"CN": "system:kube-scheduler",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
2.生成證書
[root@binary-k8s-master1 ~/TLS/k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
2021/09/02 14:50:40 [INFO] generate received request
2021/09/02 14:50:40 [INFO] received CSR
2021/09/02 14:50:40 [INFO] generating key: rsa-2048
2021/09/02 14:50:42 [INFO] encoded CSR
2021/09/02 14:50:42 [INFO] signed certificate with serial number 91388852050290848663498441480862532526947759393
2021/09/02 14:50:42 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
3.查看證書檔案
[root@binary-k8s-master1 ~/TLS/k8s]# ll
總用量 68
-rw-r--r--. 1 root root 294 9月 1 16:20 ca-config.json
-rw-r--r--. 1 root root 1001 9月 1 16:20 ca.csr
-rw-r--r--. 1 root root 264 9月 1 16:20 ca-csr.json
-rw-------. 1 root root 1679 9月 1 16:20 ca-key.pem
-rw-r--r--. 1 root root 1359 9月 1 16:20 ca.pem
-rw-r--r--. 1 root root 1277 9月 1 16:30 kube-apiserver.csr
-rw-r--r--. 1 root root 602 9月 1 16:30 kube-apiserver-csr.json
-rw-------. 1 root root 1679 9月 1 16:30 kube-apiserver-key.pem
-rw-r--r--. 1 root root 1643 9月 1 16:30 kube-apiserver.pem
-rw-r--r--. 1 root root 1045 9月 1 16:36 kube-controller-manager.csr
-rw-r--r--. 1 root root 255 9月 1 16:46 kube-controller-manager-csr.json
-rw-------. 1 root root 1675 9月 1 16:36 kube-controller-manager-key.pem
-rw-r--r--. 1 root root 1436 9月 1 16:36 kube-controller-manager.pem
-rw-r--r--. 1 root root 1029 9月 2 14:50 kube-scheduler.csr
-rw-r--r--. 1 root root 245 9月 2 14:50 kube-scheduler-csr.json
-rw-------. 1 root root 1675 9月 2 14:50 kube-scheduler-key.pem
-rw-r--r--. 1 root root 1424 9月 2 14:50 kube-scheduler.pem
4.拷貝證書檔案至指定路徑
[root@binary-k8s-master1 ~/TLS/k8s]# cp kube-scheduler*.pem /data/kubernetes/ssl/
5.生成kubeconfig檔案
#在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-master1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/data/kubernetes/config/kube-scheduler.kubeconfig
#在kubeconfig檔案中增加證書檔案資訊
[root@binary-k8s-master1 ~]# kubectl config set-credentials kube-scheduler \
--client-certificate=/data/kubernetes/ssl/kube-scheduler.pem \
--client-key=/data/kubernetes/ssl/kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/data/kubernetes/config/kube-scheduler.kubeconfig
#在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-master1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-scheduler \
--kubeconfig=/data/kubernetes/config/kube-scheduler.kubeconfig
6.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-master1 ~]# kubectl config use-context default --kubeconfig=/data/kubernetes/config/kube-scheduler.kubeconfig
5.5.3.創建systemctl腳本管理服務
[root@binary-k8s-master1 ~]# vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/config/kube-scheduler.conf
ExecStart=/data/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
5.5.4.啟動kube-scheduler組件
1.啟動服務
[root@binary-k8s-master1 ~]# systemctl daemon-reload
[root@binary-k8s-master1 ~]# systemctl start kube-scheduler
[root@binary-k8s-master1 ~]# systemctl enable kube-scheduler
2.查看埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep kube
tcp 0 0 192.168.20.10:6443 0.0.0.0:* LISTEN 28546/kube-apiserve
tcp 0 0 192.168.20.10:10257 0.0.0.0:* LISTEN 28941/kube-controll
tcp 0 0 192.168.20.10:10259 0.0.0.0:* LISTEN 6127/kube-scheduler
tcp6 0 0 :::10251 :::* LISTEN 6127/kube-scheduler
tcp6 0 0 :::10252 :::* LISTEN 28941/kube-controll
5.6.準備kubectl所需的kubeconfig檔案連接集群
kubectl想要連接集群對各種資源進行操作，需要有一個kubeconfig檔案連接apiserver才可以對集群進行操作，也就是kubeadm安裝k8s集群后在master節點生成的/root/.kube目錄，這個目錄中的config檔案就是kubectl用于連接apiserver的kubeconfig檔案，
5.6.1.生成證書檔案
1.創建證書組態檔
[root@binary-k8s-master1 ~/TLS/k8s]# vim kubectl-csr.json
{
"CN": "kubectl",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
2.生成證書
[root@binary-k8s-master1 ~/TLS/k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubectl-csr.json | cfssljson -bare kubectl
2021/09/02 17:20:44 [INFO] generate received request
2021/09/02 17:20:44 [INFO] received CSR
2021/09/02 17:20:44 [INFO] generating key: rsa-2048
2021/09/02 17:20:45 [INFO] encoded CSR
2021/09/02 17:20:45 [INFO] signed certificate with serial number 398472525484598388169457456772550114435870340604
2021/09/02 17:20:45 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
3.查看生成的證書檔案
[root@binary-k8s-master1 ~/TLS/k8s]# ll
總用量 84
-rw-r--r--. 1 root root 294 9月 1 16:20 ca-config.json
-rw-r--r--. 1 root root 1001 9月 1 16:20 ca.csr
-rw-r--r--. 1 root root 264 9月 1 16:20 ca-csr.json
-rw-------. 1 root root 1679 9月 1 16:20 ca-key.pem
-rw-r--r--. 1 root root 1359 9月 1 16:20 ca.pem
-rw-r--r--. 1 root root 1277 9月 1 16:30 kube-apiserver.csr
-rw-r--r--. 1 root root 602 9月 1 16:30 kube-apiserver-csr.json
-rw-------. 1 root root 1679 9月 1 16:30 kube-apiserver-key.pem
-rw-r--r--. 1 root root 1643 9月 1 16:30 kube-apiserver.pem
-rw-r--r--. 1 root root 1045 9月 1 16:36 kube-controller-manager.csr
-rw-r--r--. 1 root root 255 9月 1 16:46 kube-controller-manager-csr.json
-rw-------. 1 root root 1675 9月 1 16:36 kube-controller-manager-key.pem
-rw-r--r--. 1 root root 1436 9月 1 16:36 kube-controller-manager.pem
-rw-r--r--. 1 root root 1013 9月 2 17:20 kubectl.csr
-rw-r--r--. 1 root root 231 9月 2 17:20 kubectl-csr.json
-rw-------. 1 root root 1679 9月 2 17:20 kubectl-key.pem
-rw-r--r--. 1 root root 1403 9月 2 17:20 kubectl.pem
-rw-r--r--. 1 root root 1029 9月 2 14:50 kube-scheduler.csr
-rw-r--r--. 1 root root 245 9月 2 14:50 kube-scheduler-csr.json
-rw-------. 1 root root 1675 9月 2 14:50 kube-scheduler-key.pem
-rw-r--r--. 1 root root 1424 9月 2 14:50 kube-scheduler.pem
4.拷貝證書檔案到指定目錄
[root@binary-k8s-master1 ~/TLS/k8s]# \cp kubectl*.pem /data/kubernetes/ssl/
5.6.2.生成kubeconfig檔案
1.在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-master1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/root/.kube/config
2.在kubeconfig檔案中增加證書檔案資訊
[root@binary-k8s-master1 ~]# kubectl config set-credentials cluster-admin \
--client-certificate=/data/kubernetes/ssl/kubectl.pem \
--client-key=/data/kubernetes/ssl/kubectl-key.pem \
--embed-certs=true \
--kubeconfig=/root/.kube/config
3.在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-master1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=cluster-admin \
--kubeconfig=/root/.kube/config
4.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-master1 ~]# kubectl config use-context default --kubeconfig=/root/.kube/config
5.6.3.使用kubectl查看集群連接資訊
至此master節點相關組件部署完成，
[root@binary-k8s-master1 ~]# kubectl get node
No resources found
[root@binary-k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
6.在master節點部署node節點相關組件
6.1.在集群授權kubelet-bootstrap用戶允許請求證書
在此處做了這一步之后，node節點加入集群時就不需要做了，
[root@binary-k8s-master1 ~]# kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
6.2.在master節點部署kubelet組件
由于master也需要啟動某些pod，比如calico組件都是以pod方式運行的，因此在master節點也需要kubelet和kube-proxy組件，
6.2.1.將kubelet和kube-proxy的二進制檔案拷貝至對應目錄
[root@binary-k8s-master1 ~]# cp kubernetes/server/bin/{kubelet,kube-proxy} /data/kubernetes/bin/
6.2.2.創建kubelet組態檔
組態檔含義：
–hostname-override：節點名稱，集群中唯一
–network-plugin：啟用CNI網路
–kubeconfig：指定自動生成的kubeconfig檔案路徑，用于連接apiserver
–bootstrap-kubeconfig：指定首次啟動向apiserver申請證書的kubeconfig檔案路徑
–config：配置引數檔案路徑
–cert-dir：kubelet證書生成目錄路徑
–pod-infra-container-image：pod容器的根容器
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--hostname-override=binary-k8s-master1 \
--network-plugin=cni \
--kubeconfig=/data/kubernetes/config/kubelet.kubeconfig \
--bootstrap-kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig \
--config=/data/kubernetes/config/kubelet-config.yml \
--cert-dir=/data/kubernetes/ssl \
--pod-infra-container-image=pause-amd64:3.0"
6.2.3.創建kubelet-config.yaml引陣列態檔
kubelet和kube-proxy服務的引數配置是以yaml形式來配置的
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kubelet-config.yml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0 #監聽地址
port: 10250 #監聽埠
readOnlyPort: 10255
cgroupDriver: cgroupfs #驅動引擎
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /data/kubernetes/ssl/ca.pem #ca證書檔案路徑
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110 #可運行的pod的數量
6.2.4.創建bootstrap-kubeconfig檔案
1.在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-master1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
2.在kubeconfig檔案中增加token資訊
[root@binary-k8s-master1 ~]# kubectl config set-credentials "kubelet-bootstrap" \
--token=d7f96b0d86c574d0f64a713608db0922 \
--kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
#這個token就是之前生成的/data/kubernetes/config/token.csv中的token
3.在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-master1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
4.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-master1 ~]# kubectl config use-context default --kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
6.2.5.創建systemctl腳本并啟動服務
1.創建systemctl腳本
[root@binary-k8s-master1 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/data/kubernetes/config/kubelet.conf
ExecStart=/data/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
2.啟動kubelet服務
[root@binary-k8s-master1 ~]# systemctl daemon-reload
[root@binary-k8s-master1 ~]# systemctl start kubelet
[root@binary-k8s-master1 ~]# systemctl enable kubelet
6.2.6.將master節點作為node加入集群內部
當kubelet組件啟動成功后，就會想apiserver發送一個請求加入集群的資訊，只有當master節點授權同意后，才可以正常加入，雖然是master節點部署的node組件，但是也會發生一個加入集群的資訊，需要master同意，
當kubelet啟動之后，首先會在證書目錄生成一個kubelet-client.key.tmp這個檔案，當使用kubectl certificate approve命令授權成功node的請求之后，kubelet-client.key.tmp小時，隨之會生成一個kubelet-client-current.pem的證書檔案，用于與apiserver建立連接，此時再使用kubectl get node就會看到節點資訊了，
擴展：如果后期想要修改node的名稱，那么就把生成的kubelet證書檔案全部洗掉，然后使用kubectl delete node洗掉該節點，在修改kubelet組態檔中該節點的名稱，然后使用kubectl delete csr洗掉授權資訊，再重啟kubelet生成新的授權資訊，然后授權通過即可看到新的名字的node節點，
只有當授權通過后，kubelet生成了證書檔案，kubelet的埠才會被啟動
注意：當kubelet的授權被master請求通后，kube-proxy啟動成功后，節點才會正真的加入集群，即使kubectl get node看到的節點是Ready，該節點也是不可用的，必須當kube-proxy啟動完畢后，這個節點才算正真的啟動完畢<
1.直接在master節點上執行如下命令獲取請求串列
[root@binary-k8s-master1 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-JN8q9WljA6oupdWZ2mVO-TOIq2sLodFdkyL5fu6Ius4 4s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
2.授權同意此節點加入集群
[root@binary-k8s-master1 ~]# kubectl certificate approve node-csr-JN8q9WljA6oupdWZ2mVO-TOIq2sLodFdkyL5fu6Ius4
certificatesigningrequest.certificates.k8s.io/node-csr-JN8q9WljA6oupdWZ2mVO-TOIq2sLodFdkyL5fu6Ius4 approved
3.查看node節點
[root@binary-k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
binary-k8s-master1 NotReady <none> 6s v1.20.4
#此時master節點已經出現在集群節點串列中了
4.查看kubelet埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 29092/kubelet
tcp 0 0 127.0.0.1:41132 0.0.0.0:* LISTEN 29092/kubelet
tcp6 0 0 :::10250 :::* LISTEN 29092/kubelet
tcp6 0 0 :::10255 :::* LISTEN 29092/kubelet
6.3.在master節點部署kube-proxy
6.3.1.創建kube-proxy組態檔
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--config=/data/kubernetes/config/kube-proxy-config.yml"
6.3.2.創建kube-proxy引陣列態檔
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0 #監聽地址
metricsBindAddress: 0.0.0.0:10249 #監聽埠
clientConnection:
kubeconfig: /data/kubernetes/config/kube-proxy.kubeconfig #kubeconfig檔案用于和apiserver通信
hostnameOverride: binary-k8s-master1 #當前節點名稱
clusterCIDR: 10.244.0.0/16
6.3.3.生成kubeconfig檔案
1.創建證書組態檔
[root@binary-k8s-master1 ~/TLS/k8s]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2.生成證書
[root@binary-k8s-master1 ~/TLS/k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2021/09/03 16:04:23 [INFO] generate received request
2021/09/03 16:04:23 [INFO] received CSR
2021/09/03 16:04:23 [INFO] generating key: rsa-2048
2021/09/03 16:04:24 [INFO] encoded CSR
2021/09/03 16:04:24 [INFO] signed certificate with serial number 677418055440191127932354470575565723194258386145
2021/09/03 16:04:24 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
3.查看證書檔案
[root@binary-k8s-master1 ~/TLS/k8s]# ll *proxy*
-rw-r--r--. 1 root root 1009 9月 3 16:04 kube-proxy.csr
-rw-r--r--. 1 root root 230 9月 3 16:04 kube-proxy-csr.json
-rw-------. 1 root root 1679 9月 3 16:04 kube-proxy-key.pem
-rw-r--r--. 1 root root 1403 9月 3 16:04 kube-proxy.pem
4.拷貝證書檔案至指定路徑
[root@binary-k8s-master1 ~/TLS/k8s]# cp kube-proxy*.pem /data/kubernetes/ssl/
5.生成kubeconfig檔案
#在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-master1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
#在kubeconfig檔案中增加證書檔案資訊
[root@binary-k8s-master1 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/data/kubernetes/ssl/kube-proxy.pem \
--client-key=/data/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
#在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-master1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
6.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-master1 ~]# kubectl config use-context default --kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
6.3.4.創建systemctl腳本管理服務
[root@binary-k8s-master1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/data/kubernetes/config/kube-proxy.conf
ExecStart=/data/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
6.3.4.啟動kube-proxy組件
1.啟動服務
[root@binary-k8s-master1 ~]# systemctl daemon-reload
[root@binary-k8s-master1 ~]# systemctl start kube-proxy
[root@binary-k8s-master1 ~]# systemctl enable kube-proxy
2.查看埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep kube-proxy
tcp6 0 0 :::10249 :::* LISTEN 29354/kube-proxy
tcp6 0 0 :::10256 :::* LISTEN 29354/kube-proxy
6.4.授權apiserver訪問kubelet
如果不收取apiserver訪問kubelet，那么將無法使用kubectl查看集群的一些資訊，比如kubectl logs就無法使用，
實際上就是創建一個rbac資源讓apiserver能否訪問kubelet的資源，
1.撰寫資源yaml檔案
[root@binary-k8s-master1 ~]# vim apiserver-to-kubelet-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
2.創建資源
[root@binary-k8s-master1 ~]# kubectl apply -f apiserver-to-kubelet-rbac.yaml
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
7.部署kubernetes calico網路組件
在6中master節點已經加入集群，但是狀態一直處于NotReady狀態，就是由于集群沒有網路組件導致的，部署好網路組件，master節點立馬會成為Ready狀態，
1.部署calico
[root@binary-k8s-master1 ~]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
2.查看資源狀態
[root@binary-k8s-master1 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-97769f7c7-bnwcl 1/1 Running 0 11m
calico-node-mghdj 1/1 Running 0 11m
3.查看master節點的狀態
[root@binary-k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 99m v1.20.4
8.部署kubernetes node節點
8.1.解壓二進制檔案復制相關組件程式
以下操作僅在node1節點操作即可，
1.準備二進制程式
[root@binary-k8s-node1 ~]# tar xf kubernetes-server-linux-amd64.tar.gz
[root@binary-k8s-node1 ~]# mkdir -p /data/kubernetes/{bin,config,ssl,logs}
[root@binary-k8s-node1 ~]# cp kubernetes/server/bin/{kubelet,kube-proxy} /data/kubernetes/bin/
[root@binary-k8s-node1 ~]# cp kubernetes/server/bin/kubectl /usr/bin/
2.將master節點上的證書檔案拷貝至node節點
[root@binary-k8s-master1 ~]# scp -rp /data/kubernetes/ssl/* binary-k8s-node1:/data/kubernetes/ssl/
[root@binary-k8s-master1 ~]# scp -rp /data/kubernetes/config/token.csv root@binary-k8s-node1:/data/kubernetes/config
3.洗掉從master節點上拷貝過來的kubelet證書
[root@binary-k8s-node1 ~]# rm -rf /data/kubernetes/ssl/kubelet-client-*
#kubelet證書需要洗掉，當node節點的kubelet啟動后會生成臨時證書檔案，當master授權通過后，證書檔案產生
8.2.部署kubelet組件
8.2.1.創建kubelet組態檔
[root@binary-k8s-node1 ~]# vim /data/kubernetes/config/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--hostname-override=binary-k8s-node1 #注意修改節點名稱 \
--network-plugin=cni \
--kubeconfig=/data/kubernetes/config/kubelet.kubeconfig \
--bootstrap-kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig \
--config=/data/kubernetes/config/kubelet-config.yml \
--cert-dir=/data/kubernetes/ssl \
--pod-infra-container-image=pause-amd64:3.0"
8.2.2.創建kubelet引陣列態檔
[root@binary-k8s-node1 ~]# vim /data/kubernetes/config/kubelet-config.yml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /data/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
8.2.3.創建bootstrap-kubeconfig檔案
1.在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-node1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
2.在kubeconfig檔案中增加token資訊
[root@binary-k8s-master1 ~]# kubectl config set-credentials "kubelet-bootstrap" \
--token=d7f96b0d86c574d0f64a713608db0922 \
--kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
#這個token就是之前生成的/data/kubernetes/config/token.csv中的token
3.在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-master1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
4.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-master1 ~]# kubectl config use-context default --kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig
8.2.4.創建systemctl腳本并啟動服務
1.撰寫systemctl服務腳本
[root@binary-k8s-node1 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/data/kubernetes/config/kubelet.conf
ExecStart=/data/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
2.啟動kubelet服務
[root@binary-k8s-node1 ~]# systemctl daemon-reload
[root@binary-k8s-node1 ~]# systemctl start kubelet
[root@binary-k8s-node1 ~]# systemctl enable kubelet
[root@binary-k8s-node1 ~]# systemctl status kubelet
8.2.5.master節點授權同意node節點加入集群
kubelet服務啟動后，會生成一個臨時證書檔案，然后向master節點發送一個csr授權請求，當master節點授權同意后，kubelet-clinet證書檔案生成，埠也隨之啟動，節點正常加入集群，
csr串列的授權資訊也會自動清空，如果master節點的授權不及時，也可以重啟一下kubelet重新發送一個csr請求，
1.在node節點查看臨時證書檔案
[root@binary-k8s-node1 ~]# ll /data/kubernetes/ssl/*.tmp
-rw-------. 1 root root 227 9月 6 11:28 kubelet-client.key.tmp
#只要kubelet啟動就會產生一個臨時證書檔案
2.在master節點查看csr授權請求串列
[root@binary-k8s-master1 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-JmO7N8iDvyD0D-2Pu7_yHJ3ngZ5xXfA_TwRevqmHAXI 11s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
3.授權通過
[root@binary-k8s-master1 ~]# kubectl certificate approve node-csr-JmO7N8iDvyD0D-2Pu7_yHJ3ngZ5xXfA_TwRevqmHAXI
certificatesigningrequest.certificates.k8s.io/node-csr-JmO7N8iDvyD0D-2Pu7_yHJ3ngZ5xXfA_TwRevqmHAXI approved
4.此時臨時檔案已洗掉，已經生成kubelet證書檔案
[root@binary-k8s-node1 ~]# ll /data/kubernetes/ssl/kubelet-client*
-rw-------. 1 root root 1236 9月 6 11:28 kubelet-client-2021-09-06-11-28-54.pem
lrwxrwxrwx. 1 root root 59 9月 6 11:28 kubelet-client-current.pem -> /data/kubernetes/ssl/kubelet-client-2021-09-06-11-28-54.pem
5.node1節點成功加入集群
[root@binary-k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
binary-k8s-master1 Ready <none> 2d22h v1.20.4
binary-k8s-node1 Ready <none> 4h59m v1.20.4
6.在node節點查看kubelet服務的埠
[root@binary-k8s-node1 ~]# netstat -lnpt | grep kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 29220/kubelet
tcp 0 0 127.0.0.1:44151 0.0.0.0:* LISTEN 29220/kubelet
tcp6 0 0 :::10250 :::* LISTEN 29220/kubelet
tcp6 0 0 :::10255 :::* LISTEN 29220/kubelet
8.3.部署kube-proxy組件
8.3.1.創建kube-proxy組態檔
[root@binary-k8s-node1 ~]# vim /data/kubernetes/config/kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--config=/data/kubernetes/config/kube-proxy-config.yml"
8.3.2.創建kube-proxy引陣列態檔
[root@binary-k8s-node1 ~]# vim /data/kubernetes/config/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0 #監聽地址
metricsBindAddress: 0.0.0.0:10249 #監聽埠
clientConnection:
kubeconfig: /data/kubernetes/config/kube-proxy.kubeconfig #kubeconfig檔案用于和apiserver通信
hostnameOverride: binary-k8s-node1 #當前節點名稱
clusterCIDR: 10.244.0.0/16
8.3.3.生成kube-config檔案
由于kube-proxy的證書檔案在8.1中已經從master節點拷貝到node節點了，因此直接生成kubeconfig檔案即可，
集群中不同節點的組件都要用同一個證書檔案，
1.生成kubeconfig檔案
#在kubeconfig檔案中增加集群apiserver資訊
[root@binary-k8s-node1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server="https://192.168.20.10:6443" \
--kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
#在kubeconfig檔案中增加證書檔案資訊
[root@binary-k8s-node1 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/data/kubernetes/ssl/kube-proxy.pem \
--client-key=/data/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
#在kubeconfig檔案中增加用戶資訊
[root@binary-k8s-node1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
2.指定生成的kubeconfig檔案為集群使用
[root@binary-k8s-node1 ~]# kubectl config use-context default --kubeconfig=/data/kubernetes/config/kube-proxy.kubeconfig
8.3.4.創建systemctl腳本管理服務
[root@binary-k8s-node1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/data/kubernetes/config/kube-proxy.conf
ExecStart=/data/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.targ
8.3.5.啟動kube-proxy組件
1.啟動服務
[root@binary-k8s-node1 ~]# systemctl daemon-reload
[root@binary-k8s-node1 ~]# systemctl start kube-proxy
[root@binary-k8s-node1 ~]# systemctl enable kube-proxy
2.查看埠
[root@binary-k8s-node1 ~]# netstat -lnpt | grep kube-proxy
tcp6 0 0 :::10249 :::* LISTEN 26954/kube-proxy
tcp6 0 0 :::10256 :::* LISTEN 26954/kube-proxy
8.4.快速增加新的node節點
二進制部署的程式特別好的一個地方就在于，能夠快速部署一個新的服務，做法就是直接拷貝已經部署好的目錄到一個新的位置，改改其中的引數即可啟動使用了，
8.4.1.將kubelet和kube-proxy目錄拷貝至新的node節點
要拷貝kubelet和kube-proxy部署目錄以及systemctl啟動腳本檔案，
[root@binary-k8s-node1 ~]# scp -rp /data/kubernetes root@binary-k8s-node2:/data
[root@binary-k8s-node1 ~]# scp /usr/lib/systemd/system/kube* root@binary-k8s-node2:/usr/lib/systemd/system/
8.4.2.配置并啟動kubelet組件
1.洗掉沒用的證書檔案
[root@binary-k8s-node2 ~]# rm -rf /data/kubernetes/ssl/kubelet-client-*
2.修改kubelet組態檔中的節點名稱
[root@binary-k8s-node2 ~]# vim /data/kubernetes/config/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--hostname-override=binary-k8s-node2 \
--network-plugin=cni \
--kubeconfig=/data/kubernetes/config/kubelet.kubeconfig \
--bootstrap-kubeconfig=/data/kubernetes/config/bootstrap.kubeconfig \
--config=/data/kubernetes/config/kubelet-config.yml \
--cert-dir=/data/kubernetes/ssl \
--pod-infra-container-image=pause-amd64:3.0"
#將--hostname-override值修改為當前節點名稱即可
3.啟動kubelet
[root@binary-k8s-node2 ~]# systemctl daemon-reload
[root@binary-k8s-node2 ~]# systemctl start kubelet
[root@binary-k8s-node2 ~]# systemctl enable kubelet
8.4.3.master節點授權新node節點的請求
1.master節點查看授權資訊串列
[root@binary-k8s-master1 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-u_AHUS7T5rku-hnhnGsGi8uGBqlgMquOq_3oq6jrOyE 48s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
2.授權通過node節點的kubelet
[root@binary-k8s-master1 ~]# kubectl certificate approve node-csr-u_AHUS7T5rku-hnhnGsGi8uGBqlgMquOq_3oq6jrOyE
certificatesigningrequest.certificates.k8s.io/node-csr-u_AHUS7T5rku-hnhnGsGi8uGBqlgMquOq_3oq6jrOyE approved
3.成功加入集群
[root@binary-k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
binary-k8s-master1 Ready <none> 2d23h v1.20.4
binary-k8s-node1 Ready <none> 5h54m v1.20.4
binary-k8s-node2 Ready <none> 1s v1.20.4
4.查看kubelet的埠
[root@binary-k8s-node2 ~]# netstat -lnpt | grep kube
tcp 0 0 127.0.0.1:41121 0.0.0.0:* LISTEN 16694/kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 16694/kubelet
tcp6 0 0 :::10250 :::* LISTEN 16694/kubelet
tcp6 0 0 :::10255 :::* LISTEN 16694/kubelet
8.4.4.配置并啟動kube-proxy組件
1.修改kube-proxy引陣列態檔中的主機名
[root@binary-k8s-node2 ~]# vim /data/kubernetes/config/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /data/kubernetes/config/kube-proxy.kubeconfig
hostnameOverride: binary-k8s-node2
clusterCIDR: 10.244.0.0/16
2.啟動kubelet
[root@binary-k8s-node2 ~]# systemctl daemon-reload
[root@binary-k8s-node2 ~]# systemctl start kube-proxy
[root@binary-k8s-node2 ~]# systemctl enable kube-proxy
3查看kube-proxy埠
[root@binary-k8s-node2 ~]# netstat -lnpt | grep kube
tcp 0 0 127.0.0.1:41121 0.0.0.0:* LISTEN 16694/kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 16694/kubelet
tcp6 0 0 :::10249 :::* LISTEN 20410/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 16694/kubelet
tcp6 0 0 :::10255 :::* LISTEN 16694/kubelet
tcp6 0 0 :::10256 :::* LISTEN 20410/kube-proxy
9.為集群部署coredns組件
9.1.部署coredns組件
1.coredns.yaml檔案內容
[root@binary-k8s-master1 ~]# cat coredns.yaml
# Warning: This is a file generated from the base underscore template file: coredns.yaml.base
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
log
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. In order to make Addon Manager do not reconcile this replicas parameter.
# 2. Default is 1.
# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
kubernetes.io/os: linux
containers:
- name: coredns
image: coredns:1.6.7
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.0.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP
2.部署coredns
[root@binary-k8s-master1 ~]# kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
9.2.運行一個busybox容器測驗dns
[root@binary-k8s-master1 ~]# kubectl run -it --rm dns-test --image=busybox:1.28.4 sh
If you don't see a command prompt, try pressing enter.
/ # nslookup kubernetes
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
Name: kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
/ # nslookup kube-dns.kube-system
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
/ # exit
10.擴容master節點組建kubernetes高可用集群
10.1.kubernetes高可用架構概念
kubernetes集群通過健康檢查和重啟策略實作了Pod故障自愈能力，也通過調度演算法實作將Pod分布式部署，并可以通過設定Pod的副本數，實作高并發能力，即使Node節點出現故障，Master節點也會將故障的Node節點上的Pod遷移到正常作業的Node節點上，實作應用層的高可用性
針對Kubernetes集群，高可用性包括Etcd資料庫高可用、Matser節點組件的高可用，Etcd可以通過集群方式實作高可用，而只有單臺Master節點，一旦Master節點上的組件出現了故障，整個集群將會不可用，
Master節點是屬于控制整個集群的角色，所有的組件都需要與Master節點的ApiServer進行互動，不斷與Node節點上的Kubelet和Kube-Proxy進行通信來維護整個集群的作業狀態，如果ApiServer發生故障，將無法與Node節點進行通信，也就無法管理集群，
因此Kubernetes集群最主要的就是對Master節點進行高可用配置，
Master節點主要有三個服務：kube-apiserver、kube-controller-manage、kube-scheduler，當集群有多臺Master節點時，其中kube-controller-manage和kube-scheduler都可以通過自身的選舉機制實作高可用，但是kube-apiserver就沒有這種機制，因此主要針對kube-apiserver配置高可用即可，kube-apiserver提供的是HTTP API介面服務，因此可以像web服務那種，使用nginx+keepalived方式實作Master節點高可用，并且也可以水平擴容，
配置kubernetes集群高可用的主要步驟就是：
? 1、增加一臺或多臺Master節點，部署Master節點相關組件，在這個master節點上配置的監聽地址還是自身的地址；
? 2、在新增的Master節點上部署etcd，使etcd加入現有的etcd集群，使etcd的性能更強；
? 3、配置nginx+keepalived實作Apiserver組件高可用；
? 4、配置所有的Node節點，將所配置的Apiserver地址改成keepalived虛擬出來的VIP地址，實作集群高可用；
高可用kubernetes集群一般3臺master節點足矣，但是etcd資料庫一定要多多益善
10.2.在集群中新增一個etcd節點
擴容etcd步驟：
? 1、部署一臺單節點的etcd，能夠正常啟動服務
? 2、在現有etcd集群中增加新的etcd節點
? 3、將單點的etcd配置成集群模式
? 4、洗掉單點造成的資料檔案
? 5、所有節點修改組態檔增加新的etcd節點資訊
? 6、重啟所有etcd節點
10.2.1.首先新增加一臺單點的etcd
1.安裝etcd程式
[root@binary-k8s-master2 ~]# tar xf etcd-v3.4.9-linux-amd64.tar.gz
[root@binary-k8s-master2 ~]# mkdir /data/etcd/{bin,conf,ssl,data} -p
[root@binary-k8s-master2 ~]# mv etcd-v3.4.9-linux-amd64/etcd* /data/etcd/bin/
2.創建單點組態檔
[root@binary-k8s-master2 ~]# vim /data/etcd/conf/etcd.conf
#[Service]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.20.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.11:2379,http://127.0.0.1:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.11:2379,http://127.0.0.1:2379"
ETCD_INITIAL_CLUSTER="etcd-4=https://192.168.20.11:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
4.拷貝證書檔案
[root@binary-k8s-master2 ~]# scp root@192.168.20.10:/data/etcd/ssl/* /data/etcd/ssl/
5.拷貝systemctl管理腳本
[root@binary-k8s-master2 ~]# scp root@192.168.20.10:/usr/lib/systemd/system/etcd.service /usr/lib/systemd/system/
6.啟動etcd服務
[root@binary-k8s-master2 ~]# systemctl daemon-reload
[root@binary-k8s-master2 ~]# systemctl start etcd
7.查看埠
[root@binary-k8s-master2 ~]# netstat -lnpt | grep etcd
tcp 0 0 192.168.20.11:2379 0.0.0.0:* LISTEN 15753/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 15753/etcd
tcp 0 0 192.168.20.11:2380 0.0.0.0:* LISTEN 15753/etcd
8.查看節點狀態
[root@binary-k8s-master2 ~]# /data/etcd/bin/etcdctl endpoint health --write-out=table
+----------------+--------+------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+----------------+--------+------------+-------+
| 127.0.0.1:2379 | true | 7.146222ms | |
+----------------+--------+------------+-------+
10.2.2.在現有etcd集群任意一個節點上增加新etcd節點
增加節點的命令為：
/data/etcd/bin/etcdctl member add 節點名稱 --peer-urls="通信地址"
1.增加etcd-4節點
[root@binary-k8s-master1 ~]# /data/etcd/bin/etcdctl member add etcd-4 --peer-urls="https://192.168.20.11:2380"
Member aae107adddd0d3d8 added to cluster 20b119eb5f91aa4b
ETCD_NAME="etcd-4"
ETCD_INITIAL_CLUSTER="etcd-2=https://192.168.20.12:2380,etcd-1=https://192.168.20.10:2380,etcd-3=https://192.168.20.13:2380,etcd-4=https://192.168.20.11:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
#輸出的配置資訊一定要在新的etcd-4節點的組態檔寫入，否則會加入集群失敗
2.查看集群節點串列
[root@binary-k8s-master1 ~]# /data/etcd/bin/etcdctl member list --write-out=table
+------------------+-----------+--------+----------------------------+--------------------------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+-----------+--------+----------------------------+--------------------------------------------------+------------+
| 12446003b2a53d43 | started | etcd-2 | https://192.168.20.12:2380 | http://127.0.0.1:2379,https://192.168.20.12:2379 | false |
| 51ae3f86f3783687 | started | etcd-1 | https://192.168.20.10:2380 | http://127.0.0.1:2379,https://192.168.20.10:2379 | false |
| 667c9c7ba890c3f7 | started | etcd-3 | https://192.168.20.13:2380 | http://127.0.0.1:2379,https://192.168.20.13:2379 | false |
| aae107adddd0d3d8 | unstarted | | https://192.168.20.11:2380 | | false |
+------------------+-----------+--------+----------------------------+--------------------------------------------------+------------+
#發現剛剛新加入的etcd-4節點處于unstarted狀態，我們需要再配置etcd-4節點使用能夠加入集群
10.2.3.配置新增的etcd節點加入集群
在已有集群增加完新節點之后，還需要將新的etcd節點組態檔增加集群相關屬性，然后洗掉由單點時造成的etcd資料檔案，最后在所有節點的組態檔中增加新節點的通信地址，重啟所有節點的etcd服務，到此擴容成功，
主要在新的etcd節點中配置ETCD_NAME、ETCD_INITIAL_CLUSTER、ETCD_INITIAL_CLUSTER_TOKEN、ETCD_INITIAL_CLUSTER_STATE這三個引數，
ETCD_NAME：集群節點名稱
ETCD_INITIAL_CLUSTER：由單點的一個節點資訊改成集群所有節點的資訊
ETCD_INITIAL_CLUSTER_TOKEN：填寫集群的唯一標識，表示加入哪個etcd集群
ETCD_INITIAL_CLUSTER_STATE：集群狀態調整為加入已存在的集群
1.修改etcd組態檔，增加集群配置引數
[root@binary-k8s-master2 ~]# vim /data/etcd/conf/etcd.conf
#[Service]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.20.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.11:2379,http://127.0.0.1:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.11:2379,http://127.0.0.1:2379"
ETCD_INITIAL_CLUSTER="etcd-2=https://192.168.20.12:2380,etcd-1=https://192.168.20.10:2380,etcd-3=https://192.168.20.13:2380,etcd-4=https://192.168.20.11:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"
2.洗掉由單點時產生的資料檔案
#如果不洗掉，加入集群時會失敗
[root@binary-k8s-master2 ~]# rm -rf /data/etcd/data/*
3.所有etcd的組態檔中增加新節點的通信地址
注意：所有etcd節點的組態檔都要增加這一行配置
vim /data/etcd/conf/etcd.conf
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.10:2380,etcd-2=https://192.168.20.12:2380,etcd-3=https://192.168.20.13:2380,etcd-4=https://192.168.20.11:2380"
4.重啟所有節點的ectd服務
[root@binary-k8s-master1 ~]# systemctl restart etcd
[root@binary-k8s-master2 ~]# systemctl restart etcd
[root@binary-k8s-node1 ~]# systemctl restart etcd
[root@binary-k8s-node2 ~]# systemctl restart etcd
5.再次查看集群的節點資訊
[root@binary-k8s-master1 ~]# /data/etcd/bin/etcdctl member list --write-out=table
+------------------+---------+--------+----------------------------+--------------------------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+--------+----------------------------+--------------------------------------------------+------------+
| 12446003b2a53d43 | started | etcd-2 | https://192.168.20.12:2380 | http://127.0.0.1:2379,https://192.168.20.12:2379 | false |
| 51ae3f86f3783687 | started | etcd-1 | https://192.168.20.10:2380 | http://127.0.0.1:2379,https://192.168.20.10:2379 | false |
| 667c9c7ba890c3f7 | started | etcd-3 | https://192.168.20.13:2380 | http://127.0.0.1:2379,https://192.168.20.13:2379 | false |
| aae107adddd0d3d8 | started | etcd-4 | https://192.168.20.11:2380 | http://127.0.0.1:2379,https://192.168.20.11:2379 | false |
+------------------+---------+--------+----------------------------+--------------------------------------------------+------------+
#etcd到此擴容成功
10.2.4.配置kube-apiserver增加新的etcd節點
etcd節點新增完，需要配置下kube-apiserver組件，增加新的etcd節點資訊，
注意所有k8s master節點都必須修改配置kube-apiserver.conf檔案增加新的etcd節點，否則etcd也不會為k8s所用，
1.master節點修改組態檔增加新的etcd節點
[root@binary-k8s-master1 ~]# vim /data/kubernetes/config/kube-apiserver.conf
······
--etcd-servers=https://192.168.20.10:2379,https://192.168.20.12:2379,https://192.168.20.13:2379,https://192.168.20.11:2379 \
······
2.重啟apiserver組件
[root@binary-k8s-master1 ~]# systemctl restart kube-apiserver
3.查看組件資訊
[root@binary-k8s-master1 ~]# kubectl get cs -o wide
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-3 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
10.3.部署master-2節點
由于所有組件都是二進制方式部署的，因此可以在master1上將目錄直接拷貝至master2上即可使用，
10.3.1.部署docker
1.安裝docker
[root@binary-k8s-master2 ~]# tar xf docker-19.03.9.tgz
[root@binary-k8s-master2 ~]# cp docker/* /usr/bin/
2.拷貝master1節點上的docker組態檔
[root@binary-k8s-master2 ~]# scp -rp root@binary-k8s-master1:/etc/docker /etc/
3.拷貝master1節點上的docker systemctl腳本
[root@binary-k8s-master2 ~]# scp -rp root@binary-k8s-master1:/usr/lib/systemd/system/docker.service /usr/lib/systemd/system/
4.啟動docker
[root@binary-k8s-master2 ~]# systemctl daemon-reload
[root@binary-k8s-master2 ~]# systemctl start docker
[root@binary-k8s-master2 ~]# systemctl enable docker
10.3.2.部署kubernetes各個組件
由于是二進制部署，直接拷貝master1節點上的/data/kubernetes目錄即可，/data/kubernetes目錄下包含了所有的master以及node相關組件
master節點需要安裝所有的master組件和node組件，
1.準備二進制程式
1.拷貝組件檔案
[root@binary-k8s-master1 ~]# scp -rp /data/kubernetes root@binary-k8s-master2:/data
[root@binary-k8s-master1 ~]# scp /usr/bin/kubectl root@binary-k8s-master2:/usr/bin
[root@binary-k8s-master1 ~]# scp -rp /usr/lib/systemd/system/kube* root@binary-k8s-master2:/usr/lib/systemd/system/
[root@binary-k8s-master1 ~]# scp -rp .kube root@binary-k8s-master2:/root
2.如果沒有擴容新的etcd節點的情況需要拷貝etcd證書
[root@binary-k8s-master1 ~]# scp -rp /data/etcd/ssl root@binary-k8s-master2:/data/etcd/ss
3.洗掉kubelet檔案
#kubelet某些問題都是動態生成的，且每個節點都不相同，因此需要洗掉重新生成
[root@binary-k8s-master2 ~]# rm -rf /data/kubernetes/config/kubelet.kubeconfig
[root@binary-k8s-master2 ~]# rm -rf /data/kubernetes/ssl/kubelet-client-*
2.修改各個組件的組態檔
主要就是修改各個組件監聽的本機ip地址和節點名稱，生成的kubeconfig檔案中的apiserver地址無需更改，保持master1即可，因為最后高可用的時候還是會改成VIP地址，當前無需更改，
1.修改kube-apiserver組態檔中的IP地址
[root@binary-k8s-master2 ~]# vim /data/kubernetes/config/kube-apiserver.conf
······
--bind-address=192.168.20.11 \
--advertise-address=192.168.20.11 \
······
2.修改kube-controller-manager組態檔中的IP地址
[root@binary-k8s-master2 ~]# vim /data/kubernetes/config/kube-controller-manager.conf
······
--bind-address=192.168.20.11 \
······
3.修改kube-scheduler組態檔中的IP地址
[root@binary-k8s-master2 ~]# vim /data/kubernetes/config/kube-scheduler.conf
······
--bind-address=192.168.20.11"
······
4.修改kubelet組態檔中的IP地址
[root@binary-k8s-master2 ~]# vim /data/kubernetes/config/kubelet.conf
······
--hostname-override=binary-k8s-master2 \
······
5.修改kube-apiserver組態檔中的IP地址
[root@binary-k8s-master2 ~]# vim /data/kubernetes/config/kube-proxy-config.yml
······
hostnameOverride: binary-k8s-master2
······
3.啟動各個組件
[root@binary-k8s-master2 ~]# systemctl daemon-reload
[root@binary-k8s-master2 ~]# systemctl start kube-apiserver
[root@binary-k8s-master2 ~]# systemctl start kube-controller-manager
[root@binary-k8s-master2 ~]# systemctl start kube-scheduler
[root@binary-k8s-master2 ~]# systemctl start kubelet
[root@binary-k8s-master2 ~]# systemctl start kube-proxy
[root@binary-k8s-master2 ~]# systemctl enable kube-apiserver
[root@binary-k8s-master2 ~]# systemctl enable kube-controller-manager
[root@binary-k8s-master2 ~]# systemctl enable kube-scheduler
[root@binary-k8s-master2 ~]# systemctl enable kubelet
[root@binary-k8s-master2 ~]# systemctl enable kube-proxy
10.3.3.授權master2節點加入集群
1.查看授權新系串列
[root@binary-k8s-master2 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-fgCu0hUU4sK9-jaLzl8n-H4MVWi314NhzYssddgThOE 4m45s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
2.授權通過
[root@binary-k8s-master2 ~]# kubectl certificate approve node-csr-fgCu0hUU4sK9-jaLzl8n-H4MVWi314NhzYssddgThOE
certificatesigningrequest.certificates.k8s.io/node-csr-fgCu0hUU4sK9-jaLzl8n-H4MVWi314NhzYssddgThOE approved
3.查看是否加入集群
[root@binary-k8s-master2 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
binary-k8s-master1 Ready <none> 4d21h v1.20.4
binary-k8s-master2 Ready <none> 4m33s v1.20.4
binary-k8s-node1 Ready <none> 2d3h v1.20.4
binary-k8s-node2 Ready <none> 45h v1.20.4
4.查看核心組件狀態
[root@binary-k8s-master2 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
10.4.部署Nginx+Keepalived實作kubernetes高可用集群
keepalived是主流的高可用軟體，基于VIP系結實作服務器的雙機熱備，可以理解為keepalived是針對服務器IP的高可用集群，如果A機器當機了，B機器會立刻成為master角色，搶占VIP地址，使其不間斷的提供服務，從而形成高可用集群，
使用nginx+keepalived做得k8s master節點高可用集群，只要master節點上面沒有etcd組件，那么整個集群master節點只要有一個作業正常，整個集群就不會宕機，
生產環境中nginx+keepalived是獨立于集群之外的兩臺服務器，高可用集群一般情況下都是一主一備，兩個節點就可以滿足正常需求，正好master節點有2個，可以在兩個master上都部署nginx和keepalived形成高可用集群，
我們采用nginx四層負載均衡，四層負載均衡的作用就是對IP進行負載，不涉及應用層，由于我們使用keepalived做高可用集群，keepalived就是針對IP地址實作高可用，因此需要配合nginx四層負載均衡來實作，當用戶訪問keepalived的VIP時，直接將請求轉發到對應的master角色主機上，將VIP地址轉換成master節點IP+埠，這樣一來，即使master1掛掉了，master2成為了master角色，請求轉發進來，也會將VIP轉換成master2節點的地址，高可用也就實作了，
kube-apiserver高可用架構圖
10.4.1.部署Nginx負載均衡
master1和master2上的nginx部署和組態檔內容一樣，這里只寫master1的操作步驟，
nginx負載均衡采用四層負載，
1.安裝nginx和keepalived及nginx四層負載均衡模塊等軟體
[root@binary-k8s-master1 ~]# yum -y install nginx keepalived nginx-mod-stream
2.修改nginx主組態檔增加include模塊引入4層負載組態檔
[root@binary-k8s-master1 ~]# vim /etc/nginx/nginx.conf
include /etc/nginx/conf.c/*.conf; #17行左右，與http模塊同級
3.撰寫組態檔
[root@binary-k8s-master1 ~]# mkdir /etc/nginx/conf.c
[root@binary-k8s-master1 ~]# vim /etc/nginx/conf.c/k8s-apiserver.conf
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-apiserver.log main;
upstream k8s-apiserver {
server 192.168.20.11:6443;
server 192.168.20.12:6443;
}
server {
listen 16443; #由于我們的nginx與k8s master在同一臺機器上，防止埠沖突，因此改為16443埠
proxy_pass k8s-apiserver;
}
}
4.啟動nginx
[root@binary-k8s-master1 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@binary-k8s-master1 ~]# nginx
5.查看埠
[root@binary-k8s-master1 ~]# netstat -lnpt | grep 16443
tcp 0 0 0.0.0.0:16443 0.0.0.0:* LISTEN 3181/nginx: worker
10.4.2.部署keepalived雙機熱備
在配置keepalived的時候也需要配置一個vrrp_script模塊，keepalived只能做到對網路故障和keepalived本身的監控，即當出現網路故障或者keepalived本身出現問題時，進行切換，但是這些還不夠，我們還需要監控keepalived所在服務器上的其他業務行程，比如說nginx，keepalived+nginx實作nginx的負載均衡高可用，如果nginx例外，僅僅keepalived保持正常，是無法完成系統的正常作業的，因此需要根據業務行程的運行狀態決定是否需要進行主備切換，這個時候，我們可以通過撰寫腳本對nginx行程進行檢測監控，
1.MASTER節點部署
1.安裝keepalived
[root@binary-k8s-master1 ~]# yum -y install keepalived
2.配置keepalived
[root@binary-k8s-master1 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx { #定義健康檢查腳本
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER #狀態為MASTER
interface ens192 #將VIP系結在哪塊網卡上
virtual_router_id 51 #實體ID，集群所有節點都要保持一致
priority 100 #優先級，255最高
advert_int 1 #指定VRRP心跳包通告間隔時間，默認1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.20.9/23 #定義VIP地址
}
track_script {
check_nginx
}
}
3.撰寫檢查nginx狀態的檢查腳本
#當nginx例外時，自動將當前主機的keepalived行程關閉，使BACKUP上的keepalived成為MASTER繼續提供服務
[root@binary-k8s-master1 ~]# vim /etc/keepalived/check_nginx.sh
nginx_ch=`netstat -lnpt | grep 16443| egrep -cv grep`
if [ $nginx_ch -eq 0 ];then
systemctl stop keepalived
exit 1
else
exit 0
fi
4.啟動keepalived
[root@binary-k8s-master1 ~]# systemctl start keepalived
[root@binary-k8s-master1 ~]# systemctl enable keepalived
5.查看VIP地址
[root@binary-k8s-master1 ~]# ip a | grep ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 192.168.20.10/23 brd 192.168.21.255 scope global noprefixroute ens192
inet 192.168.20.9/23 scope global secondary ens192
#VIP已經準備就緒
2.BACKUP節點部署
1.安裝keepalived
[root@binary-k8s-master2 ~]# yum -y install keepalived
2.配置keepalived
[root@binary-k8s-master2 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP #狀態為BACKUP
interface ens192
virtual_router_id 51
priority 90 #優先級要比MASTER低
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.20.9/23
}
track_script {
check_nginx
}
}
3.撰寫檢查nginx狀態的檢查腳本
#當nginx例外時，自動將當前主機的keepalived行程關閉，使BACKUP上的keepalived成為MASTER繼續提供服務
[root@binary-k8s-master1 ~]# vim /etc/keepalived/check_nginx.sh
nginx_ch=`netstat -lnpt | grep 16443| egrep -cv grep`
if [ $nginx_ch -eq 0 ];then
systemctl stop keepalived
exit 1
else
exit 0
fi
4.啟動keepalived
[root@binary-k8s-master1 ~]# systemctl start keepalived
[root@binary-k8s-master1 ~]# systemctl enable keepalived
10.4.3.使用VIP訪問kubernetes服務
可以正確獲取到K8s版本資訊，說明負載均衡器搭建正常，該請求資料流程：curl -> vip(nginx) -> apiserver，
日志中也會記錄訪問記錄，
[root@binary-k8s-master1 ~]# curl -k https://192.168.20.9:16443/version
{
"major": "1",
"minor": "20",
"gitVersion": "v1.20.4",
"gitCommit": "e87da0bd6e03ec3fea7933c4b5263d151aafd07c",
"gitTreeState": "clean",
"buildDate": "2021-02-18T16:03:00Z",
"goVersion": "go1.15.8",
"compiler": "gc",
"platform": "linux/amd64"
}
[root@binary-k8s-master1 ~]# tail -f /var/log/nginx/k8s-apiserver.log
127.0.0.1 192.168.20.11:6443 - [09/Sep/2021:11:28:15 +0800] 200 79
127.0.0.1 192.168.20.11:6443 - [09/Sep/2021:11:28:20 +0800] 200 178
192.168.20.10 192.168.20.11:6443 - [09/Sep/2021:15:20:29 +0800] 200 178
192.168.20.10 192.168.20.12:6443, 192.168.20.11:6443 - [09/Sep/2021:16:19:00 +0800] 200 0, 420
10.4.4.測驗keepalived高可用
1.停掉master1上的keepalived，查看VIP是否會切換到master2節點
2.重新啟動master1上的keepalived，查看VIP是否會自動切換到master1
10.5.切換kubernetes集群為高可用模式
雖然我們增加了Master2 Node和負載均衡器，但是我們是從單Master架構擴容的，也就是說目前所有的Worker Node組件連接都還是Master1 Node，如果不改為連接VIP走負載均衡器，那么Master還是單點故障，
由于已經可以通過keepalived的VIP地址訪問到apiserver，高可用效果已達成，目前只需要將集群的所有節點（kubectl get node）能看到的一切節點，將組態檔中的apiserver的地址換成VIP地址加埠，才能真正的實作kubernetes高可用，
之前前期使用VIP測驗kube-apiserver沒問題，即使在切換高可用的情況下，所有節點也不會處于NotReady狀態，
1.切換高可用環境
1.binary-k8s-master1節點切換
[root@binary-k8s-master1 ~]# sed -ri 's#192.168.20.10:6443#192.168.20.9:16443#' /data/kubernetes/config/*
[root@binary-k8s-master1 ~]# sed -ri 's#192.168.20.10:6443#192.168.20.9:16443#' /root/.kube/config
[root@binary-k8s-master1 ~]# systemctl restart kube-controller-manager kube-scheduler kubelet kube-proxy
2.binary-k8s-master2切換
[root@binary-k8s-master2 ~]# sed -ri 's#192.168.20.10:6443#192.168.20.9:16443#' /data/kubernetes/config/*
[root@binary-k8s-master2 ~]# sed -ri 's#192.168.20.10:6443#192.168.20.9:16443#' /root/.kube/config
[root@binary-k8s-master2 ~]# systemctl restart kube-controller-manager kube-scheduler kubelet kube-proxy
3.binary-k8s-node1切換
[root@binary-k8s-node1 ~]# sed -ri 's#192.168.20.10:6443#192.168.20.9:16443#' /data/kubernetes/config/*
[root@binary-k8s-node1 ~]# systemctl restart kubelet kube-proxy
4.binary-k8s-node2切換
[root@binary-k8s-node2 ~]# sed -ri 's#192.168.20.10:6443#192.168.20.9:16443#' /data/kubernetes/config/*
[root@binary-k8s-node2 ~]# systemctl restart kubelet kube-proxy
2.查看集群狀態及資源
到此為止kubernetes高可用集群實作完畢
[root@binary-k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
binary-k8s-master1 Ready <none> 5d22h v1.20.4
binary-k8s-master2 Ready <none> 25h v1.20.4
binary-k8s-node1 Ready <none> 3d5h v1.20.4
binary-k8s-node2 Ready <none> 2d23h v1.20.4
[root@binary-k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
11.測驗kubernetes高可用集群
1.停掉master1上的keepalived驗證集群是否可用
狀態：“ok”
2.停掉master1上所有k8s組件驗證集群是否可用
狀態：“ok”
12.在kubernetes集群運行一套服務驗證集群的可用性
簡單部署一個基于nginx的web服務，
12.1.創建資源yaml檔案
[root@binary-k8s-master1 ~]# vim know-system.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-know-system
spec:
replicas: 3
selector:
matchLabels:
app: know-system-pod
template:
metadata:
labels:
app: know-system-pod
spec:
containers:
- name: know-system
image: know-system:v1
ports:
- containerPort: 80
nodeName: binary-k8s-master1
---
apiVersion: v1
kind: Service
metadata:
name: know-system-service
spec:
selector:
app: know-system-pod
type: NodePort
ports:
- port: 80
targetPort: 80
12.2.創建資源并進行測驗
[root@binary-k8s-master1 ~]# kubectl apply -f know-system.yaml
deployment.apps/deploy-know-system created
service/know-system-service created
[root@binary-k8s-master1 ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/deploy-know-system-b4c9c55d7-5mf2f 1/1 Running 0 47s
pod/deploy-know-system-b4c9c55d7-97ckx 1/1 Running 0 48s
pod/deploy-know-system-b4c9c55d7-kb97t 1/1 Running 0 47s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/know-system-service NodePort 10.0.0.38 <none> 80:32702/TCP 47s
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 10d
訪問https://集群任意節點+32702埠即可瀏覽web服務，
13.部署kubernetes dashboard
13.1.部署dashboard
1.部署yaml
[root@binary-k8s-master1 ~]# kubectl apply -f kubernetes-dashboard.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
2.創建授權賬號
[root@binary-k8s-master1 ~]# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created
[root@binary-k8s-master1 ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created
3.查看登陸使用的token字串
[root@binary-k8s-master1 ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name: dashboard-admin-token-lnm2r
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: dashboard-admin
kubernetes.io/service-account.uid: 73b370c9-b1b4-4418-b02d-fee9b6cf6342
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkgwWGJXQ1duVVI4eFh4Ykw2U25JVk9fa2hDOGZVRTRRMVZyVmdwWXM1Nk0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tbG5tMnIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNzNiMzcwYzktYjFiNC00NDE4LWIwMmQtZmVlOWI2Y2Y2MzQyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.XWJLZDB7mNk_NNpXVv64LrbKy5f1hB2PS5qER5YFATzl3U9ISX05PCrnCEY-6uVSbPkRGZbTQZTBwiGjOfsyLZljvY3cbmGlH2oW2shUS8LDqli4MKA14JyUX1ubbQ8vq9uSqkQMCQBzZTUGIuZt95jw3-IMv2rfZ9ET8_uVuXIoZXbckY6VHFy8QOB6sy1n9j0j4qcOttyKHVXN8Q5KjsIlb44Y5HtiveKxpw_LA81eTwml_aiVvO9rgMKVdSHIg8CY1Mcp06ezz0kD0jsBLt7xaAujSNZnCiXzmpg51xujbR0k-4BVlwPBBpQLaSWGoHR3X7z5E02onXttbbX6-w
ca.crt: 1359 bytes
namespace: 11 bytes
4.查看pod的狀態
[root@binary-k8s-master1 ~]# kubectl get pod,svc -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-7445d59dfd-bg9c8 1/1 Running 0 8m51s
pod/kubernetes-dashboard-5ddcdf9c99-nkgqw 1/1 Running 0 8m52s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.0.0.83 <none> 8000/TCP 8m52s
service/kubernetes-dashboard NodePort 10.0.0.153 <none> 443:30001/TCP 8m53s
13.2.訪問dashboard
訪問https://集群任意節點+30001埠，然后填寫剛剛查到的token值，點擊登陸，
儀表盤
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/302555.html
標籤：其他