實驗環境

主機名       IP地址       安裝軟體
Master01：14.0.0.50    kube-apiserver kube-controller-manager kube-scheduler etcd
Node01：  14.0.0.60    kubelet kube-proxy docker flannel etcd
Node02：  14.0.0.70    kubelet kube-proxy docker flannel etcd

實驗程序

1.Etcd集群部署

#初始化環境，準備制作證書

[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls      #從宿主機上傳進來
etcd-cert.sh etcd.sh
[root@localhost k8s]# mkdir etcd-cert
[root@localhost k8s]# mv etcd-cert.sh etcd-cert/

1.下載證書制作工具

[root@localhost k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

2.執行腳本下載 cfssl官方包

[root@localhost k8s]# bash cfssl.sh
[root@localhost k8s]# ls /usr/local/bin/
cfssl cfssl-certinfo cfssljson
#cfssl：生成證書工具 
#cfssljson：通過傳入json檔案生成證書
#cfssl-certinfo：查看證書資訊

3.開始制作證書

（1）創建ca組態檔
cat > ca-config.json <<EOF
{
  "signing": {
     "default": {
        "expiry": "87600h"
    },
    "profiles": {
    "www": {
        "expiry": "87600h",
       "usages": [
           "signing",
           "key encipherment",
           "server auth",
           "client auth"
        ]
     }
   }
  }
}
EOF
（2）創建ca證書簽名請求
cat > ca-csr.json <<EOF
{
     "CN": "etcd CA",
     "key": {
           "algo": "rsa",
           "size": 2048
 },
 "names": [
       {
       "C": "CN",
       "L": "Beijing",
       "ST": "Beijing"
         }
     ]
}
EOF

（3）生成證書
ca-key.pem：根證書的私鑰
ca.pem：ca根證書檔案
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
（4）指定 etcd三個節點之間的通信驗證
cat > server-csr.json <<EOF
{
      "CN": "etcd",
      "hosts": [
      "14.0.0.50",
     "14.0.0.60",
     "14.0.0.70"
     ],
     "key": {
     "algo": "rsa",
     "size": 2048
     },
  "names": [
         {
           "C": "CN",
           "L": "BeiJing",
           "ST": "BeiJing"
       }
    ]
}
EOF
（5）生成 ETCD證書 server-key.pem server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

#證書生成完畢，開始配置etcd軟體

#ETCD 二進制包地址
https://github.com/etcd-io/etcd/releases
將etcd-v3.3.10-linux-amd64.tar.gz上傳到/root/k8s目錄下
[root@localhost k8s]# ls
cfssl.sh etcd.sh
etcd-cert etcd-v3.3.10-linux-amd64.tar.gz
（1）將etcd軟體包進行解壓
[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz
[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
（2）創建etcd專門的組態檔，命令檔案和證書存放目錄
[root@localhost k8s]# mkdir -p /opt/etcd/{cfg,bin,ssl} 
（3）將etcd軟體包中的命令檔案移動到/opt/etcd/bin目錄下
[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
（4）拷貝證書到/opt/etcd/ssl目錄下
[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/
#啟動etcd.sh腳本生成組態檔和服務啟動腳本；并啟動etcd服務
[root@localhost k8s]# bash etcd.sh etcd01 14.0.0.50 etcd02=https://14.0.0.60:2380,etcd03=https://14.0.0.70:2380 #這時會等待其他節點加入

#使用另外一個會話打開，發現 etcd行程已經開啟
[root@localhost ~]# ps -ef | grep etcd

#使用scp拷貝組態檔，命令檔案，證書檔案到兩個node節點
[root@localhost k8s]# scp -r /opt/etcd/ root@14.0.0.60:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ root@14.0.0.70:/opt
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@14.0.0.60:/usr/lib/systemd/system/
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@14.0.0.70:/usr/lib/systemd/system/

#進入node01節點修改（node02節點同理）
[root@localhost ~]# vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://14.0.0.60:2380"   #群集內部通信埠
ETCD_LISTEN_CLIENT_URLS="https://14.0.0.60:2379"   #對外提供服務埠
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://14.0.0.60:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://14.0.0.60:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://14.0.0.50:2380,etcd02=https://14.0.0.60:2380,etcd03=https://14.0.0.70:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#啟動etcd
[root@localhost ssl]# systemctl start etcd
[root@localhost ssl]# systemctl status etcd

#使用etcdctl命令檢查群集健康狀態
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" cluster-health
member 3eae9a550e2e3ec is healthy: got healthy result from https://192.168.195.151:2379
member 26cd4dcf17bc5cbd is healthy: got healthy result from https://192.168.195.150:2379
member 2fcd2df8a9411750 is healthy: got healthy result from https://192.168.195.149:2379
cluster is healthy     #說明群集健康

2.docker引擎部署

所有node節點部署docker-ce引擎
部署docker-ce社區版可參考我之前的博客：
https://blog.csdn.net/chengu04/article/details/108723407

3.flannel網路部署

所有node節點部署flannel組件

（1）寫入分配的子網段到etcd中，供flannel使用（注意：要在/opt/etcd/ssl目錄下）
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

（2）在另一個節點查看寫入資訊
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" get /coreos.com/network/config
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
（3）在兩個node節點上部署flannel軟體
[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
flanneld
mk-docker-opts.sh
README.md
#創建k8s作業目錄，將軟體包中的命令檔案移動到/opt/kubernetes/bin/目錄下
[root@localhost ~]# mkdir -p /opt/kubernetes/{cfg,bin,ssl} 
[root@localhost ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
#通過腳本配置flannel
[root@localhost ~]# vim flannel.sh
#!/bin/bash
ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}
cat <<EOF >/opt/kubernetes/cfg/flanneld        #生成組態檔
FLANNEL_OPTIONS="-- etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat <<EOF >/usr/lib/systemd/system/flanneld.service          #生成服務啟動腳本
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld -- ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld


#開始flannel網路功能
[root@localhost ~]# bash flannel.sh https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
#修改docker的服務啟動檔案指定子網段
[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
[Service]
Type=notify
#the default is not to use systemd for cgroups because the delegate issues still
#exists and systemd currently does not support the cgroup feature set required
#for containers run by docker
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// -- containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@localhost ~]# cat /run/flannel/subnet.env       #查看flannel為該節點分配的子網段
DOCKER_OPT_BIP="--bip=172.17.42.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
##說明：bip指定啟動時的子網
DOCKER_NETWORK_OPTIONS=" --bip=172.17.42.1/24 -- ip-masq=false -- mtu=1450"
#重啟docker服務

[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart docker

#查看flannel網路
[root@localhost ~]# ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.42.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::fc7c:e1ff:fe1d:224 prefixlen 64 scopeid 0x20<link>
ether fe:7c:e1:1d:02:24 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0

#在兩個node節點中都運行一個容器，測驗能否ping通對方，能ping通則flannel部署成功
[root@localhost ~]# docker run -it centos:7 /bin/bash
[root@5f9a65565b53 /]# yum install net-tools -y
[root@5f9a65565b53 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
      inet 172.17.42.2 netmask 255.255.255.0 broadcast 172.17.84.255
      ether 02:42:ac:11:54:02 txqueuelen 0 (Ethernet)
 ....省略內容

4.部署master節點

#在 master上操作，api-server生成證書
上傳master.zip到/root/k8s目錄下
[root@localhost k8s]# unzip master.zip
[root@localhost k8s]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost k8s]# mkdir k8s-cert
[root@localhost k8s]# cd k8s-cert/
[root@localhost k8s-cert]# ls      #上傳到/root/k8s/k8s-cert/目錄下
k8s-cert.sh     #利用腳本生成k8s證書
#腳本內容如下
cat > ca-config.json <<EOF          #ca證書組態檔
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF      #ca證書簽名檔案
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
      	    "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -      
                                     #生成ca.pem和ca-key.pem（CA認證機構）
#-----------------------
cat > server-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "14.0.0.50",       #master01
      "14.0.0.80",       #master02
      "14.0.0.88",       #VIP
      "14.0.0.90",       #反向代理（master）
      "14.0.0.100",     #反向代理（backup）
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server                   
                   #生成kube-apiserver的tls認證證書和認證私鑰server.pem和server-key.pem
#-----------------------
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin          
     #kubectl的TLS認證證書和認證私鑰，具有admin權限，admin.pem和admin-key.pem     
#-----------------------
cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy   
                                       #生成kube-proxy-key.pem和kube-proxy.pem
#生成證書如下
[root@localhost k8s-cert]# ls *pem
admin-key.pem  ca-key.pem      kube-proxy-key.pem          server-key.pem
admin.pem        ca.pem              kube-proxy.pem                server.pem
#將ca證書和kube-apiserver證書復制到/opt/kubernetes/ssl/目錄下
[root@localhost k8s-cert]# cp ca*pem server*pem /opt/kubernetes/ssl/
[root@localhost k8s-cert]# cd ..

#解壓 kubernetes壓縮包
[root@localhost k8s]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@localhost k8s]# cd /root/k8s/kubernetes/server/bin
#復制軟體包中bin目錄下的關鍵命令檔案
[root@localhost bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
[root@localhost k8s]# cd /root/k8s

#使用 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 可以隨機生成序列號
[root@localhost k8s]# vim /opt/kubernetes/cfg/token.csv
0fb61c46f8991b718eb38d27b605b008,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
填寫內容：序列號， 用戶名，uid，用戶組
（該檔案為一個用戶的描述檔案，基本格式為 Token,用戶名,UID,用戶組；這個檔案在 apiserver 啟動時被 apiserver 加載，然后就相當于在集群內創建了一個這個用戶；接下來就可以用 RBAC 給他授權，）

#二進制檔案，token，證書都準備好，開啟 apiserver
[root@localhost k8s]# bash apiserver.sh 14.0.0.50 https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379
#檢查行程是否啟動成功
[root@localhost k8s]# ps aux | grep kube

#監聽的https埠
[root@localhost k8s]# netstat -ntap | grep 6443
tcp 0 0 192.168.195.149:6443 0.0.0.0:* LISTEN 46459/kube-apiserve
tcp 0 0 192.168.195.149:6443 192.168.195.149:36806 ESTABLISHED 46459/kube-apiserve
tcp 0 0 192.168.195.149:36806 192.168.195.149:6443 ESTABLISHED 46459/kube-apiserve

#監聽的http埠
[root@localhost k8s]# netstat -ntap | grep 8080
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN
46459/kube-apiserve

#啟動scheduler服務
[root@localhost k8s]# ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@localhost k8s]# ps aux | grep ku
[root@localhost k8s]# chmod +x controller-manager.sh

#啟動controller-manager服務
[root@localhost k8s]# ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

#查看master節點狀態
[root@localhost k8s]# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}

5.部署node節點

#把master01節點上的kubelet、kube-proxy拷貝到node節點上去

[root@localhost bin]# scp kubelet kube-proxy root@14.0.0.60:/opt/kubernetes/bin/
[root@localhost bin]# scp kubelet kube-proxy root@14.0.0.70:/opt/kubernetes/bin/

#nod01節點操作（上傳node.zip到/root目錄下）
[root@localhost ~]# ls
anaconda-ks.cfg flannel-v0.10.0-linux-amd64.tar.gz  node.zip  公共 視頻 檔案 音樂
flannel.sh initial-setup-ks.cfg README.md 模板 圖片 下載 桌 面
//解壓 node.zip，獲得 kubelet.sh；proxy.sh腳本
[root@localhost ~]# unzip node.zip

#在master上操作
[root@localhost k8s]# mkdir kubeconfig
[root@localhost k8s]# cd kubeconfig/
//將kubeconfig.sh檔案進行重命名
[root@localhost kubeconfig]# mv kubeconfig.sh kubeconfig
//獲取tokenID資訊
[root@localhost ~]# cat /opt/kubernetes/cfg/token.csv
6351d652249951f79c33acdab329e4c4,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
[root@localhost kubeconfig]# vim kubeconfig
#修改tokenID
kubectl config set-credentials kubelet-bootstrap \
-- token=6351d652249951f79c33acdab329e4c4 \
-- kubeconfig=bootstrap.kubeconfig
//設定環境變數 （可以寫入到 /etc/profile中）
[root@localhost kubeconfig]# echo export PATH=$PATH:/opt/kubernetes/bin/ >> /etc/profile
[root@localhost kubeconfig]# source /etc/profile
[root@localhost kubeconfig]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
//生成組態檔
[root@localhost kubeconfig]# bash kubeconfig 14.0.0.60 /root/k8s/k8s-cert/
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
[root@localhost kubeconfig]# ls
bootstrap.kubeconfig kubeconfig kube-proxy.kubeconfig

//拷貝組態檔到 拷貝組態檔到 node節點
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@14.0.0.60:/opt/kubernetes/cfg/
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@14.0.0.70:/opt/kubernetes/cfg/

//首次啟動時kubelet如何連接apiserver？
使用預設用戶kubelet-bootstrap，但是需要群集角色系結，將預設用戶 kubelet-bootstrap與內置的 ClusterRole system:node-bootstrapper 系結到一起，才能將權限用于連接 apiserver請求

[root@localhost kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

#在node01節點上操作
[root@localhost ~]# bash kubelet.sh 14.0.0.60
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

//檢查kubelet服務啟動
[root@localhost ~]# ps aux | grep kube
root 106845 1.4 1.1 371744 44780 ? Ssl 00:34 0:01 /opt/kubernetes/bin/kubelet -- logtostderr=true -- v=4 -- hostname-override=192.168.195.150 -- kubeconfig=/opt/kubernetes/cfg/kubelet.kubconfig -- bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig -- config=/opt/kubernetes/cfgkubelet.config -- cert-dir=/opt/kubernetes/ssl -- pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root 106876 0.0 0.0 112676 984 pts/0 S+ 00:35 0:00 grep -- color=auto kube
#master01上操作（手動簽發）
//檢查node01節點的請求
[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 4m27s kubelet-bootstrap Pending（等待集群給該節點頒發證書）

[root@localhost kubeconfig]# kubectl certificate approve
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A
certificatesigningrequest.certificates.k8s.io/node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A approved  

[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 8m56s kubelet-bootstrap Approved,Issued（Approved代表已經被允許加入群集）

//查看群集節點，成功加入 node01節點
[root@localhost kubeconfig]# kubectl get node
NAME STATUS ROLES AGE VERSION
14.0.0.50 Ready <none> 118s v1.12.3

#在 node01節點操作，啟動 proxy服務
[root@localhost ~]# bash proxy.sh 14.0.0.60
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@localhost ~]# systemctl status kube-proxy.service
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2020-02-02 00:47:29 CST; 11s ago
Main PID: 108006 (kube-proxy)
Memory: 7.5M
CGroup: /system.slice/kube-proxy.service
? 108006 /opt/kubernetes/bin/kube-proxy -- logtostderr=true -- v=4 -- hostname-override=1...

node02節點部署

#在node01節點操作
//把現成的 /opt/kubernetes目錄復制到其他節點進行修改即可
[root@localhost ~]# scp -r /opt/kubernetes/ root@14.0.0.70:/opt/
//把 kubelet，kube-proxy的 service檔案拷貝到 node2中
[root@localhost ~]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@14.0.0.70:/usr/lib/systemd/system/

#在node02節點上操作，進行修改
//首先洗掉復制過來的證書，等會 node02會自行申請證書
[root@localhost ~]# cd /opt/kubernetes/ssl/
[root@localhost ssl]# rm -rf *
//修改組態檔kubelet kubelet.config kube-proxy（三個組態檔）
[root@localhost ssl]# cd ../cfg/
[root@localhost cfg]# vim kubelet
KUBELET_OPTS="-- logtostderr=true \
-- v=4 \
-- hostname-override=14.0.0.70 \
-- kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
-- bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
-- config=/opt/kubernetes/cfg/kubelet.config \
-- cert-dir=/opt/kubernetes/ssl \
-- pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
[root@localhost cfg]# vim kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 14.0.0.70
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
~
[root@localhost cfg]# vim kube-proxy
KUBE_PROXY_OPTS="-- logtostderr=true \
-- v=4 \
-- hostname-override=14.0.0.70 \
-- cluster-cidr=10.0.0.0/24 \
-- proxy-mode=ipvs \
-- kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
//啟動服務
[root@localhost cfg]# systemctl start kubelet.service
[root@localhost cfg]# systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@localhost cfg]# systemctl start kube-proxy.service
[root@localhost cfg]# systemctl enable kube-proxy.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

//在master上操作查看請求
[root@localhost k8s]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU 15s kubelet-bootstrap Pending

//授權許可加入群集
[root@localhost k8s]# kubectl certificate approve node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU
certificatesigningrequest.certificates.k8s.io/node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU approved

//查看群集中的節點
[root@localhost k8s]# kubectl get node  #兩個node節點均顯示Ready代表都已成功加入
NAME STATUS ROLES AGE VERSION
14.0.0.60 Ready <none> 21h v1.12.3
14.0.0.70 Ready <none> 37s v1.12.3

實驗故障及如何處理

如果實驗最后查看群集中的節點時，發現有node處于noready狀態，如何處理？

1.先檢查網路，查看各個節點間網路是否互通；
2.網路沒問題再檢查kubelet，檢查kubeconfig中的三個配置（apiserver、tokenID、證書）是否正確；
3.使用公網上的主機訪問node中的業務，如果內外網都無法訪問服務器說明該node服務器可能已經宕機了，需要聯系現場運維工程師查看，

轉載請註明出處，本文鏈接：https://www.uj5u.com/qianduan/153946.html

標籤：其他

上一篇：云旗OS助手火了！可一站式體驗統信UOS

下一篇：用虛擬機搭建iaas先電私有云平臺

Kubernetes二進制部署（單節點）

文章目錄