引言
本文探討如何搭建k8s集群，
- k8s概念和架構
- 從零搭建k8s集群
- K8s核心概念
- 搭建集群監控平臺系統
- 從零搭建高可用k8s集群
搭建k8s環境平臺規劃
單master集群
單master集群，如果master掛了，就比較麻煩，
多master集群
多master如果掛了1個master，還有兩個master可以用，明顯可用性更強(高可用)，
部署k8s集群方式
目前主要有三種方式，
kubeadm
kubeadm是一個k8s部署工具，提供kubeadm init和kubeadm join，用于快速部署k8s集群，
如何安裝？ 可查看安裝指南，
二進制包
從github下載發行版的二進制包，手動部署每個組件，組成k8s集群，
kubeadm降低部署門檻，但屏蔽了很多細節，遇到問題很難排查，如果想更容易可控，推薦使用二進制包部署k8s集群，
下面分別用這兩種方式來搭建集群，
RKE
RKE是一款經過CNCF認證的開源Kubernetes發行版，可以在Docker容器內運行，它通過洗掉大部分主機依賴項，并為部署、升級和回滾提供一個穩定的路徑，從而解決了Kubernetes最常見的安裝復雜性問題，
使用它安裝例外簡單，
安裝虛擬機
參考文章，注意虛擬機CPU數量至少分配為2，
kubeadm命令搭建集群
現在有了3臺Centos7的虛擬機，首先做的事情是系統初始化，比如永久關閉防火墻，
系統初始化
關閉防火墻
systemctl disable firewalld
關閉swap
sed -ri 's/.*swap.*/#&/' /etc/fstab
關閉selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config
設定主機名
hostnamectl set-hostname centos1 # 在三臺機器上分別執行
hostnamectl set-hostname centos2
hostnamectl set-hostname centos3
然后在所有機器上都執行
cat >> /etc/hosts <<EOF
172.20.10.2 centos1
172.20.10.13 centos2
172.20.10.14 centos3
EOF
時間同步
yum install ntpdate -y
ntpdate time.windows.com
安裝Docker
yum install wget -y # 安裝wget
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo #下載docker
yum -y install docker-ce-18.06.1.ce-3.el7 # 安裝docker
systemctl enable docker && systemctl start docker # 設成開機啟動
docker --version # 查看版本 檢驗是否安裝成功
輸出
[root@centos3 ~]# docker --version # 查看版本 檢驗是否安裝成功
Docker version 18.06.1-ce, build e68fc7a
安裝好了之后，為了能正常下載東西，需要改成國內鏡像：
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
重啟使生效
systemctl restart docker
安裝kubeadm,kubelet和kubectl
安裝之前，需要配置yum國內源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
下載
yum install -y kubelet-1.20.4 kubeadm-1.20.4 kubectl-1.20.4
當前安裝的版本是
================================================================================
Package 架構 版本 源 大小
================================================================================
正在安裝:
kubeadm x86_64 1.20.4-0 kubernetes 8.3 M
kubectl x86_64 1.20.4-0 kubernetes 8.5 M
kubelet x86_64 1.20.4-0 kubernetes 20 M
然后設定開機啟動
systemctl enable kubelet
查看所需的鏡像
[root@centos3 ~]# kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.20.5
k8s.gcr.io/kube-controller-manager:v1.20.5
k8s.gcr.io/kube-scheduler:v1.20.5
k8s.gcr.io/kube-proxy:v1.20.5
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0
可以看到，當前已經到了1.20.5的，但是此時國內鏡像源還未更新，因此我們只能安裝1.20.4
拉取相關鏡像
kubeadm config print init-defaults >init.default.yaml
然后主要修改以下幾行：
advertiseAddress: 172.20.10.2 #修改為指定的IP地址
imageRepository: registry.aliyuncs.com/google_containers #修改鏡像源
kubernetesVersion: v1.20.4 #修改版本
然后執行
kubeadm config images pull --config=init.default.yaml
根據組態檔來拉取相關鏡像，以免下面init的時候拉取失敗，
下面就可以部署啟動master了
部署master
在想作為Master的機器上執行
kubeadm init \
--apiserver-advertise-address=172.20.10.2 \ #指定master ip地址 ，應用于多網卡情況
--image-repository registry.aliyuncs.com/google_containers \ # 指定采用國內源
--kubernetes-version=1.20.4 \ # 指定版本，很重要，不然會拉最新的版本，可能出現國內源未更新的情況，導致部署失敗
--service-cidr=10.96.0.0/12 \ #用于安裝網路插件
--pod-network-cidr=10.244.0.0/16 #用于安裝網路插件
執行結果：
[root@centos3 ~]# kubeadm init \
> --apiserver-advertise-address=172.20.10.2 \
> --image-repository registry.aliyuncs.com/google_containers \
> --kubernetes-version=1.20.4 \
> --service-cidr=10.96.0.0/12 \
> --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.20.4
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [centos3 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.20.10.2]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [centos3 localhost] and IPs [172.20.10.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [centos3 localhost] and IPs [172.20.10.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 70.005199 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node centos3 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node centos3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: h445j1.egcjfuzsap4onq5g
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.20.10.2:6443 --token h445j1.egcjfuzsap4onq5g \
--discovery-token-ca-cert-hash sha256:13405f02dd37fabccbedd202924329291da1948f0ad8cb4cfe448f454f2104f2
根據提示執行下面代碼
[root@centos3 ~]# mkdir -p $HOME/.kube
[root@centos3 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@centos3 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
或者如果是root的話，直接執行：
export KUBECONFIG=/etc/kubernetes/admin.conf
然后還有其他節點加入集群的提示：
kubeadm join 172.20.10.2:6443 --token h445j1.egcjfuzsap4onq5g \
--discovery-token-ca-cert-hash sha256:13405f02dd37fabccbedd202924329291da1948f0ad8cb4cfe448f454f2104f2
但是不著急，我們先看下狀態：
[root@centos3 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos3 NotReady control-plane,master 5m31s v1.20.4
可以看到status是NotReady，不是Running，說明有一定的問題，
其實是因為有些組件沒有起來，一種解決方法就是安裝網路插件CNI，
[root@centos3 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
The connection to the server raw.githubusercontent.com was refused - did you specify the right host or port?
該網址訪問不了，是因為在外網，此時需要大家想辦法各顯神通了，
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.13.1-rc2
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.13.1-rc2
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
開個玩笑，其實也不長，我直接貼上來了，大家可以直接復制，
[root@centos3 ~]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
我就復制到檔案kube-flannel.yml中，
不出意外，過了一會，再執行
[root@centos3 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos3 Ready control-plane,master 17m v1.20.4
已經變成Ready了，KO，
加入集群
在另外兩臺機器上執行
[root@centos1 ~]# kubeadm join 172.20.10.2:6443 --token h445j1.egcjfuzsap4onq5g \
> --discovery-token-ca-cert-hash sha256:13405f02dd37fabccbedd202924329291da1948f0ad8cb4cfe448f454f2104f2
就是master節點最后輸出的提示，
輸出
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
稍等片刻，在master上執行
[root@centos3 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos1 Ready <none> 3m18s v1.20.4
centos2 Ready <none> 3m1s v1.20.4
centos3 Ready control-plane,master 22m v1.20.4
可以看到，都是Ready了，說明集群可用了，
測驗
在集群中創建一個pod，驗證是否正常運行：
kubectl create deployment nginx --image=nginx
kubectl get
等狀態變成Running再執行：
kubectl expose deployment nginx --port=80 --type=NodePort
查看最終狀態
[root@centos3 ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-6799fc88d8-dslgr 1/1 Running 0 2m45s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35m
service/nginx NodePort 10.99.100.132 <none> 80:30362/TCP 2m45s
訪問地址：http://節點IP:Port
注意埠是上買的的30362，下面以任意節點的IP+埠訪問：
好了，下面介紹二進制方式搭建集群
二進制方式搭建集群
這節有空補上，
rke
這是最容易的部署方式了，
由于公司電腦是ubuntu，因此這里的虛擬機環境換成了ubuntu20.4.2
兩臺ubuntu虛擬機，兩臺的/ec/hosts都添加：
192.168.1.6 rancher1
192.168.1.7 rancher2
192.168.1.6 rancher.my.com
下載工具
可以從 http://mirror.cnrancher.com 復制最新下載地址
wget http://rancher-mirror.cnrancher.com/helm/v3.5.3/helm-v3.5.3-linux-amd64.tar.gz
wget http://rancher-mirror.cnrancher.com/kubectl/v1.19.6/linux-amd64-v1.19.6-kubectl
wget http://rancher-mirror.cnrancher.com/rke/v1.2.7/rke_linux-amd64
如果決議不了地址，跟著下面修改DNS:
vim /etc/NetworkManager/NetworkManager.conf
在[main]中添加dns=no，保存退出
vim /etc/resolv.conf
添加nameserver 114.114.114.114 nameserver 8.8.8.8
mv rke_linux-amd64 /usr/bin/rke
mv linux-amd64-v1.19.6-kubectl /usr/bin/kubectl
tar -zxvf helm-v3.5.3-linux-amd64.tar.gz
mv linux-amd64/helm /usr/bin/helm
chmod +x /usr/bin/rke /usr/bin/kubectl /usr/bin/helm
生成ssh訪問公鑰
執行之前確保hosts配置OK
并且注意，rke注冊不能使用root用戶，
如在centos7系統下，需要先新增一個用戶，然后切換到新用戶，再執行下面的代碼，
比如，新增一個rancher用戶：adduser rancher passwd rancher usermod -aG docker rancher newgrp docker
ssh-keygen -t rsa -C "xxxx@qq.com"
ssh-copy-id rancher1
ssh-copy-id rancher2
#集群機器都推送，包括自身
RKE創建Rancher k8s集群
創建 rancher-cluster.yml 檔案，用于 rke 推送集群配置使用：
cluster_name: rancher
nodes:
- address: 192.168.1.6
user: yjw # docker組用戶，集群虛擬機必須都有這個用戶
role: [controlplane,worker,etcd]
- address: 192.168.1.7
user: yjw # docker組用戶
role:
- worker
services:
etcd:
snapshot: true
creation: 6h
retention: 24h
network:
plugin: weave
ingress:
provider: nginx
執行集群部署
rke up --config rancher-cluster.yml
等待部署完畢，正常顯示：
INFO[0138] Finished building Kubernetes cluster successfully
使用kubectl驗證集群健康狀態
mkdir -p ~/.kube
cp kube_config_rancher-cluster.yml ~/.kube/config
kubectl get nodes
kubectl get cs
查看k8s集群
yjw@rancher1:~/temp$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.1.6 Ready controlplane,etcd,worker 4m37s v1.20.5
192.168.1.7 Ready worker 4m34s v1.20.5
yjw@rancher1:~/temp$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
參考
1.Kubernetes免費視頻教程
2.K8s官網檔案
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/272850.html
標籤：其他
下一篇：Api-廣州移動手機號段介面