
| IP | 主機名 | CPU | 運行記憶體 | 備注 |
|---|---|---|---|---|
| 192.168.1.10 | node01 | 2 | 4 | Master and etcd |
| 192.168.1.20 | node02 | 2 | 4 | Master and etcd |
| 192.168.1.30 | node03 | 2 | 4 | Master and etcd |
| 192.168.1.40 | node04 | 1 | 2 | node |
| 192.168.1.50 | node05 | 1 | 2 | node |
| 192.168.1.60 | node06 | 1 | 2 | node |
| 軟體 | 版本 |
|---|---|
| kubernetes等組件 | V1.10.1 |
| docker | V1.13.1 |
①環境初始化
分別在6臺主機設定主機名稱
hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03
hostnamectl set-hostname node04
hostnamectl set-hostname node05
hostnamectl set-hostname node06
配置主機映射
echo '192.168.1.10 node01
192.168.1.20 node02
192.168.1.30 node03
192.168.1.40 node04
192.168.1.50 node05
192.168.1.60 node06' >> /etc/hosts
node01上執行ssh免密碼登陸配置
ssh-keygen #一路回車即可
ssh-copy-id -i node01
ssh-copy-id -i node02
ssh-copy-id -i node03
ssh-copy-id -i node04
ssh-copy-id -i node05
ssh-copy-id -i node06
六臺主機配置、停防火墻、關閉Swap、關閉Selinux、設定內核、安裝依賴包、配置ntp(配置完后建議重啟一次)
systemctl stop firewalld
systemctl disable firewalld
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config
modprobe br_netfilter
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge
yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim ntpdate libseccomp libtool-ltdl
systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536" >> /etc/security/limits.conf
echo "* hard nproc 65536" >> /etc/security/limits.conf
echo "* soft memlock unlimited" >> /etc/security/limits.conf
echo "* hard memlock unlimited" >> /etc/security/limits.conf
-----------------------------------------------------------------------------------------------------------------------
=======================================================================================================================
-----------------------------------------------------------------------------------------------------------------------
重啟過后建議在執行一次
sysctl -p /etc/sysctl.d/k8s.conf
如果報錯:
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 沒有那個檔案或目錄
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 沒有那個檔案或目錄
再次執行以下命令:
modprobe br_netfilter
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge
--------------------------------------------------------------------------------
修改iptables的另一種方法 區別:
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
1.臨時開啟,(寫入記憶體,在記憶體中開啟)
echo "1" > /proc/sys/net/ipv4/ip_forward
2.永久開啟,(寫入內核)
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p
②創建etcd證書(node01上執行即可)
設定cfssl環境
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH
創建 CA 組態檔(下面配置的IP為etc節點的IP)
mkdir /root/ssl
cd /root/ssl
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes-Soulmate": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "8760h"
}
}
}
}
EOF
cat > ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shanghai",
"L": "shanghai",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.1.10",
"192.168.1.20",
"192.168.1.30"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shanghai",
"L": "shanghai",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd
node01分發etcd證書到node02、node03上面
mkdir -p /etc/etcd/ssl
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
ssh -n node02 "mkdir -p /etc/etcd/ssl && exit"
ssh -n node03 "mkdir -p /etc/etcd/ssl && exit"
scp -r /etc/etcd/ssl/*.pem node02:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node03:/etc/etcd/ssl/
安裝配置etcd (三主節點)
安裝etcd
yum install etcd -y
mkdir -p /var/lib/etcd
node01的etcd.service
cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
--name node01 \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--initial-advertise-peer-urls https://192.168.1.10:2380 \
--listen-peer-urls https://192.168.1.10:2380 \
--listen-client-urls https://192.168.1.10:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://192.168.1.10:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
node02的etcd.service
cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
--name node02 \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--initial-advertise-peer-urls https://192.168.1.20:2380 \
--listen-peer-urls https://192.168.1.20:2380 \
--listen-client-urls https://192.168.1.20:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://192.168.1.20:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
node03的etcd.service
cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
--name node03 \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--initial-advertise-peer-urls https://192.168.1.30:2380 \
--listen-peer-urls https://192.168.1.30:2380 \
--listen-client-urls https://192.168.1.30:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://192.168.1.30:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
添加自啟動(etc集群最少2個節點才能啟動,啟動報錯看mesages日志)
mv /etc/systemd/system/etcd.service /usr/lib/systemd/system/
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd
在三個etcd節點執行一下命令檢查
etcdctl --endpoints=https://192.168.1.10:2379,https://192.168.1.20:2379,https://192.168.1.30:2379 \
--ca-file=/etc/etcd/ssl/ca.pem \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem cluster-health
③所有節點安裝配置docker
因為本次要部署的K8S版本為1.10.1版本,版本較低,所以docker版本不要安裝太高的版本(親測19.03版本不兼容)
yum install docker -y
systemctl start docker && systemctl enable docker
默認安裝的是1.13版本(目前最新版本是1.13版本),不要安裝docker-ce,docker-ce會安裝19.03版本(截止目前的最新版本)
查看docker cgroup
docker info
1.10.1版本的K8S使用的是cgroup driver,1.14之后的版本才推薦使用systemd,docker與K8S兩者必須要保持一致,否則會報錯

如果查看到的cgroup不為cgroupfs,需要修改
vim /usr/lib/systemd/system/docker.service

可以在添加一個阿里云加速

修改完成后多載配置,重啟docker
systemctl daemon-reload && systemctl restart docker
查看docker版本
# docker --version
Docker version 1.13.1, build 0be3e21/1.13.1
④安裝kubeadm,kubectl,kubelet
下載必要離線包,因為不下載的話,后面在初始化kubeadm時,會去自動拉去必要鏡像,有些鏡像需要翻墻才能下得下來
安裝包連接🔗
鏈接:https://pan.baidu.com/s/1tPGxcqUkepbGnVV934bOpQ
提取碼:r0cz
復制這段內容后打開百度網盤手機App,操作更方便哦
安裝,此步驟6臺機器全部都需要執行
注:本文將所有K8S的檔案放置在/root目錄下,以下代碼在使用時記得修改為實際路徑
cd /root/kubernetes-1.10
tar -xvf kube-packages-1.10.1.tar
cd kube-packages-1.10.1
rpm -Uvh * --force --nodeps
在所有kubernetes節點上設定kubelet使用cgroupfs,與dockerd保持一致,否則kubelet會啟動報錯
默認kubelet使用的cgroup-driver=systemd
sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
建議:/etc/systemd/system/kubelet.service.d/10-kubeadm.conf檔案中添加
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/k8sth/pause-amd64:3.0"
systemctl daemon-reload && systemctl restart kubelet && systemctl enable kubelet
匯入鏡像,只匯入了必要鏡像,后續鏡像很多的話,可以考慮搭建harbor存放鏡像,此步驟三天機器全部都需要執行
cd /root/kubernetes-1.10/
docker load -i k8s-images-1.10.tar.gz
命令補全
yum -y install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
⑤初始化集群
node01、node02、node03添加集群初始組態檔(集群組態檔一樣)
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
endpoints:
- https://192.168.1.10:2379
- https://192.168.1.20:2379
- https://192.168.1.30:2379
caFile: /etc/etcd/ssl/ca.pem
certFile: /etc/etcd/ssl/etcd.pem
keyFile: /etc/etcd/ssl/etcd-key.pem
dataDir: /var/lib/etcd
networking:
podSubnet: 10.244.0.0/16
kubernetesVersion: 1.10.1
api:
advertiseAddress: "192.168.1.10"
token: "b99a00.a144ef80536d4344"
tokenTTL: "0s"
apiServerCertSANs:
- node01
- 192.168.1.10
featureGates:
CoreDNS: true
imageRepository: "registry.cn-beijing.aliyuncs.com/k8sct"
這個config
首先node01初始化集群
組態檔定義podnetwork是10.244.0.0/16
kubeadmin init –hlep可以看出,service默認網段是10.96.0.0/12
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf默認dns地址cluster-dns=10.96.0.10
kubeadm init --config config.yaml
初始化失敗后處理辦法
kubeadm reset
rm -rf $HOME/.kube
#或
rm -rf $HOME/.kube
rm -rf /etc/kubernetes/*.conf
rm -rf /etc/kubernetes/manifests/*.yaml
docker ps -a |awk '{print $1}' |xargs docker rm -f
systemctl stop kubelet
初始化正常的結果如下
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.1.10:6443 --token b99a00.a144ef80536d4344 --discovery-token-ca-cert-hash sha256:7e234163db10f31e0fbb0c383410b81b8bd32f89fae1b947ab3f4ca75bd2f058
node01上面執行如下命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubeadm生成證書密碼檔案分發到node02和node03上面去(一會添加master節點)
scp -r /etc/kubernetes/pki node02:/etc/kubernetes/
scp -r /etc/kubernetes/pki node03:/etc/kubernetes/
部署flannel網路,只需要在node01執行就行
cd /root/kubernetes-1.10
kubectl apply -f kube-flannel.yml
查看狀態kubernetes節點狀態
[root@node01 kubernetes-1.10]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node01 Ready master 3m v1.10.1
[root@node01 kubernetes-1.10]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7997f8864c-85wds 1/1 Running 0 2m
kube-system coredns-7997f8864c-9wjtx 1/1 Running 0 2m
kube-system kube-apiserver-node01 1/1 Running 0 1m
kube-system kube-controller-manager-node01 1/1 Running 0 2m
kube-system kube-flannel-ds-ls2hp 1/1 Running 0 2m
kube-system kube-proxy-77zkv 1/1 Running 0 2m
kube-system kube-scheduler-node01 1/1 Running 0 1m
將node02和node03節點同樣執行
kubeadm init --config config.yaml
node02和node03輸出的內容應該是與node01一致 此時已經以master角色加入到集群了

node04、node05、node06執行命令加入集群以node角色加入
node01查看集群狀態
[root@node01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node01 Ready master 10m v1.10.1
node02 Ready master 4m v1.10.1
node03 Ready master 4m v1.10.1
node04 Ready <none> 39s v1.10.1
node05 Ready <none> 27s v1.10.1
node06 Ready <none> 17s v1.10.1
[root@node01 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7997f8864c-85wds 1/1 Running 0 13m
kube-system coredns-7997f8864c-9wjtx 1/1 Running 0 13m
kube-system kube-apiserver-node01 1/1 Running 0 12m
kube-system kube-apiserver-node02 1/1 Running 0 7m
kube-system kube-apiserver-node03 1/1 Running 0 7m
kube-system kube-controller-manager-node01 1/1 Running 0 12m
kube-system kube-controller-manager-node02 1/1 Running 0 7m
kube-system kube-controller-manager-node03 1/1 Running 0 7m
kube-system kube-flannel-ds-4dmg7 1/1 Running 0 7m
kube-system kube-flannel-ds-8whpg 1/1 Running 1 3m
kube-system kube-flannel-ds-g66s5 1/1 Running 0 3m
kube-system kube-flannel-ds-j5dk6 1/1 Running 0 4m
kube-system kube-flannel-ds-ls2hp 1/1 Running 0 12m
kube-system kube-flannel-ds-s4vcz 1/1 Running 0 7m
kube-system kube-proxy-4vm9g 1/1 Running 0 4m
kube-system kube-proxy-5mpng 1/1 Running 0 7m
kube-system kube-proxy-77zkv 1/1 Running 0 13m
kube-system kube-proxy-f67wb 1/1 Running 0 7m
kube-system kube-proxy-n4tlk 1/1 Running 0 3m
kube-system kube-proxy-q8sbm 1/1 Running 0 3m
kube-system kube-scheduler-node01 1/1 Running 0 12m
kube-system kube-scheduler-node02 1/1 Running 0 7m
kube-system kube-scheduler-node03 1/1 Running 0 7m
⑥Dashboard部署
直接使用離線包中的三個.yaml檔案即可完成部署
cd /root/k8s/kubernetes-1.10
kubectl apply -f kubernetes-dashboard-http.yaml -f admin-role.yaml -f kubernetes-dashboard-admin.rbac.yaml
完成后查看主機是否起了31000埠,如果已啟動,可通過ip:31000打開 K8S可視化頁面
[root@node01 kubernetes-1.10]# netstat -tunlp | grep 31000
tcp6 0 0 :::31000 :::* LISTEN 56139/kube-proxy

⑦Rancher部署
docker安裝rancher
docker run -d --name rancher --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher rancher/rancher:v2.2.4
界面訪問直接訪問IP





第一條授權經測驗 不執行也可以添加成功,
如果集群有證書,直接第三條命令,執行:
[root@node01 kubernetes-1.10]# curl --insecure -sfL https://192.168.1.10/v3/import/5s2xmsfnbrj89thgf4m25pm9j4s6mcczsg7rszwsc95zm49m6ndtm5.yaml | kubectl apply -f -
namespace "cattle-system" created
serviceaccount "cattle" created
clusterrolebinding.rbac.authorization.k8s.io "cattle-admin-binding" created
secret "cattle-credentials-664b64d" created
clusterrole.rbac.authorization.k8s.io "cattle-admin" created
deployment.extensions "cattle-cluster-agent" created
daemonset.extensions "cattle-node-agent" created
查看rancher agent:
[root@node01 ~]# kubectl get pod -n cattle-system
NAME READY STATUS RESTARTS AGE
cattle-cluster-agent-6559655864-p44qs 1/1 Running 0 1m
cattle-node-agent-9tnvs 1/1 Running 0 39s
cattle-node-agent-c9x69 1/1 Running 0 51s
cattle-node-agent-pt2r6 1/1 Running 0 59s
回到瀏覽器發現集群已經倒入,集群能正常使用

轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/234855.html
標籤:其他
