Keepalived+HAProxy實作K8S高可用集群-20210927

本文采用Keepalived+HAProxy的方式構建高可用集群，
當你如果你有硬體負載均衡設備當然更好了，

準備環境：

• 架構圖
  

注意：master集群采用奇數臺數，3、5、7…

• 所有節點都進行hosts檔案決議

tail -3 /etc/hosts
192.168.10.4 k8s-master01
192.168.10.5 k8s-master02
192.168.10.6 k8s-master03

• 所有節點都要安裝keepalived和haproxy軟體

yum -y install haproxy keepalived

• 修改haproxy組態檔（所有節點配置相同）
• 最好選擇2.x版本，當然這個版本也不影響使用，只是功能沒有2.x版本多

vim /etc/haproxy/haproxy.cfg
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

listen stats
  bind    *:8006
  mode    http
  stats   enable
  stats   hide-version
  stats   uri       /stats
  stats   refresh   30s
  stats   realm     Haproxy\ Statistics
  stats   auth      admin:admin

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.10.4:6443  check
  server k8s-master02	192.168.10.5:6443  check
  server k8s-master03   192.168.10.6:6443  check

• master01節點修改keepalived組態檔

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    mcast_src_ip 192.168.10.4
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.150/24
    }
    track_script {
       chk_apiserver
    }

• master02節點修改keepalived組態檔

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    mcast_src_ip 192.168.10.5
    virtual_router_id 51
    priority 50
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.150/24
    }
    track_script {
       chk_apiserver
    }
}

• master03節點修改keepalived組態檔

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    mcast_src_ip 192.168.10.6
    virtual_router_id 51
    priority 50
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.150/24
    }
    track_script {
       chk_apiserver
    }
}

• 所有節點創建健康檢查腳本

vim /etc/keepalived/check_apiserver.sh
#!/bin/bash

err=0
for k in $(seq 1 5)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 5
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

• 啟動haproxy與keepalived服務

systemctl daemon-reload 
systemctl enable --now haproxy
systemctl enable --now keepalived

• 可以用ping和telnet命令測驗一下vip的可用性

ping 192.168.10.150
PING 192.168.10.150 (192.168.10.150) 56(84) bytes of data.
64 bytes from 192.168.10.150: icmp_seq=1 ttl=64 time=1.60 ms
64 bytes from 192.168.10.150: icmp_seq=2 ttl=64 time=0.519 ms
64 bytes from 192.168.10.150: icmp_seq=3 ttl=64 time=0.874 ms
64 bytes from 192.168.10.150: icmp_seq=4 ttl=64 time=0.786 ms
^C
--- 192.168.10.150 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 0.519/0.946/1.606/0.403 ms

telnet 192.168.10.150 16443
Trying 192.168.10.150...
Connected to 192.168.10.150.
Escape character is '^]'.
Connection closed by foreign host.

• 再嘗試一下斷開vip所在節點的keepalived，看ip是否漂移，如果vip漂移至另一節點則代表成功

• 可能難免有地方出錯，如果出錯可以留言哈