創作不易，來了的客官點點關注，收藏，訂閱一鍵三連?😜

前言

我是一個即將畢業的大學生，超超，如果你也在學習Linux,python，不妨跟著萌新超超一起學習，拿下Linux，python一起加油，共同努力，拿到理想offer！

系列文章

專案實戰1 | 基于iptables的SNAT+DNAT與Docker容器發布的專案

概述

專案實戰有利于對已學知識的整合歸納，提高對相關知識的掌握程度，本次專案實戰利用swarm+Prometheus實作一個雙VIP高可用可監控的Web服務器集群，使用Nginx做負載均衡器，同時采用swarm管理的Docker集群并對外提供Web服務，使用keepalived的雙vip實作HA，使用Prometheus+Grafana實作對Web服務器的監控，

專案概述

專案名稱：基于Swarm+Prometheus實作雙VIP可監控Web高可用集群

專案環境：CentOS8(8臺，1核2G),Docker(20.10.8),keepalived(2.1.5),Prometheus(2.29.1),Grafana(8.1.2），Nginx(1.14.1)等

專案描述：利用swarm+Prometheus實作一個雙VIP高可用可監控的Web服務器集群，使用Nginx做負載均衡器，同時采用swarm管理的Docker集群并對外提供Web服務，使用keepalived的雙vip實作HA，使用Prometheus+Grafana實作對Web服務器的監控，

專案步驟

1.規劃整個專案的拓撲結構和專案的思維導圖

2.使用swarm實作Web集群部署，實作基于4個節點的swarm集群

3.創建NFS服務服務器為所有的節點提供相同Web資料，實作資料一致性

4.使用資料卷，為容器提供一致的Web資料

5.使用Nginx+keepalived實作雙VIP負載均衡高可用

6.使用Prometheus實作對swarm集群的監控，結合Grafana成圖工具進行資料展示

專案心得

1.通過網路拓補圖和思維導圖的建立，提高了專案整體的落實和效率

2.對于容器編排工具swarm的使用和集群的部署更為熟悉

3.對于keepalived+nginx實作高可用負載均衡更為了解

4.對于Prometheus+Grafana實作系統監控有了更深的理解

5.對于腦裂現象的出現和解決有了更加清晰的認識

6.通過根據官方檔案安裝與使用swarm到集群整體的部署，進一步提高了自身的自主學習和troubleshooting能力

專案詳細代碼

一、規劃整個專案的拓撲結構和專案的思維導圖

網路拓撲圖

思維導圖

專案服務器如下：

IP：192.168.232.132 主機名：docker-manager-1 擔任角色：swarm manager

IP：192.168.232.133 主機名：docker-2 擔任角色：swarm worker node1

IP：192.168.232.134 主機名：docker-3 擔任角色：swarm worker node2

IP：192.168.232.131 主機(ubuntu)名：chaochao 擔任角色：swarm worker node3

IP：192.168.232.135 主機名：nfs-server 擔任角色：nfs服務服務器

IP：192.168.232.136 主機名：load-balancer擔任角色：負載均衡器(master)

IP：192.168.232.137 主機名：load-balancer擔任角色：負載均衡器（backup）

IP：192.168.232.138 主機名：prometheus-server 擔任角色：prometheus-server

二、使用swarm實作Web集群部署，實作基于4個節點的swarm集群

1.部署web服務集群機器環境(四臺機器，CentOS8與Ubuntu系統)

IP：192.168.232.132 主機名：docker-manager-1 擔任角色：swarm manager

IP：192.168.232.133 主機名：docker-2 擔任角色：swarm worker node1

IP：192.168.232.134 主機名：docker-3 擔任角色：swarm worker node2

IP：192.168.232.131 主機(ubuntu)名：chaochao 擔任角色：swarm worker node3

2.配置hosts檔案

swarm manager：

[root@docker-manager-1 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.232.132 manager

192.168.232.133 worker1

192.168.232.134 worker2

192.168.232.131 worker3

[root@docker-manager-1 ~]# getenforce

Disabled

另外三臺worker node操作一致

3.設定防火墻和sellinux

關閉四臺機器上的防火墻，

如果是開啟防火墻，則需要在所有節點的防火墻上依次放行2377/tcp（管理埠）、7946/udp（節點間通信埠）、4789/udp（overlay 網路埠，容器之間）埠，

[root@docker-manager-1 ~]# systemctl disable firewalld

[root@docker-manager-1 ~]#

[root@docker-2 ~]# systemctl disable firewalld

[root@docker-2 ~]#

另外兩臺操作一致，不再贅述，

4.重啟docker服務，防止導致網路例外

[root@docker-manager-1 ~]# service docker restart

Redirecting to /bin/systemctl restart docker.service

[root@docker-manager-1 ~]#

[root@docker-2 ~]# service docker restart

Redirecting to /bin/systemctl restart docker.service

[root@docker-2 ~]#

[root@docker-3 ~]# service docker restart

Redirecting to /bin/systemctl restart docker.service

[root@docker-3 ~]#

root@chaochao:~# service docker restart

root@chaochao:~#

5.創建swarm集群

對于manager：

# 命令：docker swarm init --advertise-addr manager的IP地址

[root@docker-manager-1 ~]# docker swarm init --advertise-addr 192.168.232.132

Swarm initialized: current node (ooauma1x037wufqkh21uj0j7v) is now a manager.

 

To add a worker to this swarm, run the following command:

 

    docker swarm join --token SWMTKN-1-0200k3nv43fmc6hcuurx8z1iehsqq6uro12qjfeoxrkmk9fmom-1ub4wsmlpl4zhqalzdrgukx3l 192.168.232.132:2377

 

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

 

[root@docker-manager-1 ~]#

對于三臺swarm nodes

輸入manager創建的命令： docker swarm join --token ……,此命令在manager創建swarm時會自動生成，

[root@docker-2 ~]# docker swarm join --token SWMTKN-1-0200k3nv43fmc6hcuurx8z1iehsqq6uro12qjfeoxrkmk9fmom-1ub4wsmlpl4zhqalzdrgukx3l 192.168.232.132:2377

This node joined a swarm as a worker.

[root@docker-2 ~]#

[root@docker-3 ~]# docker swarm join --token SWMTKN-1-0200k3nv43fmc6hcuurx8z1iehsqq6uro12qjfeoxrkmk9fmom-1ub4wsmlpl4zhqalzdrgukx3l 192.168.232.132:2377

This node joined a swarm as a worker.

[root@docker-3 ~]#

root@chaochao:~# docker swarm join --token SWMTKN-1-0200k3nv43fmc6hcuurx8z1iehsqq6uro12qjfeoxrkmk9fmom-1ub4wsmlpl4zhqalzdrgukx3l 192.168.232.132:2377

This node joined a swarm as a worker.

root@chaochao:~#

三、創建NFS服務服務器為所有的節點提供相同Web資料，使用資料卷，為容器提供一致的Web資料

1.準備一臺服務器擔任NFS Server

客戶機配置：CentOS8（1核/2G）

IP地址：192.168.232.135

修改好主機名：nfs-server

root@docker-4 ~]# hostnamectl set-hostname nfs-server

[root@docker-4 ~]# su

[root@nfs-server ~]#

共享的Web集群服務器資訊如下：

IP：192.168.232.132 主機名：docker-manager-1 擔任角色：swarm manager

IP：192.168.232.133 主機名：docker-2 擔任角色：swarm node1

IP：192.168.232.134 主機名：docker-3 擔任角色：swarm node2

IP：192.168.232.131 主機(ubuntu)名：chaochao 擔任角色：swarm node3

2.安裝和啟動nfs服務

[root@nfs-server ~]# yum install nfs-utils -y

[root@nfs-server ~]# service nfs-server start

Redirecting to /bin/systemctl start nfs-server.service

[root@nfs-server ~]#

查看nfs服務的行程： ps aux|grep nfs

[root@nfs-server ~]# ps aux|grep nfs

root        2346  0.0  0.1  50108  2672 ?        Ss   19:39   0:00 /usr/sbin/nfsdcld

root        2352  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2353  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2354  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2355  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2356  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2357  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2358  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2359  0.0  0.0      0     0 ?        S    19:39   0:00 [nfsd]

root        2367  0.0  0.0  12324  1064 pts/0    S+   19:40   0:00 grep --color=auto nfs

[root@nfs-server ~]#

3.通過NFS共享檔案

編輯/etc/exports,寫好共享的具體目錄、權限以及共享的網段、IP，

[root@nfs-server /]# vim /etc/exports

[root@nfs-server /]# cat /etc/exports

/web 192.168.232.0/24(rw,all_squash,sync)

/download 192.168.232.0/24(ro,all_squash,sync)

[root@nfs-server /]#

重繪輸出檔案的串列

[root@nfs-server /]# exportfs -rv

exporting 192.168.232.0/24:/download

exporting 192.168.232.0/24:/web

[root@nfs-server /]#

[root@nfs-server /]# cd /download/

[root@nfs-server download]# ls

[root@nfs-server download]# vim chao.txt

[root@nfs-server download]# ls

chao.txt

[root@nfs-server download]# exportfs -rv

exporting 192.168.232.0/24:/download

exporting 192.168.232.0/24:/web

[root@nfs-server download]#

4.關閉防火墻和selinux

[root@nfs-server download]# service firewalld stop

Redirecting to /bin/systemctl stop firewalld.service

[root@nfs-server download]# systemctl disable firewalld

[root@nfs-server download]# getenforce

Disabled

[root@nfs-server download]#

5.在客戶機上掛載宿主機上的NFS服務器共享的目錄

在客戶機上安裝NFS服務：yum install nfs-utils -y

[root@docker-2 lianxi]# yum install nfs-utils -y

查看宿主機共享的目錄

[root@docker-2 lianxi]# showmount -e 192.168.232.135

Export list for 192.168.232.135:

/download 192.168.232.0/24

/web      192.168.232.0/24

[root@docker-2 lianxi]#

6.掛載nfs-server上的共享目錄到客戶機

[root@docker-2 lianxi]# mount 192.168.232.135:/web /web

[root@docker-2 lianxi]# cd /web

[root@docker-2 web]# ls

1.jpg  index.html  rep.html

[root@docker-2 web]#

[root@docker-2 web]# df -Th

檔案系統             型別      容量  已用  可用 已用% 掛載點

devtmpfs             devtmpfs  876M     0  876M    0% /dev

tmpfs                tmpfs     896M     0  896M    0% /dev/shm

tmpfs                tmpfs     896M   18M  878M    2% /run

tmpfs                tmpfs     896M     0  896M    0% /sys/fs/cgroup

/dev/mapper/cl-root  xfs        17G  8.2G  8.9G   48% /

/dev/sda1            xfs      1014M  193M  822M   19% /boot

tmpfs                tmpfs     180M     0  180M    0% /run/user/0

overlay              overlay    17G  8.2G  8.9G   48% /var/lib/docker/overlay2/c2434295873b6ce0f136d4851cb9a9bf10b1ebf77e80f611841484967b857c94/merged

overlay              overlay    17G  8.2G  8.9G   48% /var/lib/docker/overlay2/8b5420179cd05a4a8ea039ba9357f7d59ddfec2fd3f185702a5a0d97883564f2/merged

overlay              overlay    17G  8.2G  8.9G   48% /var/lib/docker/overlay2/d9f99510abb4c5d5496e54d11ff763be47610ee7851207aec9fdbb1022f14016/merged

overlay              overlay    17G  8.2G  8.9G   48% /var/lib/docker/overlay2/5a87e34567ece3a64a835bcd4cfe59d2ebdf0d36bf74fbd07dff8c82a94f37a2/merged

overlay              overlay    17G  8.2G  8.9G   48% /var/lib/docker/overlay2/b70caa1ed0c781711a41cd82a0a91d465a6e02418633bfa00ce398c92405baff/merged

192.168.232.135:/web nfs4       17G  7.8G  9.2G   46% /web

[root@docker-2 web]#

[root@docker-manager-1 web]# ls

[root@docker-manager-1 web]# mount 192.168.232.135:/web /web

[root@docker-manager-1 web]# ls

[root@docker-manager-1 web]# cd /web

[root@docker-manager-1 web]# ls

1.jpg  index.html  rep.html

[root@docker-manager-1 web]# df -Th

……

overlay              overlay    17G  8.3G  8.8G   49% /var/lib/docker/overlay2/06cbd339366e8aeb492b21561573d953073e203122262e5089461da0d0d316a0/merged

overlay              overlay    17G  8.3G  8.8G   49% /var/lib/docker/overlay2/a132f32c9cff25a1a143e325f2aecd0186630df66748c95984bb3cf2ce9fe8b2/merged

overlay              overlay    17G  8.3G  8.8G   49% /var/lib/docker/overlay2/e1efba32267a46940402f682034d07ed51b8ee200186d5acc0c48144cd9fe31e/merged

192.168.232.135:/web nfs4       17G  7.8G  9.2G   46% /web

[root@docker-manager-1 web]#

[root@docker-3 web]# showmount 192.168.232.135 -e

Export list for 192.168.232.135:

/download 192.168.232.0/24

/web      192.168.232.0/24

[root@docker-3 web]# mount 192.168.232.135:/web /web

[root@docker-3 web]# cd /web

[root@docker-3 web]# ls

1.jpg  index.html  rep.html

[root@docker-3 web]# df -TH

檔案系統             型別      容量  已用  可用 已用% 掛載點

devtmpfs             devtmpfs  919M     0  919M    0% /dev

tmpfs                tmpfs     939M     0  939M    0% /dev/shm

tmpfs                tmpfs     939M   18M  921M    2% /run

tmpfs                tmpfs     939M     0  939M    0% /sys/fs/cgroup

/dev/mapper/cl-root  xfs        19G  8.8G  9.6G   48% /

/dev/sda1            xfs       1.1G  202M  862M   19% /boot

tmpfs                tmpfs     188M     0  188M    0% /run/user/0

overlay              overlay    19G  8.8G  9.6G   48% /var/lib/docker/overlay2/8b5420179cd05a4a8ea039ba9357f7d59ddfec2fd3f185702a5a0d97883564f2/merged

overlay              overlay    19G  8.8G  9.6G   48% /var/lib/docker/overlay2/c2434295873b6ce0f136d4851cb9a9bf10b1ebf77e80f611841484967b857c94/merged

overlay              overlay    19G  8.8G  9.6G   48% /var/lib/docker/overlay2/6b933ab92577f653a29dcac782a2c5e79bcdbbf219a7ccebb38b585ef117e0b4/merged

overlay              overlay    19G  8.8G  9.6G   48% /var/lib/docker/overlay2/40cc77a0f1c7281915947c1fefeb595837eb75fffec0d808a9994ac1fbde5f90/merged

192.168.232.135:/web nfs4       19G  8.4G  9.9G   46% /web

[root@docker-3 web]#

root@chaochao:~# mkdir /web

root@chaochao:~# mount 192.168.232.135:/web /web

root@chaochao:~# cd /web

root@chaochao:/web# ls

1.jpg  index.html  rep.html

root@chaochao:/web# df -Th

Filesystem                        Type      Size  Used Avail Use% Mounted on

udev                              devtmpfs  433M     0  433M   0% /dev

tmpfs                             tmpfs      96M  1.6M   94M   2% /run

/dev/mapper/ubuntu--vg-ubuntu--lv ext4       19G  6.9G   11G  39% /

tmpfs                             tmpfs     477M     0  477M   0% /dev/shm

tmpfs                             tmpfs     5.0M     0  5.0M   0% /run/lock

tmpfs                             tmpfs     477M     0  477M   0% /sys/fs/cgroup

/dev/sda2                         ext4      976M  202M  707M  23% /boot

/dev/loop1                        squashfs   70M   70M     0 100% /snap/lxd/19188

/dev/loop3                        squashfs   71M   71M     0 100% /snap/lxd/21029

/dev/loop2                        squashfs   56M   56M     0 100% /snap/core18/2074

/dev/loop4                        squashfs   33M   33M     0 100% /snap/snapd/12704

/dev/loop5                        squashfs   32M   32M     0 100% /snap/snapd/10707

/dev/loop6                        squashfs   56M   56M     0 100% /snap/core18/2128

tmpfs                             tmpfs      96M     0   96M   0% /run/user/1000

overlay                           overlay    19G  6.9G   11G  39% /var/lib/docker/overlay2/a2d7dea3b856302cf61d9584be91aa69614a8b25db3a7b6c91317d71a6d68a3c/merged

overlay                           overlay    19G  6.9G   11G  39% /var/lib/docker/overlay2/c6188638a4df298b840dce222da041733de01e636362c34c4c0e34cec9a34e08/merged

overlay                           overlay    19G  6.9G   11G  39% /var/lib/docker/overlay2/e94850d48ea1962457edacef2d09cfaa838fad8b9899d4455c9b31caa11c07e1/merged

192.168.232.135:/web              nfs4       17G  7.8G  9.2G  46% /web

root@chaochao:/web#

7.使用nfs服務服務器共享的檔案，通過volume資料卷是否共享檔案成功

命令：docker service create --name nfs-service-1 --mount 'type=volume,source=nfs-volume,target=/usr/share/nginx/html,volume-driver=local,volume-opt=type=nfs,volume-opt=device=:/web,"volume-opt=o=addr=192.168.232.135,rw,nfsvers=4,async"' --replicas 10 -p 8026:80 nginx:latest

source=nfs-volume  --> docker宿主機上的卷的名字

/usr/share/nginx/html -->容器里存放網頁的目錄

volume-driver=local  --> 訪問本地的某個目錄的

volume-opt=type=nfs  --> volume對nfs的支持選項

volume-opt=device=:/web --> 是nfs服務器共享目錄

volume-opt=o=addr=192.168.232.135,rw,nfsvers=4,async  掛載具體的nfs服務器的ip地址和選項

--replicas 10  副本的數量

nfsvers=4 --> nfs版本

async  --> 異步

 

[root@docker-manager-1 web]# docker service create --name nfs-service-1  --mount 'type=volume,source=nfs-volume,target=/usr/share/nginx/html,volume-driver=local,volume-opt=type=nfs,volume-opt=device=:/web,"volume-opt=o=addr=192.168.232.135,rw,nfsvers=4,async"' --replicas 10 -p 8026:80 nginx:latest

3ws4b26bo2tgi48czckm6jin7

overall progress: 10 out of 10 tasks

1/10: running   [==================================================>]

2/10: running   [==================================================>]

3/10: running   [==================================================>]

4/10: running   [==================================================>]

5/10: running   [==================================================>]

6/10: running   [==================================================>]

7/10: running   [==================================================>]

8/10: running   [==================================================>]

9/10: running   [==================================================>]

10/10: running   [==================================================>]

verify: Service converged

[root@docker-manager-1 web]#

[root@docker-manager-1 ~]# cd /var/lib/docker/volumes/nfs-volume/_data

[root@docker-manager-1 _data]# ls

1.jpg  index.html  rep.html

[root@docker-manager-1 _data]#

 

[root@docker-2 ~]# cd /var/lib/docker/volumes/nfs-volume/_data

[root@docker-2 _data]# ls

1.jpg  index.html  rep.html

[root@docker-2 _data]#

8.訪問服務，成功實作檔案共享

四、使用Nginx+keepalived實作雙VIP負載均衡高可用

首先實作負載均衡，步驟如下：

1.準備兩臺臺客戶機作為負載均衡器

IP：192.168.232.136 主機名：load-balancer擔任角色：負載均衡器（master）

IP：192.168.232.137 主機名：load-balancer擔任角色：負載均衡器（backup）

2.編譯腳本

[root@load-balancer ~]# vim onekey_install_lizhichao_nginx_v10.sh

[root@load-balancer ~]# cat onekey_install_lizhichao_nginx_v10.sh

#!/bin/bash

 

#解決軟體的依賴關系，需要安裝的軟體包

yum -y install zlib zlib-devel openssl openssl-devel pcre pcre-devel gcc gcc-c++ autoconf automake make psmisc net-tools lsof vim wget

 

#新建chaochao用戶和組

id  chaochao || useradd chaochao -s /sbin/nologin

 

#下載nginx軟體

mkdir  /lzc_load_balancing -p

cd /lzc_load_balancing

wget  http://nginx.org/download/nginx-1.21.1.tar.gz

 

#解壓軟體

tar xf nginx-1.21.1.tar.gz

#進入解壓后的檔案夾

cd nginx-1.21.1

 

#編譯前的配置

./configure --prefix=/usr/local/lzc_load_balancing  --user=chaochao --group=chaochao  --with-http_ssl_module   --with-threads  --with-http_v2_module  --with-http_stub_status_module  --with-stream

 

#如果上面的編譯前的配置失敗，直接退出腳本

if (( $? != 0));then

  exit

fi

#編譯

make -j 2

#編譯安裝

make  install

 

#修改PATH變數

echo  "PATH=$PATH:/usr/local/lzc_load_balancing/sbin" >>/root/.bashrc

#執行修改了環境變數的腳本

source /root/.bashrc

 

 

#firewalld and selinux

 

#stop firewall和設定下次開機不啟動firewalld

service firewalld stop

systemctl disable firewalld

 

#臨時停止selinux和永久停止selinux

setenforce 0

sed  -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config

 

#開機啟動

chmod +x /etc/rc.d/rc.local

echo  "/usr/local/lzc_load_balancing/sbin/nginx" >>/etc/rc.local

 

[root@load-balancer ~]#

3.安裝運行腳本

[root@load-balacer ~]# bash onekey_install_lizhichao_nginx_v10.sh

……

test -d '/usr/local/lzc_load_balancing/logs' \

|| mkdir -p '/usr/local/lzc_load_balancing/logs'

make[1]: 離開目錄“/lzc_load_balancing/nginx-1.21.1”

Redirecting to /bin/systemctl stop firewalld.service

[root@load-balancer ~]#

4.啟動nginx

命令：

nginx 啟動nginx

nginx -s stop 關閉nginx

[root@load-balancer nginx-1.21.1]# nginx

[root@load-balancer nginx-1.21.1]#

[root@load-balancer nginx-1.21.1]# ps aux|grep nginx

root        9301  0.0  0.2 119148  2176 ?        Ss   18:20   0:00 nginx: master process nginx

nginx       9302  0.0  0.9 151824  7912 ?        S    18:20   0:00 nginx: worker process

root        9315  0.0  0.1  12344  1108 pts/0    S+   18:21   0:00 grep --color=auto nginx

[root@load-banlancer nginx-1.21.1]# ss -anplut|grep nginx

tcp     LISTEN   0        128              0.0.0.0:80            0.0.0.0:*       users:(("nginx",pid=9302,fd=9),("nginx",pid=9301,fd=9))                       

tcp     LISTEN   0        128                 [::]:80               [::]:*       users:(("nginx",pid=9302,fd=10),("nginx",pid=9301,fd=10))                     

[root@load-banlancer nginx-1.21.1]#

5.配置nginx里的負載均衡功能

[root@load-balancer nginx-1.21.1]# cd /usr/local/lzc_load_balancing/

[root@load-balancer lzc_load_balancing]# ls

conf  html  logs  sbin

[root@load-balancer lzc_load_balancing]# cd conf/

[root@load-balancer conf]# ls

fastcgi.conf          fastcgi_params.default  mime.types          nginx.conf.default   uwsgi_params

fastcgi.conf.default  koi-utf                 mime.types.default  scgi_params          uwsgi_params.default

fastcgi_params        koi-win                 nginx.conf          scgi_params.default  win-utf

[root@load-balancer conf]# vim nginx.conf

[root@load-balancer conf]# cat nginx.conf #以下僅顯示修改了的腳本部分

http {

    include       mime.types;

    default_type  application/octet-stream;

 

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

    #                  '$status $body_bytes_sent "$http_referer" '

    #                  '"$http_user_agent" "$http_x_forwarded_for"';

 

    #access_log  logs/access.log  main;

 

    sendfile        on;

    #tcp_nopush     on;

 

    #keepalive_timeout  0;

    keepalive_timeout  65;

 

    #gzip  on;

    upstream chaoweb{ #定義一個負載均衡器的名字為：chaoweb

        server 192.168.232.132:8026;

        server 192.168.232.131:8026;

        server 192.168.232.133:8026;

        server 192.168.232.134:8026;

       

}

    server {

        listen       80;

        server_name  www.lizhichao.com; #設定域名為www.sc.com

 

        #charset koi8-r;

 

        #access_log  logs/host.access.log  main;

 

        location / {

        server 192.168.232.134:8026;

 

}

    server {

        listen       80;

        server_name  www.lizhichao.com; #設定域名為www.sc.com

        location /{

            proxy_pass http://chaoweb; #呼叫負載均衡器

}

[root@load-balancer conf]# nginx -s reload # 重新加載組態檔

[root@load-banlancer conf]# ps aux|grep nginx

root        9301  0.0  1.2 120068  9824 ?        Ss   18:20   0:00 nginx: master process nginx

nginx       9395  0.1  1.0 152756  8724 ?        S    19:16   0:00 nginx: worker process

root        9397  0.0  0.1  12344  1044 pts/0    S+   19:18   0:00 grep --color=auto nginx

[root@load-balancer conf]#

6.在Windows添加IP映射并查看效果

修改windows的hosts檔案，點擊此處查看方法

在C:\Windows\System32\drivers\etc的hosts檔案

在swarm集群上查看，

[root@docker-manager-1 ~]# vim /etc/hosts

[root@docker-manager-1 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.232.132 manager

192.168.232.133 worker1

192.168.232.134 worker2

192.168.232.131 worker3

192.168.232.136 www.lizhichao.com

[root@docker-manager-1 ~]# curl www.lizhichao.com

<html>

<head>

<title>chaochao</title>

</head>

<body>

<p>name:chaochao</p>

<p>sex:male</p>

<p>tel:1517388321</p>

<img src=1.jpg>

<a href=rep.html>reputation</a>

</body>

</html>

[root@docker-manager-1 ~]#

7.查看負載均衡的分配情況

用抓包工具來查看：tcpdump

[root@load-balancer ~]# yum install tcpdump -y

另一臺負載均衡器操作一致，不再贅述，（同時也可以通過克隆客戶機快速實作兩臺負載均衡

接下來展示使用keepalived實作雙VIP高可用的步驟：

1.安裝keepalived

命令：yum install keepalived -y

[root@load-balancer ~]# yum install keepalived -y

[root@load-balancer-2 ~]# yum install keepalived -y

2.配置keepalived.conf檔案

對于單VIP下的master：

[root@load-balancer ~]# vim /etc/keepalived/keepalived.conf

[root@load-balancer ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

 

global_defs {

   notification_email {

     acassen@firewall.loc

     failover@firewall.loc

     sysadmin@firewall.loc

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 192.168.200.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

   vrrp_skip_check_adv_addr

   #vrrp_strict

   vrrp_garp_interval 0

   vrrp_gna_interval 0

}

 

vrrp_instance VI_1 {

    state MASTER

    interface ens33

    virtual_router_id 168

    priority 220

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.232.168

    }

}

 

vrrp_instance VI_2 {

    state BACKUP

    interface ens33

    virtual_router_id 169

    priority 180

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.232.169

    }

}

 

[root@load-balancer ~]#

對于單VIP下的backup：

[root@load-balancer-2 ~]# vim /etc/keepalived/keepalived.conf

[root@load-balancer-2 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

 

global_defs {

   notification_email {

     acassen@firewall.loc

     failover@firewall.loc

     sysadmin@firewall.loc

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 192.168.200.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

   vrrp_skip_check_adv_addr

   #vrrp_strict

   vrrp_garp_interval 0

   vrrp_gna_interval 0

}

 

vrrp_instance VI_1 {

    state BACKUP

    interface ens33

    virtual_router_id 168

    priority 130

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.232.168

}

}

 

vrrp_instance VI_2 {

    state MASTER

    interface ens33

    virtual_router_id 169

    priority 200

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.232.169

    }

}

 

[root@load-balancer-2 ~]#

2.重啟keepalived服務

對于單VIP下的master：

[root@load-balancer ~]# service keepalived restart

Redirecting to /bin/systemctl restart keepalived.service

[root@load-balancer ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

    link/ether 00:0c:29:9a:d6:b6 brd ff:ff:ff:ff:ff:ff

    inet 192.168.232.136/24 brd 192.168.232.255 scope global dynamic noprefixroute ens33

       valid_lft 1374sec preferred_lft 1374sec

    inet 192.168.232.168/32 scope global ens33

       valid_lft forever preferred_lft forever

    inet6 fe80::b4cd:b005:c610:7b3b/64 scope link dadfailed tentative noprefixroute

       valid_lft forever preferred_lft forever

    inet6 fe80::40fb:5be0:b6f9:b063/64 scope link dadfailed tentative noprefixroute

       valid_lft forever preferred_lft forever

    inet6 fe80::2513:c641:3555:5eeb/64 scope link dadfailed tentative noprefixroute

       valid_lft forever preferred_lft forever

[root@load-balancer ~]#

對于單VIP下的backup：

[root@load-balancer-2 ~]# service keepalived restart

Redirecting to /bin/systemctl restart keepalived.service

[root@load-balancer-2 ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

    link/ether 00:0c:29:04:e5:b4 brd ff:ff:ff:ff:ff:ff

    inet 192.168.232.137/24 brd 192.168.232.255 scope global dynamic noprefixroute ens33

       valid_lft 1435sec preferred_lft 1435sec

    inet 192.168.232.169/32 scope global ens33

       valid_lft forever preferred_lft forever

    inet6 fe80::b4cd:b005:c610:7b3b/64 scope link dadfailed tentative noprefixroute

       valid_lft forever preferred_lft forever

    inet6 fe80::40fb:5be0:b6f9:b063/64 scope link dadfailed tentative noprefixroute

       valid_lft forever preferred_lft forever

    inet6 fe80::2513:c641:3555:5eeb/64 scope link dadfailed tentative noprefixroute

       valid_lft forever preferred_lft forever

[root@load-balancer-2 ~]#

3.測驗是否成功實作

[root@load-balancer-2 ~]# nginx

[root@load-balancer-2 ~]# curl 192.168.232.169

<html>

<head>

<title>chaochao</title>

</head>

<body>

<p>name:chaochao</p>

<p>sex:male</p>

<p>tel:1517388321</p>

<img src=1.jpg>

<a href=rep.html>reputation</a>

</body>

</html>

[root@load-balancer-2 ~]#

五、使用Prometheus實作對swarm集群的監控，結合Grafana成圖工具進行資料展示

1.環境部署

IP：192.168.232.138 主機名：prometheus-server 擔任角色：prometheus-server

IP：192.168.232.136 主機名：load-balancer擔任角色：負載均衡器(master)

IP：192.168.232.137 主機名：load-balancer擔任角色：負載均衡器（backup）

IP：192.168.232.132 主機名：docker-manager-1 擔任角色：swarm manager

IP：192.168.232.133 主機名：docker-2 擔任角色：swarm worker node1

IP：192.168.232.134 主機名：docker-3 擔任角色：swarm worker node2

IP：192.168.232.131 主機(ubuntu)名：chaochao 擔任角色：swarm worker node3

IP：192.168.232.135 主機名：nfs-server 擔任角色：nfs服務器

2.下載Prometheus安裝包并解壓到服務器

[root@prometheus-server ~]# mkdir /prometheus

[root@prometheus-server ~]# cp prometheus-2.29.1.linux-amd64.tar.gz /prometheus/

[root@prometheus-server ~]# cd /prometheus

[root@prometheus-server prometheus]# ls

prometheus-2.29.1.linux-amd64.tar.gz

[root@prometheus-server prometheus]# tar xf prometheus-2.29.1.linux-amd64.tar.gz

[root@prometheus-server prometheus]# ls

prometheus-2.29.1.linux-amd64  prometheus-2.29.1.linux-amd64.tar.gz

[root@prometheus-server prometheus]# cd prometheus-2.29.1.linux-amd64

[root@prometheus-server prometheus-2.29.1.linux-amd64]# ls

console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool

3.修改環境變數

[root@prometheus-server prometheus-2.29.1.linux-amd64]# PATH=$PATH:/prometheus/prometheus-2.29.1.linux-amd64

[root@prometheus-server prometheus-2.29.1.linux-amd64]# which prometheus

/prometheus/prometheus-2.29.1.linux-amd64/prometheus

[root@prometheus-server prometheus-2.29.1.linux-amd64]#

問題：誤輸入PATH命令導致ls，vim等識別不了

解決方法：輸入命令：export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

TIPS：

此步驟可以添加環境變數到.bashrc檔案

[root@prometheus-server prometheus-2.29.1.linux-amd64]# cat /root/.bashrc

# .bashrc

 

# User specific aliases and functions

 

alias rm='rm -i'

alias cp='cp -i'

alias mv='mv -i'

 

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/lzc_load_balancing/sbin

PATH=$PATH:/prometheus/prometheus-2.29.1.linux-amd64

[root@prometheus-server prometheus-2.29.1.linux-amd64]#

4.啟動Prometheus

命令：./prometheus --config.file=prometheus.yml

后臺運行命令：nohup ./prometheus --config.file=prometheus.yml &

[root@prometheus-server prometheus-2.29.1.linux-amd64]# ./prometheus --config.file=prometheus.yml

level=info ts=2021-08-25T12:00:48.976Z caller=main.go:390 msg="No time or size retention was set so using the default time retention" duration=15d

level=info ts=2021-08-25T12:00:48.977Z caller=main.go:428 msg="Starting Prometheus" version="(version=2.29.1, branch=HEAD, revision=dcb07e8eac34b5ea37cd229545000b857f1c1637)"

level=info ts=2021-08-25T12:00:48.977Z caller=main.go:433 build_context="(go=go1.16.7, user=root@364730518a4e, date=20210811-14:48:27)"

5.查看Prometheus行程

[root@prometheus-server prometheus-2.29.1.linux-amd64]# ps aux|grep prometheus

root        1677  2.6  8.1 780596 64948 pts/0    Sl+  20:00   0:01 ./prometheus --config.file=prometheus.yml

root        1733  0.0  0.1  12344  1060 pts/1    R+   20:01   0:00 grep --color=auto prometheus

[root@prometheus-server prometheus-2.29.1.linux-amd64]#

6.訪問Prometheus

http://localhost:9090/graph --》web界面的

http://localhost:9090/metrics ---》查看prometheus的各種指標

接下來在被監控的服務器上安裝exporter

在swarm manager上安裝exporter為例

1.安裝node_exporter

[root@docker-manager-1 ~]# mkdir /exporter

[root@docker-manager-1 ~]# cd /exporter/

[root@docker-manager-1 exporter]# ls

[root@docker-manager-1 exporter]# ls

node_exporter-1.2.2.linux-amd64.tar.gz

[root@docker-manager-1 exporter]#

2.運行exporter

命令：nohup ./node_exporter --web.listen-address="0.0.0.0:9100" &

[root@docker-manager-1 node_exporter-1.2.2.linux-amd64]# nohup ./node_exporter --web.listen-address="0.0.0.0:9100" &

[1] 120539

[root@docker-manager-1 node_exporter-1.2.2.linux-amd64]# nohup: 忽略輸入并把輸出追加到'nohup.out'

[root@docker-manager-1 node_exporter-1.2.2.linux-amd64]# ps aux|grep exporter

root      120539  0.0  0.6 716436 11888 pts/0    Sl   11:23   0:00 ./node_exporter --web.listen-address=0.0.0.0:9100

root      120551  0.0  0.0  12324   996 pts/0    S+   11:23   0:00 grep --color=auto exporter

[root@docker-manager-1 node_exporter-1.2.2.linux-amd64]#

3.在Prometheus-server上面修改Prometheus.yml檔案

[root@prometheus-server prometheus-2.29.1.linux-amd64]# vim prometheus.yml

[root@prometheus-server prometheus-2.29.1.linux-amd64]# cat prometheus.yml

# my global config

global:

  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).



# Alertmanager configuration

alerting:

  alertmanagers:

    - static_configs:

        - targets:

          # - alertmanager:9093



# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

  # - "first_rules.yml"

  # - "second_rules.yml"



# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: "prometheus"



    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.



    static_configs:

      - targets: ["localhost:9090"]

  - job_name: "swarm-manager"

   

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

    static_configs:

      - targets: ["192.168.232.132:9100"]

4.重啟Prometheus服務

[root@prometheus-server prometheus-2.29.1.linux-amd64]# ps aux|grep prome

root        1677  0.0  5.6 1044504 44944 ?       Sl   05:49   0:18 ./prometheus --config.file=prometheus.yml

root        2552  0.0  0.1  12344  1076 pts/1    R+   11:27   0:00 grep --color=auto prome

[root@prometheus-server prometheus-2.29.1.linux-amd64]# kill -9 1677

[root@prometheus-server prometheus-2.29.1.linux-amd64]# ps aux|grep prome

root        2556  0.0  0.1  12344  1196 pts/1    R+   11:28   0:00 grep --color=auto prome

[root@prometheus-server prometheus-2.29.1.linux-amd64]# nohup ./prometheus --config.file=prometheus.yml &

[1] 2640

[root@prometheus-server prometheus-2.29.1.linux-amd64]# nohup: 忽略輸入并把輸出追加到'nohup.out'

^C

[root@prometheus-server prometheus-2.29.1.linux-amd64]# ps aux|grep prom

root        2640  4.0 11.5 782384 92084 pts/1    Sl   11:38   0:00 ./prometheus --config.file=prometheus.yml

root        2648  0.0  0.1  12344  1044 pts/1    S+   11:38   0:00 grep --color=auto prom

5.訪問Prometheus

其他三個工具節點操作一致，在這里不再贅述

接下來在Prometheus server進行grafana的安裝部署

1.創建grafana.repo檔案

[root@prometheus-server yum.repos.d]# vim grafana.repo

[root@prometheus-server yum.repos.d]# cat grafana.repo

[grafana]

name=grafana

baseurl=https://packages.grafana.com/enterprise/rpm

repo_gpgcheck=1

enabled=1

gpgcheck=1

gpgkey=https://packages.grafana.com/gpg.key

sslverify=1

sslcacert=/etc/pki/tls/certs/ca-bundle.crt

[root@prometheus-server yum.repos.d]#

2.安裝grafana

[root@prometheus-server yum.repos.d]# yum install grafana -y

3.運行grafana server

[root@prometheus-server yum.repos.d]# systemctl start grafana-server

[root@prometheus-server yum.repos.d]#

[root@prometheus-server yum.repos.d]# ps aux|grep grafana

root        3019  0.0  0.1 169472   800 ?        Ss   14:34   0:00 gpg-agent --homedir /var/cache/dnf/grafana-ee12c6ab2813e349/pubring --use-standard-socket --daemon

grafana     3553  4.0 10.0 1302412 80140 ?       Ssl  14:36   0:01 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning

root        3563  0.0  0.1  12344  1192 pts/4    R+   14:37   0:00 grep --color=auto grafana

[root@prometheus-server yum.repos.d]#

[root@prometheus-server yum.repos.d]# ss -anplut|grep grafana

tcp     LISTEN   0        128                    *:3000                *:*       users:(("grafana-server",pid=3553,fd=8))                                      

[root@prometheus-server yum.repos.d]#

[root@prometheus-server yum.repos.d]# grafana-server -v

Version 8.1.2 (commit: 103f8fa094, branch: HEAD)

[root@prometheus-server yum.repos.d]#

4.訪問grafana

默認賬號和密碼都是admin