在 K8s 集群上部署 RabbitMQ 實戰

作者：老Z，云原生愛好者，目前專注于云原生運維，KubeSphere Ambassador，

前言

知識點

• 定級：入門級
• RabbitMQ 單節點安裝部署
• RabbitMQ 集群安裝部署
• GitOps 運維思想

演示服務器配置

演示環境涉及軟體版本資訊

• 作業系統：CentOS-7.9-x86_64
• Ansible：2.8.20
• KubeSphere：3.3.0
• Kubernetes：v1.24.1
• Rook：v1.9.7
• Ceph： v16.2.9
• GlusterFS：9.5.1
• ElasticSearch：7.17.5
• Harbor：2.5.1
• RabbitMQ：3.9.22
• RabbitMQ Cluster Operator：1.14.0

簡介

RabbitMQ 單節點如何在 K8s 集群上部署？RabbitMQ 集群如何在 K8s 集群上部署？60分鐘帶你實戰入門，

單節點 RabbitMQ 部署

思路梳理

• StatefulSet
• Headless Service：內部服務用
• External Service：外部管理用
• Secrets：管理用戶名和密碼
• Image：DockerHub 官方提供的 rabbitmq:3.9.22-management(帶 management 插件)

準備離線鏡像

此程序為可選項，離線內網環境可用，如果不配置內網鏡像，后續的資源配置清單中注意更改容器的 image 為默認值，

在一臺能同時訪問互聯網和內網 Harbor 倉庫的服務器上進行下面的操作，

• 下載鏡像

docker pull rabbitmq:3.9.22-management

• 重新打 tag

docker tag rabbitmq:3.9.22-management registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

• 推送到私有鏡像倉庫

docker push registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

• 清理臨時鏡像

docker rmi rabbitmq:3.9.22-management
docker rmi registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

資源配置清單

• rabbitmq-secret.yaml

---
kind: Secret
apiVersion: v1
metadata:
  name: rabbitmq-secret
  namespace: zdevops
data:
  pass: UEA4OHcwcmQ=
  user: YWRtaW4=
type: Opaque

• rabbitmq-sts.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: zdevops
  labels:
    app: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rabbitmq
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rabbitmq
          image: 'registry.zdevops.com.cn/library/rabbitmq:3.9.22-management'
          ports:
            - name: tcp-5672
              containerPort: 5672
              protocol: TCP
            - name: http-15672
              containerPort: 15672
              protocol: TCP
          env:
            - name: RABBITMQ_DEFAULT_USER
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: user
            - name: RABBITMQ_DEFAULT_PASS
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: pass
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 100m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
  serviceName: rabbitmq-headless

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-headless
  namespace: zdevops
  labels:
    app: rabbitmq
spec:
  ports:
    - name: tcp-rabbitmq-5672
      protocol: TCP
      port: 5672
      targetPort: 5672
  selector:
    app: rabbitmq
  clusterIP: None
  type: ClusterIP

• rabbitmq-external.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-external
  namespace: zdevops
  labels:
    app: rabbitmq-external
spec:
  ports:
    - name: http-rabbitmq-external
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app: rabbitmq
  type: NodePort

GitOps

在運維開發服務器上操作

# 在已有代碼倉庫創建 rabbitmq/single 目錄
[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/single

# 編輯資源配置清單
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-secret.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-sts.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-external.yaml

# 提交 Git
[root@zdevops-master k8s-yaml]# git add rabbitmq
[root@zdevops-master k8s-yaml]# git commit -am '添加rabbitmq 單節點資源配置清單'
[root@zdevops-master k8s-yaml]# git push

部署資源

在運維管理服務器上操作

• 更新鏡像倉庫代碼

[root@zdevops-master k8s-yaml]# git pull

• 部署資源

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/single/

驗證

• Secret

[root@zdevops-master k8s-yaml]# kubectl get secret -n zdevops
NAME              TYPE     DATA   AGE
rabbitmq-secret   Opaque   2      8s

• StatefulSet

[root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevops
NAME       READY   AGE   CONTAINERS   IMAGES
rabbitmq   1/1     25s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

• Pods

[root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevops
NAME         READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
rabbitmq-0   1/1     Running   0          26s   10.233.87.13   ks-k8s-master-1   <none>           <none>

• Service

[root@zdevops-master k8s-yaml]# kubectl get svc -o wide -n zdevops
NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE   SELECTOR
rabbitmq-external   NodePort    10.233.4.224   <none>        15672:31672/TCP   36s   app=rabbitmq
rabbitmq-headless   ClusterIP   None           <none>        5672/TCP          36s   app=rabbitmq

• 圖形化管理界面

清理資源

• 清理 StatefulSet

[root@zdevops-master k8s-yaml]# kubectl delete sts rabbitmq -n zdevops

• 清理服務

[root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-external rabbitmq-headless -n zdevops

集群模式 RabbitMQ 部署

思路梳理

使用官方提供的 RabbitMQ Cluster Operator for Kubernetes，

Open source RabbitMQ Cluster Kubernetes Operator by VMware，

官方檔案快速部署

官方提供了快速部署的例子，只需要兩步，這里我們僅作參考，后面我們參考內網離線鏡像部署的方案，

• 部署 RabbitMQ Cluster Operator

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

• 部署 RabbitMQ Cluster

kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

準備離線鏡像

此程序為可選項，離線內網環境可用，如果不配置內網鏡像，后續的資源配置清單中注意更改容器的 image 為默認值，

在一臺能同時訪問互聯網和內網 Harbor 倉庫的服務器上進行下面的操作，

• 在 Harbor 中創建專案

curl -u "admin:Harbor12345" -X POST -H "Content-Type: application/json" https://registry.zdevops.com.cn/api/v2.0/projects -d '{ "project_name": "rabbitmqoperator", "public": true}'

• 下載鏡像

docker pull rabbitmqoperator/cluster-operator:1.14.0

• 重新打 tag

docker tag rabbitmqoperator/cluster-operator:1.14.0 registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

• 推送到私有鏡像倉庫

docker push registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

• 清理臨時鏡像

docker rmi rabbitmqoperator/cluster-operator:1.14.0
docker rmi registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

資源配置清單

• 通過官網獲取 RabbitMQ Cluster Operator 部署資源配置清單「cluster-operator.yml」

wget https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

• 修改 RabbitMQ Cluster Operator image 為內網鏡像

sed -i 's#rabbitmqoperator#registry.zdevops.com.cn/rabbitmqoperator#g' cluster-operator.yml

• RabbitMQ Cluster 部署資源清單「rabbitmq-cluster.yaml」

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  namespace: zdevops
  name: rabbitmq-cluster
  labels:
    app: rabbitmq-cluster
spec:
  replicas: 3
  image: registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  resources:
    limits:
      cpu: 2
      memory: 4Gi
    requests:
      cpu: 100m
      memory: 500Mi
  rabbitmq:
    additionalConfig: |
      default_user=admin
      default_pass=P@88w0rd

更多配置引數和配置示例，請參考 官方檔案

• 管理頁面的外部訪問服務 rabbitmq-cluster-external.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-cluster-external
  namespace: zdevops
  labels:
    app: rabbitmq-cluster-external
spec:
  ports:
    - name: management
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app.kubernetes.io/name: rabbitmq-cluster
  type: NodePort

GitOps

在運維開發服務器上操作

# 在已有代碼倉庫創建 rabbitmq/cluster 目錄
[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/cluster

# 編輯資源配置清單
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/cluster-operator.yml
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster-external.yaml

# 提交 Git
[root@zdevops-master k8s-yaml]# git add rabbitmq/cluster
[root@zdevops-master k8s-yaml]# git commit -am '添加 rabbitmq 集群模式部署資源配置清單'
[root@zdevops-master k8s-yaml]# git push

部署資源

在運維管理服務器上操作

• `更新鏡像倉庫代碼

[root@zdevops-master k8s-yaml]# git pull

• 部署 RabbitMQ Cluster Operator

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/cluster-operator.yml

• 部署 RabbitMQ Cluster

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster.yaml

• 部署管理頁面外部訪問服務

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster-external.yaml

驗證

• RabbitMQ Cluster Operator Deployment

[root@zdevops-master k8s-yaml]# kubectl get deployments -n rabbitmq-system -o wide
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                             SELECTOR
rabbitmq-cluster-operator   1/1     1            1           107m   operator     registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0   app.kubernetes.io/name=rabbitmq-cluster-operator

• RabbitmqClusters

[root@zdevops-master k8s-yaml]# kubectl get rabbitmqclusters -n zdevops
NAME               ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmq-cluster   False              Unknown            23s

• StatefulSet

[root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevops
NAME                      READY   AGE   CONTAINERS   IMAGES
rabbitmq-cluster-server   3/3     74s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

• Pods

[root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevops
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE   READINESS GATES
rabbitmq-cluster-server-0   1/1     Running   0          84s   10.233.116.26   ks-k8s-master-2   <none>           <none>
rabbitmq-cluster-server-1   1/1     Running   0          83s   10.233.117.28   ks-k8s-master-0   <none>           <none>
rabbitmq-cluster-server-2   1/1     Running   0          82s   10.233.87.31    ks-k8s-master-1   <none>           <none>

• Services

[root@zdevops-master k8s-yaml]# kubectl get svc -n zdevops -o wide
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE    SELECTOR
rabbitmq-cluster            ClusterIP   10.233.56.153   <none>        15692/TCP,5672/TCP,15672/TCP   107s   app.kubernetes.io/name=rabbitmq-cluster
rabbitmq-cluster-external   NodePort    10.233.63.84    <none>        15672:31672/TCP                74m    app.kubernetes.io/name=rabbitmq-cluster
rabbitmq-cluster-nodes      ClusterIP   None            <none>        4369/TCP,25672/TCP             107s   app.kubernetes.io/name=rabbitmq-cluster

• 圖形化管理界面

從管理界面中可以看到一個三節點的集群

清理資源

• 清理 RabbitmqClusters

[root@zdevops-master k8s-yaml]# kubectl delete rabbitmqclusters rabbitmq-cluster -n zdevops

• 清理管理頁面外部服務

[root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-cluster-external -n zdevops

結束語

本系列檔案是我在云原生技術領域的學習和運維實踐的手記，用輸出倒逼輸入是一種高效的學習方法，能夠快速積累經驗和提高技術，只有把學到的知識寫出來并能夠讓其他人理解，才能說明真正掌握了這項知識，

本系列檔案內容涵蓋 (但不限于) 以下技術領域：

• KubeSphere
• Kubernetes
• Ansible
• 自動化運維
• CNCF 技術堆疊

如果你喜歡本文，請分享給你的小伙伴！

Get 檔案

• Github：https://github.com/devops/z-notes
• Gitee：https://gitee.com/zdevops/z-notes
• 知乎：https://www.zhihu.com/people/zdevops/

Get 代碼

• Github：https://github.com/devops/ansible-zdevops
• Gitee：https://gitee.com/zdevops/ansible-zdevops

Get 視頻 B 站

• 老 Z 手記 https://space.bilibili.com/1039301316

本文由博客一文多發平臺 OpenWrite 發布！

標籤：其他

上一篇：KubeSphere 網關的設計與實作（解讀）

下一篇：[機器學習]-分類問題常用評價指標、混淆矩陣及ROC曲線繪制方法-python實作