KubeSphere 3.3.0 離線安裝教程

作者：老Z，中電信數智科技有限公司山東分公司運維架構師，云原生愛好者，目前專注于云原生運維，云原生領域技術堆疊涉及Kubernetes、KubeSphere、DevOps、OpenStack、Ansible等，

KubeKey 是一個用于部署 K8s 集群的開源輕量級工具，

它提供了一種靈活、快速、便捷的方式來僅安裝 Kubernetes/K3s，或同時安裝 K8s/K3s 和 KubeSphere，以及其他云原生插件，除此之外，它也是擴展和升級集群的有效工具，

KubeKey v2.1.0 版本新增了清單 (manifest) 和制品 (artifact) 的概念，為用戶離線部署 K8s 集群提供了一種解決方案，

manifest 是一個描述當前 K8s 集群資訊和定義 artifact 制品中需要包含哪些內容的文本檔案，

在過去，用戶需要準備部署工具，鏡像 tar 包和其他相關的二進制檔案，每位用戶需要部署的 K8s 版本和需要部署的鏡像都是不同的，現在使用 KubeKey，用戶只需使用清單 manifest 檔案來定義將要離線部署的集群環境需要的內容，再通過該 manifest 來匯出制品 artifact 檔案即可完成準備作業，離線部署時只需要 KubeKey 和 artifact 就可快速、簡單的在環境中部署鏡像倉庫和 K8s 集群，

KubeKey 生成 manifest 檔案有兩種方式，

• 利用現有運行中的集群作為源生成 manifest 檔案，也是官方推薦的一種方式，具體參考 KubeSphere 官網的離線部署檔案，
• 根據 模板檔案 手動撰寫 manifest 檔案，

第一種方式的好處是可以構建 1:1 的運行環境，但是需要提前部署一個集群，不夠靈活度，并不是所有人都具備這種條件的，

因此，本文參考官方的離線檔案，采用手寫 manifest 檔案的方式，實作離線環境的安裝部署，

本文知識點

• 定級：入門級
• 了解清單 (manifest) 和制品 (artifact) 的概念
• 掌握 manifest 清單的撰寫方法
• 根據 manifest 清單制作 artifact
• 離線部署 KubeSphere 和 Kubernetes

演示服務器配置

演示環境涉及軟體版本資訊

• 作業系統：CentOS-7.9-x86_64

• KubeSphere：3.3.0

• Kubernetes：1.24.1

• Kubekey：v2.2.1

• Ansible：2.8.20

• Harbor：2.5.1

離線部署資源制作

下載 KubeKey

# 在zdevops-master 運維開發服務器執行

# 選擇中文區下載(訪問github受限時使用)
$ export KKZONE=cn

# 下載KubeKey
$ mkdir /data/kubekey
$ cd /data/kubekey/
$ curl -sfL https://get-kk.kubesphere.io | VERSION=v2.2.1 sh -

獲取 manifest 模板

參考 https://github.com/kubesphere/kubekey/blob/master/docs/manifest-example.md

有兩個參考用例，一個簡單版，一個完整版，參考簡單版就可以，

獲取 ks-installer images-list

$ wget https://github.com/kubesphere/ks-installer/releases/download/v3.3.0/images-list.txt

文中的 image 串列選用的 Docker Hub 倉庫其他組件存放的公共倉庫，國內建議統一更改前綴為 registry.cn-beijing.aliyuncs.com/kubesphereio

修改后的完整的鏡像串列在下面的 manifest 檔案中展示，

請注意，example-images 包含的 image 中只保留了 busybox，其他的在本文中沒有使用，

獲取作業系統依賴包

$ wget https://github.com/kubesphere/kubekey/releases/download/v2.2.1/centos7-rpms-amd64.iso

將該 ISO 檔案放到制作離線鏡像的服務器的 /data/kubekey 目錄下

生成 manifest 檔案

根據上面的檔案及相關資訊，生成最終 manifest.yaml，

命名為 ks-v3.3.0-manifest.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems:
  - arch: amd64
    type: linux
    id: centos
    version: "7"
    osImage: CentOS Linux 7 (Core)
    repository:
      iso:
        localPath: "/data/kubekey/centos7-rpms-amd64.iso"
        url:
  kubernetesDistributions:
  - type: kubernetes
    version: v1.24.1
  components:
    helm: 
      version: v3.6.3
    cni: 
      version: v0.9.1
    etcd: 
      version: v3.4.13
    containerRuntimes:
    - type: containerd
      version: 1.6.4
    crictl: 
      version: v1.24.0
    docker-registry:
      version: "2"
    harbor:
      version: v2.4.1
    docker-compose:
      version: v2.2.2
  images:
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.23.7
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.23.7
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.7
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.23.7
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.24.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.24.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.24.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.24.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.10
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.21.13
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.21.13
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.21.13
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.21.13
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.7
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.12.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:2.10.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:2.10.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nfs-subdir-external-provisioner:v4.0.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.21.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubefed:v0.8.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tower:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx-ingress-controller:v1.1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:5.0.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.0.25-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openldap:1.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/netshoot:v1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cloudcore:v1.9.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/iptables-manager:v1.9.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/edgeservice:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/gatekeeper:v3.5.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-apiserver:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-controller:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-tools:v3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-jenkins:v3.3.0-2.319.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/inbound-agent:4.10-2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2ioperator:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2irun:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2i-binary:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-6-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-4-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-36-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-35-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-34-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-27-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd:v2.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset:v0.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/dex:v2.30.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:6.2.6-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.5.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.34.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.25.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/grafana:8.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v1.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v1.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-curator:v5.7.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-oss:6.8.22
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.13.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.8.11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:1.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/filebeat:6.7.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-operator:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-exporter:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-ruler:v0.4.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-operator:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-webhook:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pilot:1.11.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.11.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-operator:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-agent:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-collector:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-query:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-es-index-cleaner:1.27
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali-operator:v1.38.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali:v1.38
  - registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:1.31.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/scope:1.13.0
  registry:
    auths: {}

manifest 修改說明

• 開啟 harbor 和 docker-compose 配置項，為后面通過 KubeKey 自建 harbor 倉庫推送鏡像使用，
• 默認創建的 manifest 里面的鏡像串列從 docker.io 獲取，替換前綴為 registry.cn-beijing.aliyuncs.com/kubesphereio，
• 若需要匯出的 artifact 檔案中包含作業系統依賴檔案（如：conntarck、chrony 等），可在 operationSystem 元素中的 .repostiory.iso.url 中配置相應的 ISO 依賴檔案下載地址為 localPath ，填寫提前下載好的 ISO 包在本地的存放路徑，并將 url 配置項置空，
• 您可以訪問 https://github.com/kubesphere/kubekey/releases/tag/v2.2.1 下載 ISO 檔案，

匯出制品 artifact

$ export KKZONE=cn

$ ./kk artifact export -m ks-v3.3.0-manifest.yaml -o kubesphere-v3.3.0-artifact.tar.gz

制品 (artifact) 說明

• 制品（artifact）是一個根據指定的 manifest 檔案內容匯出的包含鏡像 tar 包和相關二進制檔案的 tgz 包，
• 在 KubeKey 初始化鏡像倉庫、創建集群、添加節點和升級集群的命令中均可指定一個 artifact，KubeKey 將自動解包該 artifact 并在執行命令時直接使用解包出來的檔案，

匯出 Kubekey

$ tar zcvf kubekey-v2.2.1.tar.gz kk kubekey-v2.2.1-linux-amd64.tar.gz

K8s 服務器初始化配置

本節執行離線環境 K8s 服務器初始化配置，

Ansible hosts 配置

[k8s]
ks-k8s-master-0 ansible_ssh_host=192.168.9.91  host_name=ks-k8s-master-0
ks-k8s-master-1 ansible_ssh_host=192.168.9.92  host_name=ks-k8s-master-1
ks-k8s-master-2 ansible_ssh_host=192.168.9.93  host_name=ks-k8s-master-2

[es]
es-node-0 ansible_ssh_host=192.168.9.95 host_name=es-node-0
es-node-1 ansible_ssh_host=192.168.9.96 host_name=es-node-1
es-node-2 ansible_ssh_host=192.168.9.97 host_name=es-node-2

harbor ansible_ssh_host=192.168.9.89 host_name=harbor

[servers:children]
k8s
es

[servers:vars]
ansible_connection=paramiko
ansible_ssh_user=root
ansible_ssh_pass=F@ywwpTj4bJtYwzpwCqD

檢測服務器連通性

# 利用 ansible 檢測服務器的連通性

$ cd /data/ansible/ansible-zdevops/inventories/dev/
$ source /opt/ansible2.8/bin/activate
$ ansible -m ping all

初始化服務器配置

# 利用 ansible-playbook 初始化服務器配置

$ ansible-playbook ../../playbooks/init-base.yaml -l k8s

掛載資料盤

• 掛載第一塊資料盤

# 利用 ansible-playbook 初始化主機資料盤
# 注意 -e data_disk_path="/data" 指定掛載目錄, 用于存盤 Docker 容器資料

$ ansible-playbook ../../playbooks/init-disk.yaml -e data_disk_path="/data" -l k8s

• 掛載驗證

# 利用 ansible 驗證資料盤是否格式化并掛載
$ ansible harbor -m shell -a 'df -h'

# 利用 ansible 驗證資料盤是否配置自動掛載

$ ansible harbor -m shell -a 'tail -1  /etc/fstab'

安裝 K8s 系統依賴包

# 利用 ansible-playbook 安裝 kubernetes 系統依賴包
# ansible-playbook 中設定了啟用 GlusterFS 存盤的開關，默認開啟,不需要的可以將引數設定為 False

$ ansible-playbook ../../playbooks/deploy-kubesphere.yaml -e k8s_storage_glusterfs=false -l k8s

離線安裝集群

傳輸離線部署資源到部署節點

將以下離線部署資源，傳到部署節點 (通常是第一個 master 節點) 的 /data/kubekey 目錄，

• Kubekey：kubekey-v2.2.1.tar.gz
• 制品 artifact：kubesphere-v3.3.0-artifact.tar.gz

執行以下操作，解壓 kubekey，

$ cd /data/kubekey
$ tar xvf kubekey-v2.2.1.tar.gz

創建離線集群組態檔

• 創建組態檔

$ ./kk create config --with-kubesphere v3.3.0 --with-kubernetes v1.24.1 -f config-sample.yaml

• 修改組態檔

$ vim config-sample.yaml

修改內容說明

• 按照實際離線環境配置修改節點資訊，
• 按實際情況添加 registry 的相關資訊，

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: ks-k8s-master-0, address: 192.168.9.91, internalAddress: 192.168.9.91, user: root, password: "F@ywwpTj4bJtYwzpwCqD"}
  - {name: ks-k8s-master-1, address: 192.168.9.92, internalAddress: 192.168.9.92, user: root, password: "F@ywwpTj4bJtYwzpwCqD"}
  - {name: ks-k8s-master-2, address: 192.168.9.93, internalAddress: 192.168.9.93, user: root, password: "F@ywwpTj4bJtYwzpwCqD"}
  roleGroups:
    etcd:
    - ks-k8s-master-0
    - ks-k8s-master-1
    - ks-k8s-master-2
    control-plane: 
    - ks-k8s-master-0
    - ks-k8s-master-1
    - ks-k8s-master-2
    worker:
    - ks-k8s-master-0
    - ks-k8s-master-1
    - ks-k8s-master-2
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    internalLoadbalancer: haproxy

    domain: lb.zdevops.com.cn
    address: ""
    port: 6443
  kubernetes:
    version: v1.24.1
    clusterName: zdevops.com.cn
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    type: "harbor"
    auths:
      "registry.zdevops.com.cn":
         username: admin
         password: Harbor12345
    privateRegistry: "registry.zdevops.com.cn"
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: []
  addons: []

# 下面的內容不修改，不做展示

在 Harbor 中創建專案

本文采用提前部署好的 Harbor 來存放鏡像，部署程序參考我之前寫的 基于 KubeSphere 玩轉 k8s-Harbor 安裝手記，

你可以使用 kk 工具自動部署 Harbor，具體參考官方離線部署檔案，

• 下載創建專案腳本模板

$ curl -O https://raw.githubusercontent.com/kubesphere/ks-installer/master/scripts/create_project_harbor.sh

• 根據實際情況修改專案腳本

#!/usr/bin/env bash

# Harbor 倉庫地址
url="https://registry.zdevops.com.cn"

# 訪問 Harbor 倉庫用戶
user="admin"

# 訪問 Harbor 倉庫用戶密碼
passwd="Harbor12345"

# 需要創建的專案名串列，正常只需要創建一個**kubesphereio**即可，這里為了保留變數可擴展性多寫了兩個，
harbor_projects=(library
    kubesphereio
    kubesphere
)

for project in "${harbor_projects[@]}"; do
    echo "creating $project"
    curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}"
done

• 執行腳本創建專案

$ sh create_project_harbor.sh

推送離線鏡像到 Harbor 倉庫

將提前準備好的離線鏡像推送到 Harbor 倉庫，這一步為可選項，因為創建集群的時候也會再次推送鏡像，為了部署一次成功率，建議先推送，

$ ./kk artifact image push -f config-sample.yaml -a  kubesphere-v3.3.0-artifact.tar.gz

創建集群并安裝 OS 依賴

$ ./kk create cluster -f config-sample.yaml -a kubesphere-v3.3.0-artifact.tar.gz --with-packages

引數說明

• config-sample.yaml：離線環境集群的組態檔，
• kubesphere-v3.3.0-artifact.tar.gz：制品包的 tar 包鏡像，
• --with-packages：若需要安裝作業系統依賴，需指定該選項，

查看集群狀態

$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

正確安裝完成后，您會看到以下內容：

**************************************************
Collecting installation results ...
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://192.168.9.91:30880
Account: admin
Password: P@88w0rd

NOTES：
  1. After you log into the console, please check the
     monitoring status of service components in
     "Cluster Management". If any service is not
     ready, please wait patiently until all components 
     are up and running.
  2. Please change the default password after login.

#####################################################
https://kubesphere.io             2022-06-30 14:30:19
#####################################################

登錄 Web 控制臺

通過 http://{IP}:30880 使用默認帳戶和密碼 admin/P@88w0rd 訪問 KubeSphere 的 Web 控制臺，進行后續的操作配置，

總結

感謝您完整的閱讀完本文，為此，您應該 Get 到了以下技能

• 了解了清單 (manifest) 和制品 (artifact) 的概念
• 了解 manifest 和 image 資源的獲取地址
• 手寫 manifest 清單
• 根據 manifest 清單制作 artifact
• 離線部署 KubeSphere 和 Kubernetes
• Harbor 鏡像倉庫自動創建專案
• Ansible 使用的小技巧

目前為止，我們已經完成了最小化環境的 KubeSphere 和 K8s 集群的部署，但是，這僅僅是一個開始，后續還有很多配置和使用技巧，敬請期待 ...

本文由博客一文多發平臺 OpenWrite 發布！

標籤：其他

上一篇：超越 API ，對話 Eolink 創始人劉昊臻

下一篇：Eolink 和 JMeter 介面測驗對比