本章節通過在Jenkins創建一個kubernetes云環境，動態的在kubernetes集群中創建pod完成pipeline的構建流程，關于直接在宿主機上搭建Jenkins集群的可參照Kubernetes CI／CD（1）
部署Jenkins
-
下載Jenkins對應的鏡像
docker pull jenkins/jenkins:2.221 -
將jenkins鏡像上傳到自己的私有鏡像倉庫中
docker tag jenkins/jenkins:2.221 192.168.0.107/k8s/jenkins:2.221 docker push 192.168.0.107/k8s/jenkins:2.221 -
撰寫啟動Jenkins的yml檔案
cat > jenkins.yml << EOF kind: PersistentVolume apiVersion: v1 metadata: name: jenkins labels: type: local app: jenkins spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: /opt/k8s/yml/jenkins/data --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: jenkins-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi --- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: default automountServiceAccountToken: true --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: Jenkins-cluster-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: jenkins namespace: default --- apiVersion: v1 kind: Service metadata: name: jenkins labels: app: jenkins spec: ports: - port: 80 targetPort: 8080 nodePort: 8888 name: jenkins - port: 50000 targetPort: 50000 nodePort: 50000 name: agent selector: app: jenkins tier: jenkins type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: name: jenkins labels: app: jenkins spec: strategy: type: Recreate selector: matchLabels: app: jenkins tier: jenkins template: metadata: labels: app: jenkins tier: jenkins spec: serviceAccountName: jenkins containers: - image: 192.168.0.107/k8s/jenkins:2.221 imagePullPolicy: IfNotPresent name: jenkins securityContext: privileged: true runAsUser: 0 volumeMounts: - name: kubeconfig mountPath: /var/jenkins_home/.kube - name: docker mountPath: /var/run/docker.sock - name: docker-bin mountPath: /usr/bin/docker - name: jenkins-persistent-storage mountPath: /var/jenkins_home ports: - containerPort: 8080 name: jenkins - containerPort: 50000 name: agent volumes: - name: kubeconfig emptyDir: {} - name: docker hostPath: path: /var/run/docker.sock - name: docker-bin hostPath: path: /opt/k8s/bin/docker - name: jenkins-persistent-storage persistentVolumeClaim: claimName: jenkins-claim EOF
安裝 kubernetes相關插件
kubernetes-cd
kubernetes-client-api
kubernetes-credentials
kubernetes
配置kubernetes云（配置詳情官方網站kubernetes-plugin)
-
新加一個Cloud
在Jenkins界面執行Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds -> Add a new Cloud
-
配置cloud，點擊Kubernetes Cloud details
- Kubernetes 地址：指定要連接的k8s集群API地址，因為我們master是在k8s集群中啟動的，所以此處可以直接用https://kubernetes.default.svc.cluster.local，其中kubernetes是k8s集群給我們啟動的一個service，內部會把對他的訪問轉發給API server，如果Jenkins不在k8s集群中，或者想要啟動的構建pod和master不是一個集群，這個地方就需要相應的k8s集群地址
- 憑據：訪問k8s集群的認證憑證，我們啟動Jenkins集群時同時創建了Service account，并賦給了Jenkins容器，所以這個地方可以直接創建一個service count型別的憑據,如果是訪問其他集群，需要用服務證書key來配置
配置好后點擊:連接測驗,正常的話會出現Connection test successful的提示
-
配置Jenkins相關資訊（主要是agent和master通信的資訊）
- Jenkins 地址： 連接jenkins master的地址，因為我們jenkins對應的service啟動節點是80，所以這個地方就去掉了埠號，并且service也啟動了50000埠映射到容器的50000，所以直接配置成http://jenkins
構建一個簡單的流水線驗證cloud的構建功能
-
在Jenkins界面新建一個item，名稱hello-pipeline-cloud， 型別選擇:流水線(pipeline)
-
編輯pipeline部分
podTemplate(cloud: "kubernetes") { node(POD_LABEL) { stage('Run shell') { sh 'echo hello world' } } }- cloud: "kubernetes"，指定執行的云環境，默認是kubernetes，所以這個地方可以省略，當有多個云環境或者我們創建的cloud名稱不是kubernetes則需要明確指定
- POD_LABEL 是在1.17.0版本后引入的一個新特性，可以自動對創建的pod進行打標簽
-
執行構建
首先我們可以在要執行的k8s集群上執行如下命令觀察執行構建程序中k8s云給我做了什么事
kubectl get pod -w在Jenkins界面選擇剛創建的工程，點擊 Build Now
對應的集群的輸出
root@master:/opt/k8s/yml/jenkins# kubectl get pod -w NAME READY STATUS RESTARTS AGE jenkins-68d8b54c45-gshvp 1/1 Running 0 60m hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 0/1 Pending 0 0s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 0/1 Pending 0 0s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 0/1 ContainerCreating 0 0s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 1/1 Running 0 1s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 1/1 Terminating 0 7s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 0/1 Terminating 0 8s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 0/1 Terminating 0 9s hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v 0/1 Terminating 0 9s可以看到k8s集群給我們創建了一個新的pod：hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v，構建完成后會自動把這個pod停掉
Jenkins構建的日志
Started by user admin Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] podTemplate [Pipeline] { [Pipeline] node Still waiting to schedule task All nodes of label ‘hello-pipeline-cloud_9-7n3c4’ are offline Created Pod: hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v in namespace default Agent hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v is provisioned from template hello-pipeline-cloud_9-7n3c4-8stzj --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins/job/hello-pipeline-cloud/9/" runUrl: "job/hello-pipeline-cloud/9/" labels: jenkins: "slave" jenkins/label: "hello-pipeline-cloud_9-7n3c4" name: "hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v" spec: containers: - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_AGENT_NAME" value: "hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v" - name: "JENKINS_NAME" value: "hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins/" image: "jenkins/jnlp-slave:3.35-5-alpine" name: "jnlp" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false nodeSelector: beta.kubernetes.io/os: "linux" restartPolicy: "Never" securityContext: {} volumes: - emptyDir: medium: "" name: "workspace-volume" Running on hello-pipeline-cloud-9-7n3c4-8stzj-n8g6v in /home/jenkins/agent/workspace/hello-pipeline-cloud [Pipeline] { [Pipeline] stage [Pipeline] { (Run shell) [Pipeline] sh + echo hello world hello world [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS從日志中可以看到k8s集群采用默認的pod模版給我啟動了一個pod，并且把我們的構建內容sh 'echo hello world'在pod對應的容器中執行了
如果默認的模版不能滿足我們的要求（比如在內網作業，對應的image想用我們自己私有倉庫中的鏡像可以重寫name是jnlp的容器模版）后面一個例子，我們會重寫這個模版，并在我們自定義的容器中執行我們的構建
構建自定義podtemplage
-
在Jenkins界面新建一個item，名稱hello-pipeline-selfpodtemplate， 型別選擇:流水線(pipeline)
-
編輯pipeline部分
podTemplate(yaml: """ apiVersion: v1 kind: Pod metadata: labels: app: busybox spec: containers: - name: busybox image: 192.168.0.107/k8s/busybox:latest command: - cat tty: true """ ,containers: [containerTemplate(name: 'jnlp', image: '192.168.0.107/jenkins/jnlp-slave:3.35-5-alpine')] ,cloud: "kubernetes") { node(POD_LABEL) { container('busybox') { sh "hostname" } } }- yaml 通過這個欄位，我們定義了一個自己的podtemplate，容器名稱是busybox，并在后面使用
- containers的containerTemplate，我們重寫了jnlp容器啟動的鏡像名稱，jnlp鏡像必須啟動，他要和jenkins master通信，來告知構建狀態，并且不能配置其它容器模版的名稱為jnlp，否則會造成構建一直不會結束
- 具體的構建流程，我們通過container('busybox')指定了構建要執行的容器，這個地方我們可以根據要構建的型別，配置maven、gradle、docker等各種構建環境來配置不同的podtemplate以此滿足不同的構建需求，這也是用云環境進行構建的優勢
同樣，我們看下k8s集群的pod創建情況
root@master:/opt/k8s/yml/jenkins# kubectl get pod -w NAME READY STATUS RESTARTS AGE jenkins-68d8b54c45-gshvp 1/1 Running 0 83m hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw 0/2 Pending 0 0s hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw 0/2 Pending 0 0s hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw 0/2 ContainerCreating 0 0s hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw 2/2 Running 0 2s hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw 2/2 Terminating 0 7s看到這次k8s集群創建的pod包含了兩個容器，如果我們describe這個pod可以看到里面是我們指定的容器
Jenkins日志
Started by user admin Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw in namespace default Still waiting to schedule task Waiting for next available executor on ‘hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw’ Agent hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw is provisioned from template hello-pipeline-selfpodtemplate_6-ch89k-0v48m --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins/job/hello-pipeline-selfpodtemplate/6/" runUrl: "job/hello-pipeline-selfpodtemplate/6/" labels: app: "busybox" jenkins: "slave" jenkins/label: "hello-pipeline-selfpodtemplate_6-ch89k" name: "hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw" spec: containers: - command: - "cat" image: "192.168.0.107/k8s/busybox:latest" name: "busybox" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_AGENT_NAME" value: "hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw" - name: "JENKINS_NAME" value: "hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins/" image: "192.168.0.107/jenkins/jnlp-slave:3.35-5-alpine" imagePullPolicy: "IfNotPresent" name: "jnlp" resources: limits: {} requests: {} securityContext: privileged: false tty: false volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false nodeSelector: beta.kubernetes.io/os: "linux" restartPolicy: "Never" securityContext: {} volumes: - emptyDir: medium: "" name: "workspace-volume" Running on hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw in /home/jenkins/agent/workspace/hello-pipeline-selfpodtemplate [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] sh + hostname hello-pipeline-selfpodtemplate-6-ch89k-0v48m-xxsqw [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS- 可以看到jnlp對應的鏡像名稱變成我們指定的鏡像
- pod template 的內容默認是列印出來的，可以通過配置不顯示 podTemplate(showRawYaml:false,...)
遇到問題
-
追加kubernetes service account憑據時報錯
HTTP ERROR 403 No valid crumb was included in the request簡單解決方法，在Configure Global Security配置頁面去掉CSRF Protection
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/41213.html
標籤：其他