前言
上一篇文章 云原生之旅 - 10）手把手教你安裝 Jenkins on Kubernetes 我們介紹了在 Kubernetes 上安裝 Jenkins，本文介紹下如何設定k8s pod作為Jenkins 構建job的 agent，
Jenkins master 和 agent 均以 pod 的形式運行在 Kubernetes 節點上，Master 運行在其中一個節點上，其配置資料 Jenkins home 使用存盤卷掛載，master pod重啟不會導致資料丟失，agent 運行在各個節點上，根據需求動態創建并自動釋放，這樣做的好處很多，比如高可用，高伸縮性，資源利用率高，
關鍵詞：Jenkins on Kubernetes 實踐，Jenkins 和 Kubernetes，在Kubernetes上安裝Jenkins，Jenkins 高可用安裝，Jenkins 動態伸縮構建， Kubernetes Pod as Jenkins build agent
準備
- 已搭建 Jenkins master on kubernetes 云原生之旅 - 10）手把手教你安裝 Jenkins on Kubernetes
- 準備一個 Service Account，對目標 cluster 具有k8s admin權限，以便部署，
- 防火墻已開通 Jenkins 出站到Docker hub，方便 push/pull image
- 防火墻已開通 Jenkins 到 目標 cluster，以便部署，
插件安裝
- Kubernetes Plugin
- Google Kubernetes Engine Plugin (我的例子是部署到 GKE cluster)
Jenkins 配置
Manage Nodes and Clouds
1. Go to `Manage Jenkins` –> `Manage Nodes and Clouds` 2. Click `Configure Clouds` 3. Add a new Cloud select `Kubernetes` 4. Click `Kubernetes Cloud Detail 5. Enter `jenkins` namespace in `Kubernetes Namespace` field 6. Click `Test Connection` --> result show `Connected to Kubernetes v1.22.12-gke.2300` 7. Click `Save` 8. Enter `http://jenkins-service.jenkins.svc.cluster.local:8080` in `Jenkins URL` field 9. Enter `jenkins-agent:50000` in `Jenkins tunnel` field
10. Click `Add Pod Template` then `Pod Template Details`
11. Input `Name`=`jenkins-agent`, `Namespace`=`jenkins`, `Labels`=`kubeagent`
12. (Optional) 如果不添加 container template, the Jenkins Kubernetes plugin will use the default JNLP image from the Docker hub to spin up the agents. 如果你要覆寫默認的jnlp image 可以 Click `Add Container` to add Container Template, 輸入 `Name`=`jnlp`, `Docker Image`=`your_registry/jenkins/inbound-agent:4.11-1-jdk11`
Ensure that you remove the sleep and 9999999 default argument from the container template.
Manage Credentials
- Add `Usernames with password` for docker hub account/pwd，比如 wade_test_dockerhub
- Add `Google Service Account from private key` 比如 gcp_sa_json_key
Credentials 會在Jenkinsfile里面用到，
### 本文首發于博客園 https://www.cnblogs.com/wade-xu/p/16863955.html
Test a freestyle project
Go to Jenkins home –> New Item and create a freestyle project，命名為 quick-test 在 job description 部分, add the label `kubeagent` for `Restrict where this project can be run`.
這個label 和我們上面創建 pod template時用的label一致. 這樣的話 Jenkins就知道用哪個 pod template 作為 agent container.
隨便添加一個shell 作為build steps
點Build Now
查看Console OutputAgent jenkins-agent-l7hw9 is provisioned from template jenkins-agent ...... Building remotely on jenkins-agent-l7hw9 (kubeagent) in workspace /home/jenkins/agent/workspace/quick-test [quick-test] $ /bin/sh -xe /tmp/jenkins17573873264046707236.sh + echo test pipeline test pipeline Finished: SUCCESS
### 本文首發于博客園 https://www.cnblogs.com/wade-xu/p/16863955.html
Jenkinsfile
CI
接著我們用 Jenkinsfile 寫一個 Declarative pipeline - build/push docker image 到docker hub 首先需要定義一個 pod.yaml 作為啟動 agent 的container
kind: Pod
spec:
containers: # list of containers that you want present for your build, you can define a default container in the Jenkinsfile
- name: maven
image: maven:3.5.4-jdk-8-slim
command: ["tail", "-f", "/dev/null"] # this or any command that is bascially a noop is required, this is so that you don't overwrite the entrypoint of the base container
imagePullPolicy: Always # use cache or pull image for agent
resources: # request and limit the resources your build contaienr
requests:
memory: 4Gi
cpu: 2
limits:
memory: 4Gi
cpu: 2
volumeMounts:
- mountPath: /root/.m2 # maven .m2 cache directory
name: maven-home
- name: git
image: bitnami/git:2.38.1
imagePullPolicy: IfNotPresent
command: ["tail", "-f", "/dev/null"]
resources: # limit the resources your build contaienr
limits:
cpu: 100m
memory: 256Mi
- name: kubectl-kustomize
image: line/kubectl-kustomize:1.25.3-4.5.7
imagePullPolicy: IfNotPresent
command: ["tail", "-f", "/dev/null"]
resources: # limit the resources your build contaienr
limits:
cpu: 100m
memory: 256Mi
- name: docker
image: docker:18.06.1
command: ["tail", "-f", "/dev/null"]
imagePullPolicy: Always
volumeMounts:
- name: docker
mountPath: /var/run/docker.sock # We use the k8s host docker engine
volumes:
- name: docker
hostPath:
path: /var/run/docker.sock
- name: maven-home
persistentVolumeClaim:
claimName: maven-repo-storage
build-pod.yaml
在Jenkinsfile里面定義agent 使用這個yaml file
agent {
kubernetes {
idleMinutes 3 // how long the pod will live after no jobs have run on it
yamlFile './build-pod.yaml' // path to the pod definition relative to the root of our project
defaultContainer 'docker' // define a default container if more than a few stages use it, otherwise default to jnlp container
}
下面步驟是 docker login/build/tag/push
environment { DOCKER_HUB_REGISTRY='https://index.docker.io/v1/' DOCKER_HUB_CREDS = credentials('wade_test_dockerhub') }
stage('Build and Push Docker Image') {
steps {
script {
dir(dir_path) {
container('docker') {
// docker login, Using single-quotes instead of double-quotes when referencing these sensitive environment variables prevents this type of leaking.
sh 'echo $DOCKER_HUB_CREDS_PSW | docker login -u $DOCKER_HUB_CREDS_USR --password-stdin $DOCKER_HUB_REGISTRY'
// build image with git tag
sh """
docker build -t $PROJECT_IMAGE_WITH_TAG .
docker tag $PROJECT_IMAGE_WITH_TAG $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
"""
// push image_tag to docker hub
sh """
docker push $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
"""
}
}
}
}
}
我這里沒有選擇用 docker.withRegistry
docker.withRegistry("$DOCKER_HUB_REGISTRY", "$DOCKER_HUB_CREDENTIAL") {}
因為會有不安全的log提示
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
CI + Kustomize + CD
這個例子是上面的 CI 之后 加上 - 利用 Kustomize build K8S resource manifests 然后 CD 到一個 Cluster
Kustomize 可以參考 云原生之旅 - 6）不能錯過的一款 Kubernetes 應用編排管理神器 Kustomize
// assume your k8s manifests in another repo, mine is same repo, just in order to show git clone step
stage('Checkout K8S manifests') {
steps {
script {
dir(dir_path) {
container('git') {
if (! fileExists('learning_by_doing/README.md')) {
sh """
git clone https://github.com/wadexu007/learning_by_doing.git
ls -lhrt
"""
} else {
sh 'echo manifes repo already exist.'
}
}
}
}
}
}
stage('Build manifests with Kustomize') {
steps {
script {
dir(dir_path) {
container('kubectl-kustomize') {
sh """
cd learning_by_doing/Kustomize/demo-manifests/services/demo-app/dev/
kustomize edit set image $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
kustomize build > $WORKSPACE/$dir_path/deployment.yaml
"""
}
}
}
}
}
stage('Deploy to GKE test cluster') {
environment{
PROJECT_ID = 'xperiences-eng-cn-dev'
CLUSTER_NAME = 'xpe-spark-test-gke'
REGION = 'asia-east2'
CREDENTIALS_ID = 'gcp_sa_json_key'
}
steps {
script {
dir(dir_path) {
container('kubectl-kustomize') {
sh """
chown 1000:1000 deployment.yaml
echo start to deploy to cluster $CLUSTER_NAME
"""
step([
$class: 'KubernetesEngineBuilder',
projectId: env.PROJECT_ID,
clusterName: env.CLUSTER_NAME,
location: env.REGION,
manifestPattern: 'deployment.yaml',
credentialsId: env.CREDENTIALS_ID,
verifyDeployments: false])
// verifyDeployments does not work for non-default namespace
}
}
}
}
}
View Code
Pipeline: Input Step
這個例子是利用 Jenkins pipeline的 Input step 來做一個人工介入Approve的步驟， 然后再來一個多cluster 部署，選不同region 部署到不同的cluster的示例， stage('Wait for SRE Approval') {
steps {
timeout(time:72, unit:'HOURS') {
input message: "Approved Prod deployment?", submitter: 'sre-team'
}
}
}
// deployment to multipe k8s clusters
stage('Deploy to GKE Prod cluster') {
environment{
PROJECT_ID = 'sre-cn-dev'
CREDENTIALS_ID = 'gcp_sa_json_key'
CLUSTER_COMMON_NAME = 'demo-gke-prod'
}
steps {
script {
env.REGION = input message: 'Choose which region you want to deploy?',
parameters: [choice(name: 'Region',
description: 'Select Region to Deloy',
choices: ['europe-west1', 'us-central1'])
]
dir(dir_path) {
if ( env.REGION == "europe-west1" ) {
def eu_cluster_name = env.CLUSTER_COMMON_NAME + "-eu"
container('kubectl-kustomize') {
sh "echo deploy to cluster $eu_cluster_name in region: $REGION"
}
}
if ( env.REGION == "us-central1" ) {
def us_cluster_name = env.CLUSTER_COMMON_NAME + "-us"
container('kubectl-kustomize') {
sh "echo deploy to cluster $us_cluster_name in region: $REGION"
}
}
}
}
}
}
所有例子均在我的 github repo，
### 本文首發于博客園 https://www.cnblogs.com/wade-xu/p/16863955.html
測驗
現在你可以創建一個 Pipeline 或者 Multibranch Pipeline job 來測驗， Repository URL = `https://github.com/wadexu007/learning_by_doing` Script Path, e.g. `Jenkins/k8s_pod_as_build_agent/demo-app-java/Jenkinsfile` 你會看到每啟動一個job 都會相應的產生一個pod 來作為Jenkins agent運行，結束后根據idleMinutes自動釋放，
總結
如果你已經成功創建并測驗 CI/CD pipeline，可以繼續加強，比如加上 Post notifications最佳實踐
- 設定 resource requests and limits on each container in your Pod
- 如果使用maven 構建 java專案，.m2 cache目錄需要 mount 出來，這樣加快后面的maven build速度，
- 使用 Jenkins Shared Libraries 抽取Pipeline的共用代碼
- 在容器里構建容器化應用（Run docker in docker) 我的例子是通過 mount docker.sock 利用k8s 主機 docker engine來實作的，這種方式需要 privileges mode 不安全，推薦使用Kaniko，下一篇文章會介紹，
感謝閱讀，如果您覺得本文的內容對您的學習有所幫助，您可以打賞和推薦，您的鼓勵是我創作的動力， Learning by Doing
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/532574.html
標籤：其他
上一篇：如何選擇開源許可證？