安裝下載依賴包
-
安裝docker-compose
從 docker compose 發布頁面下載最新的 docker-compose 二進制檔案,本文以1.25.4為例
cd /opt/k8s/work/harbor/ wget https://github.com/docker/compose/releases/download/1.25.4/docker-compose-Linux-x86_64 mv docker-compose-Linux-x86_64 /opt/k8s/bin/docker-compose chmod a+x /opt/k8s/bin/docker-compose export PATH=/opt/k8s/bin:$PATH -
下載harbor二進制檔案
從 harbor 發布頁面下載最新的 harbor 離線安裝包，本文以v1.9.4為例
cd /opt/k8s/work/harbor/ wget https://github.com/goharbor/harbor/releases/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz tar -xzvf harbor-offline-installer-v1.5.1.tgz -
匯入 docker images
匯入離線安裝包中 harbor 相關的 docker imagescd /opt/k8s/work/harbor/harbor docker load -i harbor.v1.9.4.tar.gz
以http形式啟動
-
修改harbor.yml檔案
修改 hostname、data_volume屬性值cd /opt/k8s/work/harbor/harbor cp harbor.yml harbor.yml.bak vim harbor.yml修改資訊如下
diff harbor.yml harbor.yml.bak 5c5 < hostname: 192.168.0.107 --- > hostname: reg.mydomain.com 40c40 < data_volume: /data/k8s/harbor/data --- > data_volume: /data -
加載和啟動 harbor 鏡像
cd /opt/k8s/work/harbor/harbor mkdir -p /data/k8s/harbor/data chmod 777 /var/run/docker.sock /data/k8s/harbor/data ./install.sh啟動日志
Note: docker version: 18.09.6 Note: docker-compose version: 1.25.4 # Configuration file of Harbor [Step 1]: loading Harbor images ... Loaded image: goharbor/harbor-core:v1.9.4 Loaded image: goharbor/clair-photon:v2.1.0-v1.9.4 Loaded image: goharbor/harbor-portal:v1.9.4 Loaded image: goharbor/nginx-photon:v1.9.4 Loaded image: goharbor/chartmuseum-photon:v0.9.0-v1.9.4 Loaded image: goharbor/prepare:v1.9.4 Loaded image: goharbor/redis-photon:v1.9.4 Loaded image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.4 Loaded image: goharbor/notary-server-photon:v0.6.1-v1.9.4 Loaded image: goharbor/harbor-log:v1.9.4 Loaded image: goharbor/harbor-db:v1.9.4 Loaded image: goharbor/harbor-jobservice:v1.9.4 Loaded image: goharbor/harbor-registryctl:v1.9.4 Loaded image: goharbor/notary-signer-photon:v0.6.1-v1.9.4 Loaded image: goharbor/harbor-migrator:v1.9.4 [Step 2]: preparing environment ... prepare base dir is set to /opt/k8s/work/harbor/harbor Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /secret/keys/secretkey Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir [Step 3]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating redis ... done Creating harbor-portal ... done Creating registry ... done Creating registryctl ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ? ----Harbor has been installed and started successfully.---- Now you should be able to visit the admin portal at http://192.168.0.107. For more details, please visit https://github.com/goharbor/harbor .查看啟動狀態
root@master:/opt/k8s/work/harbor/harbor# docker-compose ps Name Command State Ports --------------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Up (healthy) harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /entrypoint.sh /etc/regist ... Up (healthy) 5000/tcp registryctl /harbor/start.sh Up (healthy)-
如果有不是healthy狀態的物件，到/var/log/harbor/目錄下查看對應物件的日志
-
harbor官方提供的啟動檔案中容器的日志型別是syslog，不支持用docker logs查看
ls /var/log/harbor/
-
core.log jobservice.log portal.log postgresql.log proxy.log redis.log registryctl.log registry.log
```
瀏覽器訪問 http://192.168.0.107,用賬號 admin 和 harbor.yml 組態檔中的默認密碼 Harbor12345 登陸系統，
-
創建一個新的專案
-
docker命令拉取和上傳鏡像
-
執行login
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. Error response from daemon: Get https://192.168.0.107/v2/: dial tcp 192.168.0.107:443: connect: connection refused -
因為docker命令默認采用https和API互動，而我們的harbor是http的，所以不能執行，需要在/etc/docker/daemon.json中追加insecure-registries配置
{ "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"], "insecure-registries": ["192.168.0.107"], "max-concurrent-downloads": 20, "live-restore": true, "max-concurrent-uploads": 10, "data-root": "/data/k8s/docker/data", "log-opts": { "max-size": "100m", "max-file": "5" } } -
重新啟動docker服務
root@slave:~# systemctl restart docker root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded -
向harbor中剛創建的私有專案里面上傳鏡像
root@slave:~# docker tag k8s.gcr.io/pause:3.1 192.168.0.107/k8s/pause:3.1 root@slave:~# docker push 192.168.0.107/k8s/pause:3.1 The push refers to repository [192.168.0.107/k8s/pause] e17133b79956: Pushed 3.1: digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d size: 527 -
瀏覽器查看
-
從harbor中下載鏡像
root@slave:~# docker rmi 192.168.0.107/k8s/pause:3.1 Untagged: 192.168.0.107/k8s/pause:3.1 Untagged: 192.168.0.107/k8s/pause@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d root@slave:~# docker pull 192.168.0.107/k8s/pause:3.1 3.1: Pulling from k8s/pause Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d Status: Downloaded newer image for 192.168.0.107/k8s/pause:3.1
-
以https形式啟動
下列操作的作業目錄/opt/k8s/work/harbor/harbor是解壓harbor離線安裝檔案后 生成的 harbor 目錄
-
停止harbor（如果沒有啟動過，則跳過該步驟）
cd /opt/k8s/work/harbor/harbor root@master:/opt/k8s/work/harbor/harbor# COMPOSE_HTTP_TIMEOUT=200 docker-compose down -v Stopping nginx ... done Stopping harbor-jobservice ... done Stopping harbor-core ... done Stopping registryctl ... done Stopping redis ... done Stopping harbor-portal ... done Stopping harbor-db ... done Stopping registry ... done Stopping harbor-log ... done Removing harbor-log ... done Removing network harbor_harbor-
執行docker-compose down -v后，有可能對應的容器還在，可以執行強制停掉容器
docker ps | grep harbor | awk '{print $1}' | xargs -I {} docker rm -f {}
-
-
創建 harbor nginx 服務器使用的 x509 證書，本檔案采用cfssl工具生成，也可以利用openssl生成，具體步驟參考harbor-openssl
-
創建harbor用的CA根證書
-
創建組態檔
cd /opt/k8s/work/harbor/ cat > harbor-ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "harbor": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF -
創建證書簽名請求檔案
cd /opt/k8s/work/harbor/ cat > harbor-ca-csr.json <<EOF { "CN": "harbor-ca", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "NanJing", "L": "NanJing", "O": "k8s", "OU": "system" } ], "ca": { "expiry": "87600h" } } EOF -
生成自簽名的根證書,分發證書
cd /opt/k8s/work/harbor/ cfssl gencert -initca harbor-ca-csr.json | cfssljson -bare harbor-ca ls harbor-ca*pem mkdir -p /etc/harbor/cert mv harbor-ca*pem harbor-ca-config.json /etc/harbor/cert
-
-
創建harbor用的證書
-
生成證書請求檔案
cd /opt/k8s/work/harbor/ cat > harbor-server-csr.json <<EOF { "CN": "harbor", "hosts": [ "127.0.0.1", "192.168.0.107" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "NanJing", "L": "NanJing", "O": "k8s", "OU": "system" } ] } EOF- hosts 欄位指定授權使用該證書的當前部署節點 IP
-
生成證書
cfssl gencert -ca=/etc/harbor/cert/harbor-ca.pem \
-
-
-ca-key=/etc/harbor/cert/harbor-ca-key.pem
-config=/etc/harbor/cert/harbor-ca-config.json
-profile=harbor harbor-server-csr.json | cfssljson -bare harbor-server
ls harbor-server*pem
mv harbor-server*pem /etc/harbor/cert
```
-
編輯組態檔
修改 hostname、data_volume屬性值,去除默認的http方式，追加https的配置
cd /opt/k8s/work/harbor/harbor vim harbor.yml修改內容如下
diff harbor.yml harbor.yml.bak 5c5 < hostname: 192.168.0.107 --- > hostname: reg.mydomain.com 8c8 < #http: --- > http: 10c10 < # port: 80 --- > port: 80 13c13 < https: --- > # https: 15c15 < port: 443 --- > # port: 443 17,18c17,18 < certificate: /etc/harbor/cert/harbor-server.pem < private_key: /etc/harbor/cert/harbor-server-key.pem --- > # certificate: /your/certificate/path > # private_key: /your/private/key/path 40c40 < data_volume: /data/k8s/harbor/data --- > data_volume: /data -
生成組態檔（如果沒有啟動過，則跳過該步驟）
cd /opt/k8s/work/harbor/harbor ./prepare執行日志
prepare base dir is set to /opt/k8s/work/harbor/harbor Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/jobservice/config.yml Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /config/db/env Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registry/root.crt Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/core/env Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /secret/keys/secretkey Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir修改生成檔案的訪問權限
cd /opt/k8s/work/harbor/harbor chmod -R 777 common -
啟動harbor
如果是首次啟動 執行
cd /opt/k8s/work/harbor/harbor mkdir -p /data/k8s/harbor/data chmod 777 /var/run/docker.sock /data/k8s/harbor/data ./install.sh否則執行
docker-compose up -d -
查看啟動狀態
root@master:/opt/k8s/work/harbor/harbor# docker-compose ps Name Command State Ports --------------------------------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Up (healthy) harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /entrypoint.sh /etc/regist ... Up (healthy) 5000/tcp registryctl /harbor/start.sh Up (healthy)- nginx 暴露了兩個埠http的80和https的443，使用http訪問80會自動重定向到https
-
瀏覽器訪問 https://192.168.0.107,用賬號 admin 和 harbor.yml 組態檔中的默認密碼 Harbor12345 登陸系統，
- 其中的k8s是在http模式下創建的專案，直接啟動https的可參照上面創建
-
docker命令拉取和上傳鏡像
如果之前已經執行過docker login，配置修改后需要重新docker login，需要洗掉 /root/.docker/config.json檔案
如果在/etc/docker/daemon.json中配置過insecure-registries，需要去掉,并重新啟動docker服務
-
執行login
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
-
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://192.168.0.107/v2/: x509: certificate signed by unknown authority
```
錯誤是說我們用的證書是自簽名的證書，簽發證書機構未經認證
解決方法是將簽署 harbor-server 證書的 CA 證書拷貝到 /etc/docker/certs.d/192.168.0.107 目錄下（需要在所有要訪問harbor的節點上都執行此操作）
```
root@slave:~# mkdir -p /etc/docker/certs.d/192.168.0.107
root@slave:~# scp [email protected]:/etc/harbor/cert/harbor-ca.pem /etc/docker/certs.d/192.168.0.107/ca.crt
[email protected]'s password:
harbor-ca.pem 100% 1306 283.7KB/s 00:00
root@slave:~# ls /etc/docker/certs.d/192.168.0.107/
ca.crt
```
重新執行login
```
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
```
-
上傳鏡像
root@slave:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE kubernetesui/metrics-scraper v1.0.3 3327f0dbcb4a 2 weeks ago 40.1MB busybox latest 6d5fcfe5ff17 7 weeks ago 1.22MB coredns/coredns 1.6.6 cc4d8e8c6169 2 months ago 40.8MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB 192.168.0.107/k8s/pause 3.1 da86e6ba6ca1 2 years ago 742kB kubeimage/pause 3.1 da86e6ba6ca1 2 years ago 742kB nginx 1.9.1 94ec7e53edfc 4 years ago 133MB root@slave:~# docker tag busybox:latest 192.168.0.107/k8s/busybox:latest root@slave:~# docker push 192.168.0.107/k8s/busybox:latest The push refers to repository [192.168.0.107/k8s/busybox] 195be5f8be1d: Pushed latest: digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 size: 527 -
通過瀏覽器查看剛上傳的鏡像
-
從harbor下載鏡像
root@slave:~# docker rmi 192.168.0.107/k8s/busybox:latest Untagged: 192.168.0.107/k8s/busybox:latest Untagged: 192.168.0.107/k8s/busybox@sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 root@slave:~# docker pull 192.168.0.107/k8s/busybox:latest latest: Pulling from k8s/busybox Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 Status: Downloaded newer image for 192.168.0.107/k8s/busybox:latest
kubernetes集群中使用私有的harbor鏡像倉庫
-
kubelet配置
kubernetes以pod為管理單元，而不是docker容器，在創建pod時，會先啟動一個名為k8s.gcr.io/pause:3.1的鏡像，
該鏡像存在于google的鏡像倉庫中，國內不能直接訪問，需要翻墻下載，如果集群新加節點，或者節點上這個鏡像被清理掉，將導致pod不能啟動
一般做法是搭建私有鏡像倉庫，把這個鏡像push到我們的私有鏡像倉庫，再配置kubelet使用我們私有倉庫中的pause鏡像
具體做法是修改kubelet服務的啟動檔案:/etc/systemd/system/kubelet.service,在其中追加--pod-infra-container-image屬性，屬性值指向自己私有倉庫中的pause鏡像
例如
--pod-infra-container-image=192.168.0.107/k8s/pause:3.1重啟kubelet服務
systemctl restart kubelet -
kubernetes和harbor進行認證
kubernetes提供了各種方式來訪問各種鏡像倉庫，具體參考Using a Private Registry,本文采用配置節點的方式來使用私有倉庫
-
在集群的任意一個節點上登陸harbor
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded成功后會生成認證檔案/root/.docker/config.json
root@slave:~# ls /root/.docker/ config.json root@slave:~# cat /root/.docker/config.json { "auths": { "192.168.0.107": { "auth": "YWRtaW46SGFyYm9yMTIzNDU=" } }, "HttpHeaders": { "User-Agent": "Docker-Client/18.09.6 (linux)" } } -
配置kubelet使用此認證檔案
默認情況下，kubelet會從如下路徑中查找私有倉庫的認證檔案
- {--root-dir:-/var/lib/kubelet}/config.json
- {cwd of kubelet}/config.json
- ${HOME}/.docker/config.json #須在環境變數檔案中為 kubelet 顯式設定 HOME=/root
- /.docker/config.json
此處我們將私有倉庫的認證檔案copy到kubelet的root目錄下，此目錄在kubelet.service中進行設定
例如--root-dir=/data/k8s/k8s/kubelet -
copy私有倉庫的認證檔案到kubelet的root目錄（所有運行kubelet，需要訪問harbor的節點都需要），本環境中有兩個節點
cp /root/.docker/config.json /data/k8s/k8s/kubelet/config.json scp /root/.docker/config.json [email protected]:/data/k8s/k8s/kubelet/config.json
-
-
驗證
創建一個pod,image 指定私有倉庫里面的鏡像
cd /opt/k8s/yml cat > private-busybox.yml << EOF apiVersion: v1 kind: Pod metadata: name: private-busybox spec: containers: - name: private-busybox image: 192.168.0.107/k8s/busybox:latest command: - sleep - "3600" EOF創建pod
root@master:/opt/k8s/yml# kubectl create -f private-busybox.yml查看運行狀態
root@master:/opt/k8s/yml# kubectl get pod | grep private-busybox private-busybox 1/1 Running 0 108s通過docker命令，查看啟動pod對應的容器的鏡像名稱
root@master:/opt/k8s/yml# docker ps | grep -e IMAGE -e private-busybox CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c2b43d9d9b3e 192.168.0.107/k8s/busybox "sleep 3600" About a minute ago Up About a minute k8s_private-busybox_private-busybox_default_98a4d4bd-fd46-4d9d-8ef9-95845ce7e53d_0 a4a84461d5fe 192.168.0.107/k8s/pause:3.1 "/pause" About a minute ago Up About a minute k8s_POD_private-busybox_default_98a4d4bd-fd46-4d9d-8ef9-95845ce7e53d_0 1/1 Running 0 5s可以看到啟動的容器對應的鏡像都是我們自己的私有倉庫中的鏡像
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/41200.html
標籤：其他
下一篇：Docker 網路原理