@[TOC]( 運維實戰 kubernetes(k8s) 之 service )
1. service 介紹
- Service可以看作是一組提供相同服務的Pod對外的訪問介面，借助Service，應用可以方便地實作服務發現和負載均衡，
- service默認只支持4層負載均衡能力，沒有7層功能，（可以通過Ingress實作）
- service的型別：
ClusterIP：默認值，k8s系統給service自動分配的虛擬IP，只能在集群內部訪問，
NodePort：將Service通過指定的Node上的埠暴露給外部，訪問任意一個NodeIP:nodePort都將路由到ClusterIP，
LoadBalancer：在 NodePort 的基礎上，借助 cloud provider 創建一個外部的負載均衡器，并將請求轉發到 :NodePort，此模式只能在云服務器上使用，
ExternalName：將服務通過 DNS CNAME 記錄方式轉發到指定的域名（通過 spec.externlName 設定）， - Service 是由 kube-proxy 組件，加上 iptables 來共同實作的.
- kube-proxy 通過 iptables 處理 Service 的程序，需要在宿主機上設定相當多的 iptables 規則，如果宿主機有大量的Pod，不斷重繪iptables規則，會消耗大量的CPU資源，
- IPVS模式的 service，可以使K8s集群支持更多量級的Pod，
先啟動倉庫，然后輸入變數，查看kubectl的狀態；
[root@server1 ~]# cd harbor/
[root@server1 harbor]# docker-compose start
[root@server2 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
[root@server2 ~]# kubectl get pod -n kube-system
2. 開啟 kube-proxy 的 ipvs 模式
要確保倉庫的存在；此處用的是本機的源；
[root@server2 k8s]# cd /etc/yum.repos.d/
[root@server2 yum.repos.d]# ls
docker.repo dvd.repo k8s.repo redhat.repo
[root@server2 yum.repos.d]# vim k8s.repo
[root@server2 yum.repos.d]# cat k8s.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=0 ##將此處改為0，不啟用該源；使用本機的源來安裝
gpgcheck=0
每個結點安裝yum install -y ipvsadm 軟體；
安裝完成之后：
[root@server2 ~]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@server2 ~]# ipvsadm -ln
##查看 iptables 的規則，它是內核功能
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@server2 ~]# lsmod | grep ip_vs
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 0
ip_vs 145497 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 133095 10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
[root@server2 ~]# kubectl -n kube-system get cm
## 查看配置資訊
NAME DATA AGE
coredns 1 25h
extension-apiserver-authentication 6 25h
kube-flannel-cfg 2 24h
kube-proxy 2 25h
kube-root-ca.crt 1 25h
kubeadm-config 2 25h
kubelet-config-1.21 1 25h
[root@server2 ~]# kubectl -n kube-system edit cm kube-proxy
##編輯配置資訊，指定使用 ipvs 的模式，不寫時默認用的是 iptables
configmap/kube-proxy edited
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: ""
strictARP: false
syncPeriod: 0s
tcpFinTimeout: 0s
tcpTimeout: 0s
udpTimeout: 0s
kind: KubeProxyConfiguration
metricsBindAddress: ""
mode: "ipvs"
configmap/kube-proxy edited
修改完資訊之后，需要多載；由于當前的服務是由控制器所管理，此時只需洗掉之前的pod ，會再次讀取組態檔重新拉取pod；
kube-proxy 通過 linux 的 IPVS 模塊，以 rr 輪詢方式調度 service 的Pod，
[root@server2 ~]# kubectl -n kube-system get daemonsets.apps
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-flannel-ds 3 3 3 3 3 <none> 24h
kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 25h
[root@server2 ~]# kubectl -n kube-system get pod | grep kube-proxy | awk '{system("kubectl -n kube-system delete pod "$1"")}'
pod "kube-proxy-866lg" deleted
pod "kube-proxy-hxgbt" deleted
pod "kube-proxy-jrc9z" deleted
[root@server2 ~]# ipvsadm -ln
##重啟之后，此時在每個結點上都可以看到 iptables 策略；其中10.96.0.10是 CLUSTER-IP 的地址；0.8 和 0.9 是 dns 所在 pod 的地址；
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 172.25.25.2:6443 Masq 1 0 0
TCP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.0.8:9153 Masq 1 0 0
-> 10.244.0.9:9153 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
[root@server2 ~]# kubectl -n kube-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8d
[root@server2 ~]# kubectl -n kube-system get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-85ffb569d4-85kp7 1/1 Running 3 8d 10.244.0.9 server2 <none> <none>
coredns-85ffb569d4-bd579 1/1 Running 3 8d 10.244.0.8 server2 <none> <none>
etcd-server2 1/1 Running 3 8d 172.25.25.2 server2 <none> <none>
kube-apiserver-server2 1/1 Running 3 8d 172.25.25.2 server2 <none> <none>
kube-controller-manager-server2 1/1 Running 3 8d 172.25.25.2 server2 <none> <none>
kube-flannel-ds-f8qhr 1/1 Running 2 8d 172.25.25.4 server4 <none> <none>
kube-flannel-ds-hvfwp 1/1 Running 2 8d 172.25.25.3 server3 <none> <none>
kube-flannel-ds-mppbp 1/1 Running 3 8d 172.25.25.2 server2 <none> <none>
kube-proxy-6f78h 1/1 Running 0 4m10s 172.25.25.2 server2 <none> <none>
kube-proxy-7jvkr 1/1 Running 0 4m12s 172.25.25.4 server4 <none> <none>
kube-proxy-9d5s7 1/1 Running 0 4m5s 172.25.25.3 server3 <none> <none>
kube-scheduler-server2 1/1 Running 3 8d 172.25.25.2 server2 <none> <none>
IPVS 模式下，kube-proxy 會在 service 創建后，在宿主機上添加一個虛擬網卡：kube-ipvs0，并分配 service IP，
[root@server2 ~]# ip addr show kube-ipvs0
10: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
link/ether 52:54:5e:c0:51:56 brd ff:ff:ff:ff:ff:ff
inet 10.96.0.10/32 scope global kube-ipvs0
valid_lft forever preferred_lft forever
inet 10.96.0.1/32 scope global kube-ipvs0
valid_lft forever preferred_lft forever
新建一個來觀察效果：
[root@server2 k8s]# kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
[root@server2 k8s]# ls
cronjob.yaml daemonset.yaml deployment.yaml job.yaml pod.yaml rs.yaml svc.yaml
[root@server2 k8s]# vim deployment.yaml
[root@server2 k8s]# kubectl apply -f deployment.yaml
deployment.apps/deployment-example created
[root@server2 k8s]# kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-example-5b768f7647-9wlvc 1/1 Running 0 4s
deployment-example-5b768f7647-j6bvs 1/1 Running 0 4s
deployment-example-5b768f7647-ntmk7 1/1 Running 0 4s
[root@server2 k8s]# kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
deployment-example-5b768f7647-9wlvc 1/1 Running 0 52s app=nginx,pod-template-hash=5b768f7647
deployment-example-5b768f7647-j6bvs 1/1 Running 0 52s app=nginx,pod-template-hash=5b768f7647
deployment-example-5b768f7647-ntmk7 1/1 Running 0 52s app=nginx,pod-template-hash=5b768f7647
[root@server2 k8s]# ipvsadm -ln
##此時雖然已經有了 pod 但是并沒有加進去，沒有 svc，
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 172.25.25.2:6443 Masq 1 0 0
TCP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.0.8:9153 Masq 1 0 0
-> 10.244.0.9:9153 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
將指令轉為 yaml 檔案；
[root@server2 k8s]# kubectl expose deployment deployment-example --port=80 --target-port=80
service/deployment-example exposed
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deployment-example ClusterIP 10.105.194.76 <none> 80/TCP 8s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
[root@server2 k8s]# kubectl describe svc deployment-example
Name: deployment-example
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.194.76
IPs: 10.105.194.76
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.26:80,10.244.2.33:80,10.244.2.34:80
Session Affinity: None
Events: <none>
[root@server2 k8s]# kubectl get svc deployment-example -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2021-06-12T13:30:52Z"
name: deployment-example
namespace: default
resourceVersion: "60216"
uid: 7729b22e-4e26-4e6e-afa1-7c4e0a37e019
spec:
clusterIP: 10.105.194.76
clusterIPs:
- 10.105.194.76
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
[root@server2 k8s]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 172.25.25.2:6443 Masq 1 0 0
TCP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.0.8:9153 Masq 1 0 0
-> 10.244.0.9:9153 Masq 1 0 0
TCP 10.105.194.76:80 rr
##此時查看時，會有三個pod
-> 10.244.1.26:80 Masq 1 0 0
-> 10.244.2.33:80 Masq 1 0 0
-> 10.244.2.34:80 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
此時測驗時，會負載均衡到后端的三個 pod 上
[root@server2 k8s]# curl 10.105.194.76
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@server2 k8s]# curl 10.105.194.76/hostname.html
deployment-example-5b768f7647-j6bvs
[root@server2 k8s]# curl 10.105.194.76/hostname.html
deployment-example-5b768f7647-9wlvc
[root@server2 k8s]# curl 10.105.194.76/hostname.html
deployment-example-5b768f7647-ntmk7
[root@server2 k8s]# ipvsadm -ln
測驗之后，可以用此命令查看調度的次數
當用命令kubectl delete svc deployment-example 將服務洗掉時，此時也就在 ipvs中看不到資訊，
除了上述用指令生成 yaml檔案的方法之外，還可以直接撰寫 yaml檔案；
[root@server2 k8s]# vim svc.yaml
[root@server2 k8s]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
service/myservice created
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
myservice ClusterIP 10.104.41.30 <none> 80/TCP 5s
[root@server2 k8s]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 172.25.25.2:6443 Masq 1 0 0
TCP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.0.8:9153 Masq 1 0 0
-> 10.244.0.9:9153 Masq 1 0 0
TCP 10.104.41.30:80 rr
-> 10.244.1.26:80 Masq 1 0 0
-> 10.244.2.33:80 Masq 1 0 0
-> 10.244.2.34:80 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.0.8:53 Masq 1 0 0
-> 10.244.0.9:53 Masq 1 0 0
3. 創建 service：（NodePort方式）
為了從外部訪問 service 的第一種方式，用 NodePort 的方式會系結節點的埠，供外部來訪問，
以上的方式都是 ClusterIP 的方式，此時修改一下格式：
[root@server2 k8s]# vim svc.yaml
[root@server2 k8s]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
type: NodePort
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
service/myservice created
[root@server2 k8s]# kubectl get svc
##此時會將埠暴露出來，外部在訪問時需要指定埠來訪問
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
myservice NodePort 10.100.227.116 <none> 80:32204/TCP 6s
在書寫 yaml 檔案時，可以用命令來獲得幫助 kubectl explain service.spec等后面跟得資訊都可以查看幫助，
4. DNS 插件 Service
Kubernetes 提供了一個 DNS 插件 Service，
在集群內部直接用DNS記錄的方式訪問，而不需要一個VIP，
[root@server2 k8s]# yum install -y bind-utils.x86_64
##安裝插件
[root@server2 k8s]# cat /etc/resolv.conf
nameserver 114.114.114.114
[root@server2 k8s]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
service/myservice created
[root@server2 k8s]# dig -t A myservice.default.svc.cluster.local @10.96.0.10
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> -t A myservice.default.svc.cluster.local @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4147
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN A 10.108.31.117
##此時決議到的地址為 myservice上的地址
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Jun 13 21:13:34 CST 2021
;; MSG SIZE rcvd: 115
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
myservice ClusterIP 10.108.31.117 <none> 80/TCP 10h
[root@server2 k8s]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 rr
-> 172.25.25.2:6443 Masq 1 3 0
TCP 10.96.0.10:53 rr
-> 10.244.179.65:53 Masq 1 0 0
-> 10.244.179.66:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.179.65:9153 Masq 1 0 0
-> 10.244.179.66:9153 Masq 1 0 0
TCP 10.108.31.117:80 rr
-> 10.244.1.31:80 Masq 1 0 0
-> 10.244.2.40:80 Masq 1 0 0
-> 10.244.2.41:80 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.179.65:53 Masq 1 0 1
-> 10.244.179.66:53 Masq 1 0 1
以上創建的程序中，IP會隨著pod 而變化，但是域名并不會變化；在訪問時可以直接指定域名來訪問，此時也是負載均衡的，
[root@server2 k8s]# kubectl run demo --image=busyboxplus -it --restart=Never
If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # curl myservice
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
/ # curl myservice/hostname.html
deployment-example-5b768f7647-rkpgr
/ #
/ # curl myservice/hostname.html
deployment-example-5b768f7647-czchr
/ # curl myservice/hostname.html
deployment-example-5b768f7647-brg9n
/ # curl myservice/hostname.html
deployment-example-5b768f7647-rkpgr
- Headless Service “無頭服務”
Headless Service不需要分配一個 VIP，而是直接以 DNS 記錄的方式決議出被代理 Pod 的IP地址，
域名格式：$ (servicename).$(namespace).svc.cluster.local
[root@server2 k8s]# kubectl delete -f svc.yaml
service "myservice" deleted
[root@server2 k8s]# vim svc.yaml
[root@server2 k8s]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
type: ClusterIP
clusterIP: None
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
service/myservice created
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28h
myservice ClusterIP None <none> 80/TCP 20s
此時看到的是沒有分配到的VIP，但是可以根據 DNS 記錄中的pod 的地址來訪問；
[root@server2 k8s]# dig -t A myservice.default.svc.cluster.local @10.96.0.10
; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> -t A myservice.default.svc.cluster.local @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11968
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myservice.default.svc.cluster.local. IN A
;; ANSWER SECTION:
myservice.default.svc.cluster.local. 30 IN A 10.244.2.41
myservice.default.svc.cluster.local. 30 IN A 10.244.1.31
myservice.default.svc.cluster.local. 30 IN A 10.244.2.40
;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Jun 13 21:24:28 CST 2021
;; MSG SIZE rcvd: 217
[root@server2 k8s]# kubectl describe svc myservice
Name: myservice
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.90:80,10.244.2.77:80,10.244.2.78:80
Session Affinity: None
Events: <none>
5. pod 滾動更新
以上無頭服務，在 pod 滾動更新之后，其 IP 的變化是隨著 pod 自動更新的；
[root@server2 k8s]# kubectl delete pod --all
pod "demo" deleted
pod "deployment-example-5b768f7647-brg9n" deleted
pod "deployment-example-5b768f7647-czchr" deleted
pod "deployment-example-5b768f7647-rkpgr" deleted
[root@server2 k8s]# kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-example-5b768f7647-2psc6 1/1 Running 0 21s
deployment-example-5b768f7647-cdfdk 1/1 Running 0 21s
deployment-example-5b768f7647-q76rp 1/1 Running 0 21s
[root@server2 k8s]# kubectl describe svc myservice
Name: myservice
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.94:80,10.244.1.95:80,10.244.2.79:80
Session Affinity: None
Events: <none>
6. 創建 service: （LoadBalancer）
從外部訪問 Service 的第二種方式，適用于公有云上的 Kubernetes 服務，這時候，可以指定一個 LoadBalancer 型別的 Service，
[root@server2 k8s]# kubectl delete -f svc.yaml
service "myservice" deleted
[root@server2 k8s]# vim svc.yaml
[root@server2 k8s]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
type: LoadBalancer
#clusterIP: None
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
service/myservice created
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28h
myservice LoadBalancer 10.99.186.132 <pending> 80:31894/TCP 37s
此時是在 nodeport的基礎之上，從云端來分配一個 IP；此處沒有云端時會一直處于 <pending> 的狀態，
- 當是云環境時會通過驅動去分配一個IP，供其訪問；
- 當前是裸金屬環境，那么分配IP 的動作由誰去做呢？
metallb
官網：https://metallb.universe.tf/installation/
設定ipvs模式：
[root@server2 k8s]# kubectl edit configmaps -n kube-system kube-proxy
configmap/kube-proxy edited
37 strictARP: true
44 mode: "ipvs"
[root@server2 k8s]# kubectl -n kube-system get pod | grep kube-proxy | awk '{system("kubectl -n kube-system delete pod "$1"")}'
pod "kube-proxy-6f78h" deleted
pod "kube-proxy-7jvkr" deleted
pod "kube-proxy-9d5s7" deleted
##讓策略生效
部署：先下載資源清單，
[root@server2 k8s]# mkdir metallb
[root@server2 k8s]# cd metallb/
[root@server2 metallb]# wget https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
[root@server2 metallb]# cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
labels:
app: metallb
[root@server2 metallb]# kubectl apply -f namespace.yaml
namespace/metallb-system created
##創建 namespace
[root@server2 metallb]# wget https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
配置清單metallb.yaml 指定的鏡像需要提前下載，并將其上傳至私有倉庫；
[root@server1 ~]# docker pull metallb/speaker:v0.9.6
[root@server1 ~]# docker pull metallb/controller:v0.9.6
[root@server1 harbor]# docker tag metallb/speaker:v0.9.6 reg.westos.org/metallb/speaker:v0.9.6
[root@server1 harbor]# docker tag metallb/controller:v0.9.6 reg.westos.org/metallb/controller:v0.9.6
[root@server1 harbor]# docker push reg.westos.org/metallb/controller；
[root@server1 harbor]# docker push reg.westos.org/metallb/controller:v0.9.6
上傳完之后，此處私有倉庫的地址和檔案中的一致，然后開始部署：
[root@server2 metallb]# kubectl apply -f metallb.yaml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
在第一次部署時，需要生成一個密鑰：
[root@server2 metallb]# kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
[root@server2 metallb]# kubectl -n metallb-system get all
NAME READY STATUS RESTARTS AGE
pod/controller-64f86798cc-qgbsw 1/1 Running 0 2m59s
pod/speaker-f7vtr 1/1 Running 0 2m59s
pod/speaker-jdqv4 1/1 Running 0 2m59s
pod/speaker-t8675 1/1 Running 0 2m59s
##在客戶端控制節點創建一個，在3個節點運行客戶端代理
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 2m59s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 2m59s
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-64f86798cc 1 1 1 2m59s
[root@server2 metallb]# kubectl -n metallb-system get secrets
NAME TYPE DATA AGE
controller-token-hk6k2 kubernetes.io/service-account-token 3 3m18s
default-token-22jtp kubernetes.io/service-account-token 3 9m39s
memberlist Opaque 1 105s
speaker-token-phglk kubernetes.io/service-account-token 3 3m18s
以上創建完成之后，編輯一個檔案：設定其分配IP的范圍，
[root@server2 metallb]# vim config.yaml
[root@server2 metallb]# cat config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 172.25.25.100-172.25.25.200
[root@server2 metallb]# kubectl apply -f config.yaml
configmap/config created
[root@server2 metallb]# kubectl -n metallb-system get cm
NAME DATA AGE
config 1 10s
kube-root-ca.crt 1 11m
[root@server2 metallb]# kubectl -n metallb-system describe cm config
Name: config
Namespace: metallb-system
Labels: <none>
Annotations: <none>
Data
====
config:
----
address-pools:
- name: default
protocol: layer2
addresses:
- 172.25.15.100-172.25.15.200
Events: <none>
來測驗外部能否獲取VIP；
[root@server2 metallb]# cd ..
[root@server2 k8s]# vim svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
#type: ClusterIP
#type: NodePort
#clusterIP: None
type: LoadBalancer
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
service/myservice unchanged
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29h
myservice LoadBalancer 10.99.186.132 172.25.15.100 80:31894/TCP 29m
此時已經分配到了vip ，外部直接可以訪問：
[root@foundation15 ~]# curl 172.25.15.100
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@foundation15 ~]# curl 172.25.15.100/hostname.html
deployment-example-5b768f7647-cdfdk
7. 創建 service :(ExternalName)
從外部訪問的第三種方式叫做ExternalName，決議名稱，常用于外部控制，
[root@server2 k8s]# vim ex-svc.yaml
[root@server2 k8s]# cat ex-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: ex-svc
spec:
type: ExternalName
externalName: www.baidu.com
[root@server2 k8s]# kubectl apply -f ex-svc.yaml
service/ex-svc created
[root@server2 k8s]# kubectl get svc
##此處沒有分配ip,用域名的方式可以訪問
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ex-svc ExternalName <none> www.baidu.com <none> 10s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29h
myservice LoadBalancer 10.99.186.132 172.25.15.100 80:31894/TCP 33m
[root@server2 k8s]# kubectl describe svc ex-svc
Name: ex-svc
Namespace: default
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ExternalName
IP Families: <none>
IP:
IPs: <none>
External Name: www.baidu.com
Session Affinity: None
Events: <none>
[root@server2 k8s]# dig -t A ex-svc.default.svc.cluster.local @10.96.0.10
;; ANSWER SECTION:
ex-svc.default.svc.cluster.local. 30 IN CNAME www.baidu.com.
www.baidu.com. 30 IN CNAME www.a.shifen.com.
www.a.shifen.com. 30 IN A 36.152.44.95
www.a.shifen.com. 30 IN A 36.152.44.96
假設外部資源的域名發生變化：
[root@server2 k8s]# vim ex-svc.yaml
[root@server2 k8s]# cat ex-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: ex-svc
spec:
type: ExternalName
externalName: www.westos.org
[root@server2 k8s]# kubectl apply -f ex-svc.yaml
service/ex-svc configured
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ex-svc ExternalName <none> www.westos.com <none> 2m44s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
myservice LoadBalancer 10.99.186.132 172.25.25.100 80:31981/TCP 7m45s
[root@server2 k8s]# dig -t A ex-svc.default.svc.cluster.local @10.96.0.10
;; ANSWER SECTION:
ex-svc.default.svc.cluster.local. 30 IN CNAME www.westos.org.
www.westos.org. 30 IN CNAME applkdmhnt09730.pc-cname.xiaoe-tech.com.
applkdmhnt09730.pc-cname.xiaoe-tech.com. 30 IN A 118.25.119.100
可以發現即使外部域名發生變化，不變的是svc，集群內部可以將地址設定為 svc 的地址；將其做個映射就可以，不用做太大的變更，
以上的方式是分配地址，service 允許為其分配一個公有IP，
8. ingress
Kubernetes 里的 Ingress 服務是一種全域的、為了代理不同后端 Service 而設定的負載均衡服務，
Ingress由兩部分組成：Ingress controller 和 Ingress 服務，
Ingress Controller 會根據你定義的 Ingress 物件，提供對應的代理能力，業界常用的各種反向代理專案，比如 Nginx、HAProxy、Envoy、Traefik 等，都已經為Kubernetes 專門維護了對應的 Ingress Controller，
8.1 ingress的配置
官網：https://kubernetes.github.io/ingress-nginx/
應用 ingress controller 定義檔案:
[root@server2 k8s]# mkdir ingress
[root@server2 k8s]# cd ingress/
[root@server2 ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.47.0/deploy/static/provider/baremetal/deploy.yaml
根據檔案內容需要下載兩個鏡像；此處用直接下載好經過打包的鏡像上傳至私有倉庫，
[root@server1 ~]# docker load -i ingress-nginx-v0.46.0.tar
[root@server1 ~]# docker push reg.westos.org/ingress-nginx/controller:v0.46.0
[root@server1 ~]# docker push reg.westos.org/ingress-nginx/kube-webhook-certgen:v1.5.1
然后修改檔案中的鏡像指向，并部署：
[root@server2 ingress]# vim deploy.yaml
324 image: ingress-nginx/controller:v0.46.0
589 image: ingress-nginx/kube-webhook-certgen:v1.5.1
635 image: ingress-nginx/kube-webhook-certgen:v1.5.1
[root@server2 ingress]# kubectl apply -f deploy.yaml
部署好 ingress之后，查看其相關一些資訊；可以看到有 NodePort是 供集外部可以訪問 和ClusterIP是集群內部訪問；
[root@server2 ingress]# kubectl -n ingress-nginx get all
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-n49ww 0/1 Completed 0 10s
pod/ingress-nginx-admission-patch-mqmxq 0/1 Completed 1 10s
pod/ingress-nginx-controller-56c7fc94cb-dzvft 0/1 ContainerCreating 0 10s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.103.186.206 <none> 80:32594/TCP,443:32643/TCP 10s
service/ingress-nginx-controller-admission ClusterIP 10.103.30.139 <none> 443/TCP 10s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 10s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-56c7fc94cb 1 1 0 10s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 10s
job.batch/ingress-nginx-admission-patch 1/1 5s 10s
[root@server2 ingress]# kubectl -n ingress-nginx get pod
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-n49ww 0/1 Completed 0 62s
ingress-nginx-admission-patch-mqmxq 0/1 Completed 1 62s
ingress-nginx-controller-56c7fc94cb-dzvft 1/1 Running 0 62s
[root@server2 ingress]# kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.47.0
helm.sh/chart=ingress-nginx-3.33.0
Annotations: <none>
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.103.186.206
IPs: 10.103.186.206
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32594/TCP
Endpoints: 10.244.1.35:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 32643/TCP
Endpoints: 10.244.1.35:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
[root@server2 ingress]# kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.103.186.206 <none> 80:32594/TCP,443:32643/TCP 107s
ingress-nginx-controller-admission ClusterIP 10.103.30.139 <none> 443/TCP 107s
[root@server2 ingress]# curl 10.103.186.206
##集群內此時并不能訪問到資訊，是因為還沒有定義 ingress
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
為了可以訪問，需要有可用的svc；
[root@server2 k8s]# vim svc.yaml
[root@server2 k8s]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
#type: LoadBalancer
type: ClusterIP
#clusterIP: None
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 k8s]# kubectl apply -f svc.yaml
[root@server2 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
myservice ClusterIP 10.97.144.118 <none> 80/TCP 13s
[root@server2 k8s]# kubectl describe svc myservice
##此時三個 pod 已經就緒
Name: myservice
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.94:80,10.244.1.95:80,10.244.2.79:80
Session Affinity: None
Events: <none>
[root@server2 k8s]# kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-example-5b768f7647-2psc6 1/1 Running 0 129m
deployment-example-5b768f7647-cdfdk 1/1 Running 0 129m
deployment-example-5b768f7647-q76rp 1/1 Running 0 129m
將后端的三個 pod 暴露出去，供外部可以訪問；當前的 ClusterIP不能讓外部訪問，此時需要用ingress來訪問；
[root@server2 ingress]# vim ingress.yaml
[root@server2 ingress]# cat ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www1
spec:
rules:
- host: www1.westos.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
[root@server2 ingress]# kubectl apply -f ingress.yaml
[root@server2 ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-www1 <none> www1.westos.org 80 13s
[root@server2 ingress]# kubectl describe ingress ingress-www1
Name: ingress-www1
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www1.westos.org
/ myservice:80 (10.244.1.94:80,10.244.1.95:80,10.244.2.79:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 26s nginx-ingress-controller Scheduled for sync
此時在訪問 www1.westos.org 時，會調度到后端的 pod 上；但是此時對于域名的訪問并不能被識別，需要做決議；將其決議到 nginx 的控制器上以NodePort來識別,訪問時必須加上埠號；
[root@westos ~]# tail -n 1 /etc/hosts
172.25.25.2 server2 www1.westos.org
[root@westos ~]# curl www1.westos.org
curl: (7) Failed to connect to www1.westos.org port 80: Connection refused
[root@westos ~]# curl www1.westos.org:32594
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@westos ~]# curl www1.westos.org:32594/hostname.html
deployment-example-5b768f7647-t769f
[root@westos ~]# curl www1.westos.org:32594/hostname.html
deployment-example-5b768f7647-rlvlw
[root@westos ~]# curl www1.westos.org:32594/hostname.html
deployment-example-5b768f7647-jv7kf
除了NodePort形式之外，還可以用LoadBalancer來分配外部 IP；
[root@server2 ingress]# kubectl -n ingress-nginx edit svc
49 type: LoadBalancer
50 status:
51 loadBalancer: {}
service/ingress-nginx-controller edited
service/ingress-nginx-controller-admission skipped
[root@server2 ingress]# kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.103.186.206 172.25.25.100 80:32594/TCP,443:32643/TCP 91m
ingress-nginx-controller-admission ClusterIP 10.103.30.139 <none> 443/TCP
可以看到分配到了一個外部 IP；此時在修改決議檔案，不能再決議到之前的 ip，當之前的主機掛掉之后，就不能訪問；但是指向分配的IP，便不存在該問題,訪問時不用加埠；
[root@westos ~]# vim /etc/hosts
[root@westos ~]# tail -n1 /etc/hosts
172.25.25.100 www1.westos.org
[root@westos ~]# ping -c1 -w1 www1.westos.org
PING www1.westos.org (172.25.25.100) 56(84) bytes of data.
64 bytes from www1.westos.org (172.25.25.100): icmp_seq=1 ttl=64 time=0.542 ms
--- www1.westos.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.542/0.542/0.542/0.000 ms
[root@westos ~]# curl www1.westos.org
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@westos ~]# curl www1.westos.org/hostname.html
deployment-example-5b768f7647-t769f
[root@westos ~]# curl www1.westos.org/hostname.html
deployment-example-5b768f7647-jv7kf
[root@westos ~]# curl www1.westos.org/hostname.html
deployment-example-5b768f7647-t769f
再添加來觀察效果：
[root@server2 ingress]# cp ../svc.yaml .
[root@server2 ingress]# vim svc.yaml
[root@server2 ingress]# cat svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: nginx
#type: LoadBalancer
type: ClusterIP
#clusterIP: None
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
spec:
selector:
app: myapp
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
[root@server2 ingress]# kubectl apply -f svc.yaml
service/myservice unchanged
service/nginx-svc created
[root@server2 ingress]# kubectl describe svc nginx-svc
##此時沒有Endpoints控制器
Name: nginx-svc
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=myapp
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.108.174.65
IPs: 10.108.174.65
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
[root@server2 ingress]# kubectl get pod --show-labels
##也沒有myapp的標簽
NAME READY STATUS RESTARTS AGE LABELS
deployment-example-5b768f7647-2psc6 1/1 Running 0 144m app=nginx,pod-template-hash=5b768f7647
deployment-example-5b768f7647-cdfdk 1/1 Running 0 144m app=nginx,pod-template-hash=5b768f7647
deployment-example-5b768f7647-q76rp 1/1 Running 0 144m app=nginx,pod-template-hash=5b768f7647
為其添加控制器和標簽：
[root@server2 ingress]# cp ../deployment.yaml .
[root@server2 ingress]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-www2
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:v2
livenessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 2
periodSeconds: 3
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /hostname.html
port: 80
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 1
[root@server2 ingress]# kubectl apply -f deployment.yaml
[root@server2 ingress]# kubectl get pod --show-labels
##此時查看時就有了標簽，也有了pod 資訊
[root@server2 ingress]# kubectl describe svc nginx-svc
Name: nginx-svc
Namespace: default
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.97:80,10.244.1.98:80,10.244.2.83:80
Session Affinity: None
Events: <none>
此時編輯 ingress 添加一個頁面；此時是動態更新；
[root@server2 ingress]# vim ingress.yaml
[root@server2 ingress]# cat ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www1
spec:
rules:
- host: www1.westos.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
- host: www2.westos.org
http:
paths:
- path: /
backend:
serviceName: nginx-svc
servicePort: 80
[root@server2 ingress]# kubectl apply -f ingress.yaml
[root@server2 ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-www1 <none> www1.westos.org,www2.westos.org 172.25.15.100 80 18m
[root@server2 ingress]# kubectl describe ingress ingress-www1
Name: ingress-www1
Namespace: default
Address: 172.25.15.100
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www1.westos.org
/ myservice:80 (10.244.1.94:80,10.244.1.95:80,10.244.2.79:80)
www2.westos.org
/ nginx-svc:80 (10.244.1.97:80,10.244.1.98:80,10.244.2.83:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 31s (x3 over 18m) nginx-ingress-controller Scheduled for sync
8.2 Ingress TLS 配置
要做加密之前首先來生成key:
[root@server2 ingress]# openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"
Generating a 2048 bit RSA private key
..........................+++
...+++
writing new private key to 'tls.key'
-----
[root@server2 ingress]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created
##將生成的證書和key 存起來
[root@server2 ingress]# kubectl get secrets
NAME TYPE DATA AGE
default-token-z4gbr kubernetes.io/service-account-token 3 9d
tls-secret kubernetes.io/tls 2 9s
對網站www1.westos.org 進行加密：
[root@server2 ingress]# vim ingress.yaml
[root@server2 ingress]# cat ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www1
spec:
tls:
- hosts:
- www1.westos.org
secretName: tls-secret
rules:
- host: www1.westos.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www1
spec:
rules:
- host: www2.westos.org
http:
paths:
- path: /
backend:
serviceName: nginx-svc
servicePort: 80
[root@server2 ingress]# kubectl apply -f ingress.yaml
[root@server2 ingress]# kubectl describe ingress
Name: ingress-www1
Namespace: default
Address: 172.25.25.3
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
tls-secret terminates www1.westos.org
Rules:
Host Path Backends
---- ---- --------
www1.westos.org
/ myservice:80 (10.244.1.34:80,10.244.2.42:80,10.244.2.43:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 13s (x2 over 13s) nginx-ingress-controller Scheduled for sync
Name: ingress-www2
Namespace: default
Address: 172.25.25.3
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
www2.westos.org
/ nginx-svc:80 (10.244.1.38:80,10.244.2.46:80,10.244.2.47:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 13s (x2 over 13s) nginx-ingress-controller Scheduled for sync
[root@server2 ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-www1 <none> www1.westos.org 172.25.25.3 80, 443 47s
ingress-www2 <none> www2.westos.org 172.25.25.3 80 47s
測驗：由于此時開啟了443，便會重定向到443，如果沒有開啟由于其是加密的便不能訪問；www2沒有做加密，直接回傳值，
[root@westos ~]# curl -I www1.westos.org
HTTP/1.1 308 Permanent Redirect
Date: Mon, 14 Jun 2021 03:32:05 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://www1.westos.org
[root@westos ~]# curl -I www2.westos.org
HTTP/1.1 200 OK
Date: Mon, 14 Jun 2021 03:55:54 GMT
Content-Type: text/html
Content-Length: 65
Connection: keep-alive
Last-Modified: Sun, 25 Feb 2018 06:04:32 GMT
ETag: "5a9251f0-41"
Accept-Ranges: bytes
8.3 Ingress 認證配置
[root@server2 ingress]# yum install -y httpd-tools
[root@server2 ingress]# htpasswd -c auth admin
New password:
Re-type new password:
Adding password for user admin
[root@server2 ingress]# htpasswd auth zxk
New password:
Re-type new password:
Adding password for user zxk
[root@server2 ingress]# cat auth
admin:$apr1$NAtpYX/0$bqGqb.8Vo7DqDCoILmUpv1
zxk:$apr1$zuNeydPF$nbL1qU65BmtgMMp9DGeAg0
[root@server2 ingress]# kubectl create secret generic basic-auth --from-file=auth
secret/basic-auth created
[root@server2 ingress]# kubectl get secrets
NAME TYPE DATA AGE
basic-auth Opaque 1 4m13s
default-token-z4gbr kubernetes.io/service-account-token 3 10d
tls-secret kubernetes.io/tls 2 42m
編輯檔案來器用認證：
[root@server2 ingress]# cat ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www1
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - westps'
spec:
tls:
- hosts:
- www1.westos.org
secretName: tls-secret
rules:
- host: www1.westos.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www2
spec:
rules:
- host: www2.westos.org
http:
paths:
- path: /
backend:
serviceName: nginx-svc
servicePort: 80
[root@server2 ingress]# kubectl apply -f ingress.yaml
8.4 Ingress地址重寫
- 當訪問
www2.westos.org時直接改寫為www2.westos.org/hostname.html；
[root@server2 ingress]# cat ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www1
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - westps'
spec:
tls:
- hosts:
- www1.westos.org
secretName: tls-secret
rules:
- host: www1.westos.org
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-www2
annotations: ##當訪問www2.westos.org時直接改寫為www2.westos.org/hostname.html
nginx.ingress.kubernetes.io/app-root: /hostname.html
spec:
rules:
- host: www2.westos.org
http:
paths:
- path: /
backend:
serviceName: nginx-svc
servicePort: 80
[root@server2 ingress]# kubectl apply -f ingress.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/ingress-www1 configured
ingress.networking.k8s.io/ingress-www2 configured
- 路徑重寫
[root@server2 ingress]# vim ingress-rewrite.yaml
[root@server2 ingress]# cat ingress-rewrite.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: ingress-rewrite
namespace: default
spec:
rules:
- host: rewrite.westos.org
http:
paths:
- backend:
serviceName: nginx-svc
servicePort: 80
path: /westos(/|$)(.*)
[root@server2 ingress]# kubectl apply -f ingress.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/ingress-www1 configured
ingress.networking.k8s.io/ingress-www2 configured
[root@server2 ingress]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10d
myservice ClusterIP 10.97.144.118 <none> 80/TCP 136m
nginx-svc ClusterIP 10.108.174.65 <none> 80/TCP 85m
外部訪問策略是，先來訪問 ingress 中的 svc開啟的pod 去 調度后端的 svc，
測驗：
[root@westos ~]# curl -I rewrite.westos.org/westos
HTTP/1.1 200 OK
Date: Mon, 14 Jun 2021 04:38:52 GMT
Content-Type: text/html
Content-Length: 65
Connection: keep-alive
Last-Modified: Sun, 25 Feb 2018 06:04:32 GMT
ETag: "5a9251f0-41"
Accept-Ranges: bytes
[root@westos ~]# curl -I rewrite.westos.org/westos/hostname.html
HTTP/1.1 200 OK
Date: Mon, 14 Jun 2021 04:39:03 GMT
Content-Type: text/html
Content-Length: 32
Connection: keep-alive
Last-Modified: Mon, 14 Jun 2021 03:11:07 GMT
ETag: "60c6c8cb-20"
Accept-Ranges: bytes
[root@westos ~]# curl -I rewrite.westos.org/westos/test.html
HTTP/1.1 404 Not Found
Date: Mon, 14 Jun 2021 04:39:11 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
[root@westos ~]# curl rewrite.westos.org/westos/
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[root@westos ~]# curl rewrite.westos.org/westos/hostname.html
deployment-www2-6bb947d8b-tnlhf
[root@westos ~]# curl rewrite.westos.org/westos/hostname.html
deployment-www2-6bb947d8b-9cbfp
常用的一些引數含義：
9. k8s 網路通信
- k8s通過CNI介面接入其他插件來實作網路通訊，目前比較流行的插件有flannel，calico等，
- CNI插件存放位置：# cat /etc/cni/net.d/10-flannel.conflist
插件使用的解決方案如下：
虛擬網橋，虛擬網卡，多個容器共用一個虛擬網卡進行通信，
多路復用：MacVLAN，多個容器共用一個物理網卡進行通信，
硬體交換：SR-LOV，一個物理網卡可以虛擬出多個介面，這個性能最好， - 容器間通信：同一個pod內的多個容器間的通信，通過lo即可實作；
- pod 之間的通信：
同一節點的pod之間通過cni網橋轉發資料包，
不同節點的pod之間的通信需要網路插件支持， - pod 和service通信: 通過iptables或ipvs實作通信，ipvs取代不了iptables，因為ipvs只能做負載均衡，而做不了nat轉換，
- pod 和外網通信：iptables的MASQUERADE，
- Service與集群外部客戶端的通信；（ingress、nodeport、loadbalancer）
9.1 flannel 網路
- 通信結構
前面用到的都是flannel，Flannel vxlan 模式跨主機通信原理：
-
VXLAN，即Virtual Extensible LAN（虛擬可擴展局域網），是Linux本身支持的一網種網路虛擬化技術，VXLAN可以完全在內核態實作封裝和解封裝作業，從而通過“隧道”機制，構建出覆寫網路（Overlay Network），
-
VTEP：VXLAN Tunnel End Point（虛擬隧道端點），在Flannel中 VNI的默認值是1，這也是為什么宿主機的VTEP設備都叫flannel.1的原因，
-
Cni0: 網橋設備，每創建一個pod都會創建一對 veth pair，其中一端是pod中的eth0，另一端是Cni0網橋中的埠（網卡），
-
Flannel.1: TUN設備(虛擬網卡)，用來進行 vxlan 報文的處理（封包和解包），不同node之間的pod資料流量都從overlay設備以隧道的形式發送到對端，
-
Flanneld：flannel在每個主機中運行flanneld作為agent，它會為所在主機從集群的網路地址空間中，獲取一個小的網段subnet，本主機內所有容器的IP地址都將從中分配，同時Flanneld監聽K8s集群資料庫，為flannel.1設備提供封裝資料時必要的mac、ip等網路資料資訊，
-
通信原理
當容器發送IP包，通過veth pair 發往cni網橋，再路由到本機的flannel.1設備進行處理，
VTEP設備之間通過二層資料幀進行通信，源VTEP設備收到原始IP包后，在上面加上一個目的MAC地址，封裝成一個內部資料幀，發送給目的VTEP設備，
內部資料楨，并不能在宿主機的二層網路傳輸，Linux內核還需要把它進一步封裝成為宿主機的一個普通的資料幀，承載著內部資料幀通過宿主機的eth0進行傳輸，
Linux會在內部資料幀前面，加上一個VXLAN頭，VXLAN頭里有一個重要的標志叫VNI，它是VTEP識別某個資料楨是不是應該歸自己處理的重要標識，
flannel.1設備只知道另一端flannel.1設備的MAC地址，卻不知道對應的宿主機地址是什么，在linux內核里面，網路設備進行轉發的依據，來自FDB的轉發資料庫，這個flannel.1網橋對應的FDB資訊，是由flanneld行程維護的，
linux內核在IP包前面再加上二層資料幀頭，把目標節點的MAC地址填進去，MAC地址從宿主機的ARP表獲取，
此時flannel.1設備就可以把這個資料幀從eth0發出去，再經過宿主機網路來到目標節點的eth0設備，目標主機內核網路堆疊會發現這個資料幀有VXLAN Header，并且VNI為1，Linux內核會對它進行拆包，拿到內部資料幀，根據VNI的值，交給本機flannel.1設備處理,flannel.1拆包，根據路由表發往cni網橋，最后到達目標容器，
flannel支持多種后端：
Vxlan：報文封裝，默認
Directrouting ：直接路由，跨網段使用vxlan，同網段使用 host-gw 模式，
host-gw： 主機網關，性能好，但只能在二層網路中，不支持跨網路， 如果有成千上萬的Pod，容易產生廣播風暴，不推薦；
UDP： 性能差，不推薦
配置flannel：
[root@server2 ~]# kubectl -n kube-system edit cm kube-flannel-cfg
30 "Backend": {
31 "Type": "host-gw"
32 }
configmap/kube-flannel-cfg edited
[root@server2 ~]# kubectl -n kube-system get pod | grep kube-flannel | awk '{system("kubectl -n kube-system delete pod "$1"")}'
[root@server2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.25.25.250 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.1.0 172.25.25.3 255.255.255.0 UG 0 0 0 eth0
10.244.2.0 172.25.25.4 255.255.255.0 UG 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.25.25.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
當訪問本地網路時直接走cni，1網段走eth0的172.25.25.3; 2 網段直接走eth0到172.25.25.4 ，
在所有結點上會生成主機網關；此模式的前提是所有節點在一個 vlan中，
修改配置資訊：
[root@server2 ~]# kubectl -n kube-system edit cm kube-flannel-cfg
27 net-conf.json: |
28 {
29 "Network": "10.244.0.0/16",
30 "Backend": {
31 "Type": "vxlan"
32 "Directrouting": true
33 }
表示再一個網段中使用的是 host-gw的模式，不在一個網段使用的是vxlan ，
9.2 calico網路插件
官網：https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises
- calico簡介：
flannel實作的是網路通信，calico的特性是在pod之間的隔離，
通過BGP路由，但大規模端點的拓撲計算和收斂往往需要一定的時間和計算資源，
純三層的轉發，中間沒有任何的NAT和overlay，轉發效率最好，
Calico 僅依賴三層路由可達，Calico 較少的依賴性使它能適配所有 VM、Container、白盒或者混合環境場景，
安裝calico：在安裝之前先清理之前插件的資訊，避免兩個之間沖突；
[root@server2 ~]# kubectl delete -f kube-flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy "psp.flannel.unprivileged" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deleted
[root@server2 ~]# kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
coredns-85ffb569d4-85kp7 1/1 Running 4 9d
coredns-85ffb569d4-bd579 1/1 Running 4 9d
etcd-server2 1/1 Running 4 9d
kube-apiserver-server2 1/1 Running 4 9d
kube-controller-manager-server2 1/1 Running 4 9d
kube-proxy-6f78h 1/1 Running 1 17h
kube-proxy-7jvkr 1/1 Running 1 17h
kube-proxy-9d5s7 1/1 Running 1 17h
kube-scheduler-server2 1/1 Running 4 9d
##在所有節點做個清理的動作；
[root@server2 ~]# cd /etc/cni/net.d/
[root@server2 net.d]# ls
10-flannel.conflist
[root@server2 net.d]# mv 10-flannel.conflist /mnt/
用命令 arp -an 查看主機的 mac地址.
- 部署 calico 插件
使用calico插件：
[root@server2 ~]# mkdir calico/
[root@server2 calico]# wget https://docs.projectcalico.org/manifests/calico.yaml
[root@server2 calico]# vim calico.yaml
根據檔案內容，來下載所需的鏡像放入私有軟體倉庫中；
[root@server1 ~]# docker pull docker.io/calico/cni:v3.19.1
[root@server1 ~]# docker pull docker.io/calico/pod2daemon-flexvol:v3.19.1
[root@server1 ~]# docker pull docker.io/calico/node:v3.19.1
[root@server1 ~]# docker pull docker.io/calico/kube-controllers:v3.19.1
[root@server1 ~]# docker images | grep calico
calico/node v3.19.1 c4d75af7e098 3 weeks ago 168MB
calico/pod2daemon-flexvol v3.19.1 5660150975fb 3 weeks ago 21.7MB
calico/cni v3.19.1 5749e8b276f9 3 weeks ago 146MB
calico/kube-controllers v3.19.1 5d3d5ddc8605 3 weeks ago 60.6MB
[root@server1 ~]# docker images | grep calico | awk '{system("docker tag "$1":"$2" reg.westos.org/"$1":"$2"")}'
##修該標簽
[root@server1 ~]# docker images | grep reg.westos.org\/calico | awk '{system("docker push "$1":"$2"")}'
##上傳
然后編輯檔案將鏡像路徑更改過來，然后在修改內容，
IPIP作業模式：適用于互相訪問的pod不在同一個網段中，跨網段訪問的場景，
BGP作業模式：適用于互相訪問的pod在同一個網段，適用于大型網路，
[root@server2 calico]# vim calico.yaml
3657 - name: CALICO_IPV4POOL_IPIP
3658 value: "off"
3683 - name: CALICO_IPV4POOL_CIDR
3684 value: "10.244.0.0/16"
此時在應用清單檔案之后，會看到calico的插件資訊；
[root@server2 calico]# kubectl apply -f calico.yaml
[root@server2 calico]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-784b4f4c9-c9w6c 1/1 Running 0 41s
calico-node-4rn49 1/1 Running 0 41s
calico-node-6nmvk 1/1 Running 0 41s
calico-node-9st7d 1/1 Running 0 41s
coredns-85ffb569d4-85kp7 1/1 Running 4 9d
coredns-85ffb569d4-bd579 1/1 Running 4 9d
etcd-server2 1/1 Running 4 9d
kube-apiserver-server2 1/1 Running 4 9d
kube-controller-manager-server2 1/1 Running 4 9d
kube-proxy-6f78h 1/1 Running 1 17h
kube-proxy-7jvkr 1/1 Running 1 17h
kube-proxy-9d5s7 1/1 Running 1 17h
kube-scheduler-server2 1/1 Running 4 9d
[root@server2 calico]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.25.25.250 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.1.0 172.25.25.3 255.255.255.255 UGH 0 0 0 eth0
10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1
10.244.2.0 172.25.25.4 255.255.255.255 UGH 0 0 0 eth0
10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 flannel.1
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.25.25.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
- calico 網路架構
Felix：監聽ECTD中心的存盤獲取事件，用戶創建pod后，Felix負責將其網卡、IP、MAC都設定好，然后在內核的路由表里面寫一條，注明這個IP應該到這張網卡，同樣如果用戶制定了隔離策略，Felix同樣會將該策略創建到ACL中，以實作隔離，
BIRD：一個標準的路由程式，它會從內核里面獲取哪一些IP的路由發生了變化，然后通過標準BGP的路由協議擴散到整個其他的宿主機上，讓外界都知道這個IP在這里，路由的時候到這里來，
- 網路策略
NetworkPolicy策略模型：控制某個 namespace 下的 pod 網路出入站規則;
官網：https://kubernetes.io/zh/docs/concepts/services-networking/network-policies/
- 限制訪問指定服務：
[root@server2 ingress]# vim ingress.yaml
##洗掉里面的認策略，用于測驗
[root@server2 ingress]# kubectl apply -f ingress.yaml
[root@server2 k8s]# vim deployment.yaml
[root@server2 k8s]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-example
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: myapp:v1
[root@server2 k8s]# kubectl apply -f deployment.yaml
deployment.apps/deployment-example created
[root@server2 k8s]# kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
demo 1/1 Running 1 7m27s run=demo
deployment-example-6456d7c676-4d52s 1/1 Running 0 20s app=nginx,pod-template-hash=6456d7c676
deployment-example-6456d7c676-b8wcl 1/1 Running 0 20s app=nginx,pod-template-hash=6456d7c676
deployment-example-6456d7c676-bf67v 1/1 Running 0 20s app=nginx,pod-template-hash=6456d7c676
[root@server2 k8s]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demo 1/1 Running 1 7m59s 10.244.22.0 server4 <none> <none>
deployment-example-6456d7c676-4d52s 1/1 Running 0 52s 10.244.22.2 server4 <none> <none>
deployment-example-6456d7c676-b8wcl 1/1 Running 0 52s 10.244.141.192 server3 <none> <none>
deployment-example-6456d7c676-bf67v 1/1 Running 0 52s 10.244.22.1 server4 <none> <none>
注：此處不能用之前flunnel創建的pod 來測驗.
[root@server2 calico]# vim policy.yaml
[root@server2 calico]# cat policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-nginx
spec:
podSelector:
matchLabels:
app: nginx
[root@server2 calico]# kubectl apply -f policy.yaml
networkpolicy.networking.k8s.io/deny-nginx created
[root@server2 calico]# kubectl get networkpolicies.
NAME POD-SELECTOR AGE
deny-nginx app=nginx 3m24s
[root@server2 calico]# kubectl describe networkpolicies. deny-nginx
Name: deny-nginx
Namespace: default
Created on: 2021-06-14 15:44:05 +0800 CST
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=nginx
Allowing ingress traffic:
<none> (Selected pods are isolated for ingress connectivity)
Not affecting egress traffic
Policy Types: Ingress
此時在測驗時標簽為nginx的被限制；
[root@server2 calico]# curl 10.244.22.0
^C
[root@server2 calico]#
[root@server2 calico]# kubectl run web --image=nginx
pod/web created
[root@server2 calico]# kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
deployment-example-6456d7c676-4d52s 1/1 Running 0 6m5s app=nginx,pod-template-hash=6456d7c676
deployment-example-6456d7c676-b8wcl 1/1 Running 0 6m5s app=nginx,pod-template-hash=6456d7c676
deployment-example-6456d7c676-bf67v 1/1 Running 0 6m5s app=nginx,pod-template-hash=6456d7c676
web 1/1 Running 0 11s run=web
[root@server2 calico]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-example-6456d7c676-4d52s 1/1 Running 0 6m27s 10.244.22.2 server4 <none> <none>
deployment-example-6456d7c676-b8wcl 1/1 Running 0 6m27s 10.244.141.192 server3 <none> <none>
deployment-example-6456d7c676-bf67v 1/1 Running 0 6m27s 10.244.22.1 server4 <none> <none>
web 1/1 Running 0 33s 10.244.22.3 server4 <none> <none>
[root@server2 calico]# curl 10.244.22.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
- 允許指定pod訪問服務:
[root@server2 calico]# vim policy.yaml
[root@server2 calico]# cat policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-nginx
spec:
podSelector:
matchLabels:
app: nginx
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-demo
spec:
podSelector:
matchLabels:
app: nginx
ingress:
- from:
- podSelector:
matchLabels:
run: demo
[root@server2 calico]# kubectl apply -f policy.yaml
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/access-demo created
[root@server2 calico]# curl 10.244.141.192
^C
[root@server2 calico]# kubectl run demo --image=busyboxplus -it
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.141.192
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
/ #
當前只有run: demo才能訪問 nginx 服務，其他默認還是被拒絕訪問，
- 禁止 namespace 中所有 Pod 之間的相互訪問
[root@server2 calico]# cat policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-nginx
spec:
podSelector:
matchLabels:
app: nginx
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-demo
spec:
podSelector:
matchLabels:
app: nginx
ingress:
- from:
- podSelector:
matchLabels:
run: demo
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: default
spec:
podSelector: {}
[root@server2 calico]# kubectl apply -f policy.yaml
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/access-demo unchanged
networkpolicy.networking.k8s.io/default-deny created
[root@server2 calico]# kubectl get networkpolicies.
NAME POD-SELECTOR AGE
access-demo app=nginx 5m19s
default-deny <none> 16s
deny-nginx app=nginx 13m
[root@server2 calico]# kubectl create namespace demo
namespace/demo created
[root@server2 calico]# kubectl get ns
NAME STATUS AGE
default Active 10d
demo Active 6s
ingress-nginx Active 6h50m
kube-node-lease Active 10d
kube-public Active 10d
kube-system Active 10d
metallb-system Active 18h
[root@server2 calico]# kubectl run demo --image=busyboxplus -it -n demo
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.141.192
^C
/ # 10.244.22.3
[root@server2 calico]# kubectl delete -f policy.yaml
networkpolicy.networking.k8s.io "deny-nginx" deleted
networkpolicy.networking.k8s.io "access-demo" deleted
networkpolicy.networking.k8s.io "default-deny" deleted
[root@server2 calico]# kubectl -n demo attach demo -it
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.22.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
- 禁止其他 namespace 訪問服務
策略之間可能會有干擾，刪掉之前的策略在來做實驗
[root@server2 calico]# kubectl delete -f policy.yaml
networkpolicy.networking.k8s.io "deny-nginx" deleted
networkpolicy.networking.k8s.io "access-demo" deleted
networkpolicy.networking.k8s.io "default-deny" deleted
[root@server2 calico]# vim policy2.yaml
[root@server2 calico]# cat policy2.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: deny-namespace
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}
[root@server2 calico]# kubectl apply -f policy2.yaml
networkpolicy.networking.k8s.io/deny-namespace created
[root@server2 calico]# kubectl -n demo attach demo -it
If you don't see a command prompt, try pressing enter.
此時不能訪問，
- 只允許指定namespace訪問服務
[root@server2 calico]# vim policy2.yaml
[root@server2 calico]# cat policy2.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: deny-namespace
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-namespace
spec:
podSelector:
matchLabels:
run: web
ingress:
- from:
- namespaceSelector:
matchLabels:
role: prod
[root@server2 calico]# kubectl label namespaces demo role=prod
namespace/demo labeled
[root@server2 calico]# kubectl apply -f policy2.yaml
networkpolicy.networking.k8s.io/deny-namespace configured
networkpolicy.networking.k8s.io/access-namespace created
[root@server2 calico]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-example-6456d7c676-4d52s 1/1 Running 0 39m 10.244.22.2 server4 <none> <none>
deployment-example-6456d7c676-b8wcl 1/1 Running 0 39m 10.244.141.192 server3 <none> <none>
deployment-example-6456d7c676-bf67v 1/1 Running 0 39m 10.244.22.1 server4 <none> <none>
web 1/1 Running 0 33m 10.244.22.3 server4 <none> <none>
[root@server2 calico]# kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
deployment-example-6456d7c676-4d52s 1/1 Running 0 39m app=nginx,pod-template-hash=6456d7c676
deployment-example-6456d7c676-b8wcl 1/1 Running 0 39m app=nginx,pod-template-hash=6456d7c676
deployment-example-6456d7c676-bf67v 1/1 Running 0 39m app=nginx,pod-template-hash=6456d7c676
web 1/1 Running 0 33m run=web
[root@server2 calico]# kubectl -n demo attach demo -it
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.22.2
^C
/ # curl 10.244.22.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
/ #
- 允許外網訪問服務
[root@server2 calico]# cat policy2.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: deny-namespace
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-namespace
spec:
podSelector:
matchLabels:
run: web
ingress:
- from:
- namespaceSelector:
matchLabels:
role: prod
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-allow-external
spec:
podSelector:
matchLabels:
app: nginx
ingress:
- ports:
- port: 80
from: []
[root@server2 calico]# kubectl apply -f policy2.yaml
networkpolicy.networking.k8s.io/deny-namespace configured
networkpolicy.networking.k8s.io/access-namespace unchanged
networkpolicy.networking.k8s.io/web-allow-external created
[root@westos ~]# curl www1.westos.org/hostname.html
deployment-example-6456d7c676-4d52s
[root@westos ~]# curl www1.westos.org/hostname.html
deployment-example-6456d7c676-b8wcl
更多的策略資訊看官網介紹：https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/288086.html
標籤：其他
上一篇：平面中判斷線段與矩形是否相交