k8s 部署 TCP node應用

之前我們已經成功的將node的http應用部署到k8s集群內，但某些業務不僅僅需要http應用還需要TCP長連接應用，本文將對部署tcp應用的步驟及問題進行總結
本文需要具備一定nodejs基礎，涉及框架：koa
為什么采用koa而不是express或其他web框架：koa輕量級，中間件按需加載
相關技術可參考：
centos8部署k8s:《k8s部署》
k8s部署web應用：《k8s部署 nodejs web應用，使用ingress-nginx映射公網》

k8s環境
linux： centos 8.x
k8s：1.21

$ kubectl get node -owide

NAME         STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-master   Ready    control-plane,master   11d   v1.21.0   172.16.66.167   <none>        CentOS Linux 8   4.18.0-305.19.1.el8_4.x86_64   docker://20.10.10
k8s-node1    Ready    <none>                 11d   v1.21.0   172.16.66.168   <none>        CentOS Linux 8   4.18.0-305.19.1.el8_4.x86_64   docker://20.10.10
k8s-node2    Ready    <none>                 11d   v1.21.0   172.16.66.170   <none>        CentOS Linux 8   4.18.0-305.19.1.el8_4.x86_64   docker://20.10.10

tcp專案準備
微服務集群中，單個業務除了需要對外提供服務外，不同業務之間還存在通信
比如玩家通過tcp服務連接進入我們的游戲服，此時官方通過后臺web服務給該用戶發放了一封郵件，此時就需要web服務通知到tcp服，那既然tcp有對外開放的介面，為什么web服不會通過tcp通信告訴tcp業務呢？
第一，由于該埠對外開放，暴露給公網，如果有人通過該tcp埠模擬發送業務訊息，會導致安全隱患，簡單說，tcp不應該處理玩家操作以外的業務，比如發放獎勵郵件，
第二，由于后臺操作頻率很低（遠遠達不到tcp操作的頻率），此處如果改用tcp長連接會有一定的資源浪費，http這種通知后就斷開的方式被廣泛用于業務通知，比如現在的三方支付（微信或支付寶）的異步通知就是通過http/https協議實作
考慮到這種情況，我們需要在tcp業務中，集成http/https業務

$ vim app/app.js

const net = require("net");
const path = require("path");
const fs = require("fs");
const Koa = require('koa');
const router = require('koa-router')();
const tools = require('./tools/tools')

class App {
    constructor(net) {
        this.env = process.argv[2] || 'dev';
        this.tcpPort = 10086;                       //tcp開放埠
        this.httpPort = 30086;                      //http開放埠
        this.net = net;                             //tcp實體
        this.app = new Koa();                       //koa實體
        this.server = null;

        //初始化
        this.init();
    }

    //初始化
    async init() {
        //啟動tcp server
        this.tcpRun();

        //啟動koa http server
        this.koaRun();
    }

    //tcp run
    tcpRun() {
        this.server = this.net.createServer();
        this.server.on('listening', () => {
            console.log('tcp server is listening on port', this.tcpPort);
        });
        this.server.on('connection', socket => {
            console.log('tcp has a new connection', socket.remoteAddress);
            socket.on('data', data => {
                data = data.toString();
                console.log(`client:${socket.remoteAddress}`, data);
            })
            this.server.getConnections((err, count) => {
                console.log(err, count);
            })
            // socket.end();
            // this.server.close();
        });
        this.server.on('close', () => {
            console.log('tcp server is now closed');
        });
        this.server.on('error', err => {
            console.log('Error occurred:', err.message);
        });
        this.server.listen(this.tcpPort);
    }

    //koa run
    koaRun() {
        this.app.use(router.routes(), router.allowedMethods());

        //路由配置
        router.get('/web/login', async (ctx, next) => {
            ctx.body = 'ok';
        })

        this.app.listen(this.httpPort);
    }
}

new App(net);

撰寫dockerfile檔案并構建鏡像
在app同級檔案夾創建Dockerfile檔案

$ vim Dockerfile

#運行環境
FROM node:10.15.1
#作者
MAINTAINER [SCH]
#將同級app檔案夾加入docker內
ADD app /opt/app
#指定作業目錄
WORKDIR /opt/app
#執行陳述句
RUN npm install
#指定對外開放埠
EXPOSE 10086 30086
#啟動命令
CMD ["nohup", "npm", "run", "dev", "&"]

在k8s-node1與k8s-node2節點構建鏡像：

#構建鏡像并命名為app v1版本
$ docker build -t app:v1 .

#查看鏡像
$ docker images

REPOSITORY                                                TAG       IMAGE ID       CREATED         SIZE
app                                                       v1        7d462447d52e   4 days ago      899MB
calico/node                                               v3.20.1   355c1ee44040   8 weeks ago     156MB
calico/pod2daemon-flexvol                                 v3.20.1   55fa5eb71e09   8 weeks ago     21.7MB
calico/cni                                                v3.20.1   e69ccb66d1b6   8 weeks ago     146MB
k8s.gcr.io/ingress-nginx/controller                       v0.48.1   ac0e4fe3e6b0   4 months ago    279MB
registry.aliyuncs.com/google_containers/kube-proxy        v1.21.0   38ddd85fe90e   7 months ago    122MB
registry.aliyuncs.com/google_containers/pause             3.4.1     0f8457a4c2ec   10 months ago   683kB
coredns/coredns                                           1.8.0     296a6d5035e2   12 months ago   42.5MB
registry.aliyuncs.com/google_containers/coredns/coredns   v1.8.0    296a6d5035e2   12 months ago   42.5MB
node                                                      10.15.1   8fc2110c6978   2 years ago     897MB

可以看到app已經成功構建為docker 鏡像

撰寫yaml檔案部署app容器到k8s

$ vim app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  selector:
    matchLabels:
      app: app
  replicas: 6                          #啟動6個app容器副本
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: app
        ports:
        - containerPort: 10086         #對外開放埠 10086與30086
        - containerPort: 30086

---

apiVersion: v1         
kind: Service                          #使用service統一副本的入口
metadata:
  name: app-service
spec:
  selector:
    app: app
  ports:
  - protocol: TCP
    name: tcp
    port: 10086
    targetPort: 10086
  - protocol: TCP
    name: http
    port: 30086
    targetPort: 30086

$ kubectl apply -f app.yaml

$ kubectl get pod -owide

NAME                                     READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
app-deployment-56d8495cd8-6b4jt          1/1     Running   0          3d23h   10.244.36.91     k8s-node1   <none>           <none>
app-deployment-56d8495cd8-cphqj          1/1     Running   0          3d23h   10.244.36.92     k8s-node1   <none>           <none>
app-deployment-56d8495cd8-dtpkx          1/1     Running   0          3d23h   10.244.169.144   k8s-node2   <none>           <none>
app-deployment-56d8495cd8-q59xm          1/1     Running   0          3d23h   10.244.36.90     k8s-node1   <none>           <none>
app-deployment-56d8495cd8-vphbn          1/1     Running   0          3d23h   10.244.169.145   k8s-node2   <none>           <none>
app-deployment-56d8495cd8-w4g2g          1/1     Running   1          3d23h   10.244.36.89     k8s-node1   <none>           <none>

此時已經成功部署app到兩個節點內

部署ingress規則

$ vim app-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
spec:
  rules:
    - host: k8s.test.com            #指定域名
      http:
        paths:
        - path: /tcp                  #一級路由        
          pathType: Prefix            #匹配規則 Prefix：前綴
          backend:
            service:
              name: app-service        #指向的service
              port:
                number: 30086          #對應的service暴露的埠

$ kubectl apply -f app-ingress.yaml

此時我們已經將http30086埠開放對外，后面如果該業務只用于接收內部請求，可以通過阿里云安全組或linux防火墻對該埠的請求IP進行管控

部署tcp-ingress tcp配置引數

$ vim tcp.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  10086: "default/app-service:10086"  #埠名："ns名/service名：埠"

$ kubectl apply -f tcp.yaml

修改ingress-controller配置
在args添加- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

$ ingress.yaml

# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
#kind: Deployment
#apiVersion: extensions/v1beta1
# 修改為DaemonSet型別，隨每個node節點創建和洗掉，配合污點容忍可以實作ingress-nginx高可用
kind: DaemonSet
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      #開啟本機網路
      hostNetwork: true
      containers:
        - name: controller
          image: k8s.gcr.io/ingress-nginx/controller:v0.48.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

$ kubectl apply -f ingress.yaml

此時我們已經將10086埠通過ingress-nginx配置對外開放，在本地執行tcp連接測驗連接到
k8s.scbczx.com:10086已經成功訪問
若有不正確，歡迎指出

轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/357069.html

標籤：其他

上一篇：第十一周-周報

下一篇：自動化測驗之流量回放技術