目錄
文章目錄
- 目錄
- 服務行程
- 單點部署
- 關鍵啟動選型清單
- 集群部署
- 靜態配置集群
服務行程
etcd 目前默認使用 2379 埠提供 HTTP API 服務,2380 埠提供 Peer 通信(這兩個埠已經被 IANA 官方預留給 etcd),在之前的版本中,可能會分別使用 4001 和 7001,在使用的程序中需要注意這個區別,
雖然 etcd 也支持單點部署,但是在生產環境中推薦集群方式部署,一般 etcd 節點數會選擇 3、5、7,etcd 會保證所有的節點都會保存資料,并保證資料的一致性和正確性,
單點部署
因為 etcd 是 Golang 撰寫的,安裝只需要下載對應的二進制檔案,并放到合適的路徑就行,如果在測驗環境,啟動一個單點的 etcd 服務,只需要運行 etcd 執行即可,
git clone https://github.com/etcd-io/etcd.git
cd etcd
./build
使用 build 腳本構建會在當前專案的 bin 目錄生產 etcd 和 etcdctl 可執行程式,etcd 就是 etcd Server,而 etcdctl 主要為 etcd Server 提供指令行操作,
查看版本:
$ ./bin/etcd --version
etcd Version: 3.5.0-pre
Git SHA: ab4cc3cae
Go Version: go1.14.4
Go OS/Arch: darwin/amd64
$ ./bin/etcdctl version
etcdctl version: 3.5.0-pre
API version: 3.5
啟動 etcd Server:
$ ./bin/etcd
{"level":"info","ts":"2020-10-04T07:39:14.751+0800","caller":"etcdmain/etcd.go:69","msg":"Running: ","args":["./bin/etcd"]}
{"level":"info","ts":"2020-10-04T07:39:14.751+0800","caller":"etcdmain/etcd.go:94","msg":"failed to detect default host","error":"default host not supported on darwin_amd64"}
{"level":"warn","ts":"2020-10-04T07:39:14.751+0800","caller":"etcdmain/etcd.go:99","msg":"'data-dir' was empty; using default","data-dir":"default.etcd"}
{"level":"info","ts":"2020-10-04T07:39:14.751+0800","caller":"embed/etcd.go:113","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2020-10-04T07:39:14.752+0800","caller":"embed/etcd.go:121","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2020-10-04T07:39:14.753+0800","caller":"embed/etcd.go:266","msg":"starting an etcd server","etcd-version":"3.5.0-pre","git-sha":"ab4cc3cae","go-version":"go1.14.4","go-os":"darwin","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"default","data-dir":"default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"default.etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=http://localhost:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""}
{"level":"info","ts":"2020-10-04T07:39:14.764+0800","caller":"etcdserver/backend.go:78","msg":"opened backend db","path":"default.etcd/member/snap/db","took":"9.908726ms"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"etcdserver/raft.go:444","msg":"starting local member","local-member-id":"8e9e05c52164694d","cluster-id":"cdf818194e3a8c32"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"raft/raft.go:1528","msg":"8e9e05c52164694d switched to configuration voters=()"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"raft/raft.go:701","msg":"8e9e05c52164694d became follower at term 0"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"raft/raft.go:383","msg":"newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2020-10-04T07:39:14.863+0800","caller":"raft/raft.go:701","msg":"8e9e05c52164694d became follower at term 1"}
{"level":"info","ts":"2020-10-04T07:39:14.863+0800","caller":"raft/raft.go:1528","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"warn","ts":"2020-10-04T07:39:14.888+0800","caller":"auth/store.go:1231","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2020-10-04T07:39:14.912+0800","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2020-10-04T07:39:14.924+0800","caller":"etcdserver/server.go:752","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.0-pre","cluster-version":"to_be_decided"}
{"level":"info","ts":"2020-10-04T07:39:14.925+0800","caller":"etcdserver/server.go:640","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"warn","ts":"2020-10-04T07:39:14.925+0800","caller":"etcdserver/metrics.go:212","msg":"failed to get file descriptor usage","error":"cannot get FDUsage on darwin"}
{"level":"info","ts":"2020-10-04T07:39:14.925+0800","caller":"raft/raft.go:1528","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2020-10-04T07:39:14.925+0800","caller":"membership/cluster.go:385","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2020-10-04T07:39:14.927+0800","caller":"embed/etcd.go:513","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2020-10-04T07:39:14.927+0800","caller":"embed/etcd.go:235","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2020-10-04T07:39:15.866+0800","caller":"raft/raft.go:788","msg":"8e9e05c52164694d is starting a new election at term 1"}
{"level":"info","ts":"2020-10-04T07:39:15.866+0800","caller":"raft/raft.go:714","msg":"8e9e05c52164694d became candidate at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.866+0800","caller":"raft/raft.go:848","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.867+0800","caller":"raft/raft.go:766","msg":"8e9e05c52164694d became leader at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.867+0800","caller":"raft/node.go:327","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.868+0800","caller":"etcdserver/server.go:2285","msg":"setting up initial cluster version","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"membership/cluster.go:523","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"embed/serve.go:97","msg":"ready to serve client requests"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdserver/server.go:2305","msg":"cluster version is updated","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdserver/server.go:1863","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://localhost:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdmain/main.go:47","msg":"notifying init daemon"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdmain/main.go:53","msg":"successfully notified init daemon"}
{"level":"info","ts":"2020-10-04T07:39:15.877+0800","caller":"embed/serve.go:139","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2379"}
- name 表示節點名稱,默認為 default,
- data-dir 表示 WAL 日志和 Snapshot 資料儲存目錄,默認為 ./default.etcd/ 目錄,
- 使用 http://localhost:2380 和 etcd Cluster 中其他節點通信,
- 使用 http://localhost:2379 提供 HTTP API 服務,與客戶端通信,
- heartbeat 為 100ms,表示 Leader 多久發送一次心跳到所有 Followers,
- election-timeout 為 1s,該引數的作用是重新投票的超時時間,如果 Follow 在該時間間隔內沒有收到 Leader 發出的心跳包,就會觸發重新投票,
- snapshot-count 為 100000,該引數的作用是指定有多少次事務被提交后觸發快照截取動作并持久化到磁盤,
- cluster-id 為 cdf818194e3a8c32,
- raft.node 為 8e9e05c52164694d,
- 啟動的時候,會運行 Raft,選舉出 Leader:
elected leader 8e9e05c52164694d at term 2,
上述方法只是簡單的啟動了一個 etcd Server,當然,在生產環境中,通常使用 Systemd 來進行管理,
- 建立相關目錄:
$ mkdir -p /var/lib/etcd/
$ mkdir -p /etc/etcd/config/
- 設定 etcd 組態檔:
$ cat <<EOF | sudo tee /etc/etcd/config/etcd.conf
# 節點名稱
ETCD_NAME=$(hostname -s)
# 資料存放路徑
ETCD_DATA_DIR=/var/lib/etcd
EOF
- 創建 systemd 組態檔:
$ cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target
[Service]
User=root
Type=notify
EnvironmentFile=-/opt/etcd/config/etcd.conf
ExecStart=~/workspace/etcd/bin
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target
EOF
- 啟動 etcd Server:
$ systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
關鍵啟動選型清單
- –name:指定 etcd Node 名稱,可以使用 hostname,
- –data-dir:指定 etcd Server 持久化資料存盤目錄路徑,
- –snapshot-count:指定有多少事務(transaction)被提交后,觸發截取快照并持久化到磁盤,
- –heartbeat-interval:指定 Leader 多久發送一次心跳到 Followers,
- –eletion-timeout:指定重新投票的超時時間,如果 Follow 在該時間間隔沒有收到 Leader 發出的心跳包,則會觸發重新投票,
- –listen-peer-urls:指定和 Cluster 其他 Node 通信的地址,比如:http://IP:2380,如果有多個,則使用逗號分隔,需要所有節點都能夠訪問,所以不要使用 localhost,
- –listen-client-urls:指定對外提供服務的地址,比如:http://IP:2379,http://127.0.0.1:2379,
- –advertise-client-urls:對外通告的該節點的客戶端監聽地址,會告訴集群中其他節點,
- –initial-advertise-peer-urls:對外通告該節點的同伴(Peer)監聽地址,這個值會告訴集群中其他節點,
- –initial-cluster:指定集群中所有節點的資訊,通常為 IP:Port 資訊,格式為:node1=http://ip1:2380,node2=http://ip2:2380,…,注意,這里的 node1 就是 --name 指定的名字,ip1:2380 就是 --initial-advertise-peer-urls 指定的值,
- –initial-cluster-state:新建集群時,這個值為 new;假如已經存在了集群,這個值為 existing,
- –initial-cluster-token:創建集群的 token,這個值每個集群保持唯一,這樣的話,如果你要重新創建集群,即使配置和之前一樣,也會再次生成新的集群和節點 UUID;否則會導致多個集群之間的沖突,造成未知的錯誤,
集群部署
etcd Cluster 的部署,實際上就是多個主機上都部署 etcd Server,然后將它們加入到一個 Cluster 中,
在安裝和啟動 etcd 服務行程的時候,各個 Node 都需要知道 Cluster 中其他 Nodes 的資訊,一般是 IP:Port 資訊,根據用戶是否提前知曉(規劃)了每個 Node 的 IP 地址,有以下幾種不同的集群部署方案:
- 靜態配置:在啟動 etcd Server 的時候,通過 --initial-cluster 引數配置好所有的節點資訊,
- 注冊到已有的 etcd Cluster:比如官方提供的 discovery.etcd.io,
- 使用 DNS 啟動,
靜態配置集群
如果 etcd Cluster 中的成員是已知的,且具有固定的 IP 地址,就可以靜態的初始化一個集群,
每個 Node 都使用如下環境變數:
ETCD_INITIAL_CLUSTER="radon=http://10.0.2.1:2380,neon=http://10.0.3.1:2380"
ETCD_INITIAL_CLUSTER_STATE=new
或者使用如下指令行引數來指定集群成員:
--initial-cluster radon=http://10.0.2.1:2380,neon=http://10.0.3.1:2380
--initial-cluster-state new
初始化集群:
etcd --name radon --initial-advertise-peer-urls http://10.0.2.1:2380
--listen-peer-urls http://10.0.2.1:2380
--listen-client-urls http://10.0.2.1:2379,http://127.0.0.1:2379
--advertise-client-urls http://10.0.2.1:2380
--initial-cluster-token etcd.gmem.cc
--initial-cluster radon=http://10.0.2.1:2380,neon=http://10.0.3.1:2380
--initial-cluster-state new
注:所有以 --initial-cluster* 開頭的選項,在第一次運行(Bootstrap)后都被忽略,
使用 TLS 加密,etcd 支持基于 TLS 加密的集群內部、集群外部(客戶端與集群之間)的安全通信,每個集群節點都應該擁有被共享 CA 簽名的證書:
# 密鑰對、證書簽名請求
openssl genrsa -out radon.key 2048
export SAN_CFG=$(printf "\n[SAN]\nsubjectAltName=IP:127.0.0.1,IP:10.0.2.1,DNS:radon.gmem.cc")
openssl req -new -sha256 -key radon.key -out radon.csr \
-subj "/C=CN/ST=BeiJing/O=Gmem Studio/CN=Server Radon" \
-reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(echo $SAN_CFG))
# 執行簽名
openssl x509 -req -sha256 -in radon.csr -out radon.crt -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -days 3650 \
-extensions SAN -extfile <(echo "${SAN_CFG}")
初始化集群命令需要修改為:
etcd --name radon --initial-advertise-peer-urls https://10.0.2.1:2380
--listen-peer-urls https://10.0.2.1:2380
--listen-client-urls https://10.0.2.1:2379,https://127.0.0.1:2379
--advertise-client-urls https://10.0.2.1:2380
--initial-cluster-token etcd.gmem.cc
--initial-cluster radon=https://10.0.2.1:2380,neon=https://10.0.3.1:2380 # 指定集群成員串列
--initial-cluster-state new # 初始化新集群時使用
--initial-cluster-state existing # 加入已有集群時使用
# 客戶端 TLS 相關引數
--client-cert-auth
--trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt
--cert-file=/opt/etcd/cert/radon.crt
--key-file=/opt/etcd/cert/radon.key
# 集群內部 TLS 相關引數
--peer-client-cert-auth
--peer-trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt
--peer-cert-file=/opt/etcd/cert/radon.crt
--peer-key-file=/opt/etcd/cert/radon.key
轉載請註明出處,本文鏈接:https://www.uj5u.com/houduan/158965.html
標籤:python
