一、Redis Cluster 作業原理
在引入哨兵機制后，解決了Redis主從架構Master故障時的主從切換問題，保證了Redis服務可用性，但依舊無法解決單機節點出現的寫入性能瓶頸（網卡速率、單機記憶體容量、并發數量）
1、早期為解決單機性能瓶頸問題采用的解決方案：
1、客戶端分片：由客戶端程式進行讀寫key的redis節點判斷和分配，并且由客戶端自行處理讀寫請求分配、高可用管理及故障轉移操作
2、proxy代理模式：引入第三方代理程式，客戶端通過連接proxy代理服務器對資料進行讀寫，由proxy程式進行讀寫判斷分配，并對集群節點進行管理，但導致proxy又出現單點故障風險，并增加了一層資料處理環節
redis3.0 之后官方推出了無中心化的Cluster集群機制，集群中的每個節點單獨保持當前節點資料和集群狀態資訊，每個節點需要和其余全部節點保持連接，
2、Redis Cluster作業特點
1、Redis cluster集群中的各個節點之間（采用ping機制）進行互聯
2、集群中的某個節點失效，是有整個集群中超過半數的節點監測失效（sdown），才會真正判定為節點不可用（odown）
3、客戶端無需配置連接proxy即可直連redis，應用程式需配置集群的所有redis節點IP
4、redis cluster將集群中所有的節點平均映射到0-16383（16384個）slot槽位上，將資料庫的讀寫全部映射到對應的節點上操作，因此每增加n個集群主節點就能對redis進行n倍擴展，每個主節點將分擔16384/n個slot
5、redis cluster預先將16384個槽位分配至集群中的主節點，當接收到對redis的寫入請求時，會通過 CRC16(key)%16384 計算出當前key的所屬slot編號，從而將資料寫入到對應的主節點上，從而解決單機產生的性能瓶頸
3、Redis Cluster 架構設計
Redis Cluster 基本架構
Redis Cluster 主從架構
為解決Cluster集群中各Master節點的高可用性，可采用Cluster主從架構部署，將部分加入集群的節點設定為Slave角色，從而解決Master節點的高可用性
Cluster可自行實作類似哨兵機制的每組Master-Slave的主從切換，無需額外配置哨兵機制，并且支持每組節點的 1-n 模式（不支持1-n-n）
Redis Cluster 部署架構設計
3、Redis Cluster局限性
1、大多數時候客戶端連接性能會“降低”，由于cluster在讀寫資料時需要先進行solt計算并重定向訪問到對應節點上，造成了資料處理“延遲性”，
2、部分命令無法跨節點使用：mget、keys、scan、flush、sinter 等
3、客戶端維護復雜：SDK 和 應用自身消耗增加（更多的資料連接池）
4、不支持多資料庫：cluster 模式下僅支持使用db0
5、復制僅支持一層：不支持級聯復制
6、key事務、Lua 的支持有限：key操作必須在同一個節點上，事務和Lua無法跨節點使用
二、Redis Cluster 相關配置詳解
################################ REDIS CLUSTER ###############################
# Normal Redis instances can't be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
#
# cluster-enabled yes #啟用Cluster模式
# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system do not have
# overlapping cluster configuration file names.
#
# cluster-config-file nodes-6379.conf #Cluster節點組態檔名稱
# Cluster node timeout is the amount of milliseconds a node must be unreachable
# for it to be considered in failure state.
# Most other internal time limits are multiple of the node timeout.
#
# cluster-node-timeout 15000 #Cluster節點超時時間，單位：毫秒
# A replica of a failing master will avoid to start a failover if its data
# looks too old.
#
# There is no simple way for a replica to actually have an exact measure of
# its "data age", so the following two checks are performed:
#
# 1) If there are multiple replicas able to failover, they exchange messages
# in order to try to give an advantage to the replica with the best
# replication offset (more data from the master processed).
# Replicas will try to get their rank by offset, and apply to the start
# of the failover a delay proportional to their rank.
#
# 2) Every single replica computes the time of the last interaction with
# its master. This can be the last ping or command received (if the master
# is still in the "connected" state), or the time that elapsed since the
# disconnection with the master (if the replication link is currently down).
# If the last interaction is too old, the replica will not try to failover
# at all.
#
# The point "2" can be tuned by user. Specifically a replica will not perform
# the failover if, since the last interaction with the master, the time
# elapsed is greater than:
#
# (node-timeout * replica-validity-factor) + repl-ping-replica-period
#
# So for example if node-timeout is 30 seconds, and the replica-validity-factor
# is 10, and assuming a default repl-ping-replica-period of 10 seconds, the
# replica will not try to failover if it was not able to talk with the master
# for longer than 310 seconds.
#
# A large replica-validity-factor may allow replicas with too old data to failover
# a master, while a too small value may prevent the cluster from being able to
# elect a replica at all.
#
# For maximum availability, it is possible to set the replica-validity-factor
# to a value of 0, which means, that replicas will always try to failover the
# master regardless of the last time they interacted with the master.
# (However they'll always try to apply a delay proportional to their
# offset rank).
#
# Zero is the only value able to guarantee that when all the partitions heal
# the cluster will always be able to continue.
#
# cluster-replica-validity-factor 10
# 集群副本有效性因子（0表示slave將始終嘗試故障轉移為master，默認被注釋掉）
# (node-timeout * replica-validity-factor) + repl-ping-replica-period
# 計算時間（秒），如果slave無法與master通信的時間超過了該時間，則slave將不會嘗試故障轉移
# Cluster replicas are able to migrate to orphaned masters, that are masters
# that are left without working replicas. This improves the cluster ability
# to resist to failures as otherwise an orphaned master can't be failed over
# in case of failure if it has no working replicas.
#
# Replicas migrate to orphaned masters only if there are still at least a
# given number of other working replicas for their old master. This number
# is the "migration barrier". A migration barrier of 1 means that a replica
# will migrate only if there is at least 1 other working replica for its master
# and so forth. It usually reflects the number of replicas you want for every
# master in your cluster.
#
# Default is 1 (replicas migrate only if their masters remain with at least
# one replica). To disable migration just set it to a very large value.
# A value of 0 can be set but is useful only for debugging and dangerous
# in production.
#
# cluster-migration-barrier 1 #集群遷移屏障，僅當master至少有1個其他作業副本時，副本才會遷移
# By default Redis Cluster nodes stop accepting queries if they detect there
# is at least an hash slot uncovered (no available node is serving it).
# This way if the cluster is partially down (for example a range of hash slots
# are no longer covered) all the cluster becomes, eventually, unavailable.
# It automatically returns available as soon as all the slots are covered again.
#
# However sometimes you want the subset of the cluster which is working,
# to continue to accept queries for the part of the key space that is still
# covered. In order to do so, just set the cluster-require-full-coverage
# option to no.
#
# cluster-require-full-coverage yes # 集群需要全覆寫（默認為 yes，被注釋掉）
# 如果redis集群節點檢測到至少有一個未覆寫的哈希槽（沒有可用的節點提供服務），則會停止接受查詢
# 如果集群部分關閉（例如不再覆寫一系列哈希槽），整個集群將不可用
# 當所有插槽再次被覆寫，集群將自動回傳可用狀態
# This option, when set to yes, prevents replicas from trying to failover its
# master during master failures. However the master can still perform a
# manual failover, if forced to do so.
#
# This is useful in different scenarios, especially in the case of multiple
# data center operations, where we want one side to never be promoted if not
# in the case of a total DC failure.
#
# cluster-replica-no-failover no #主節點故障期間不啟用故障轉移
# In order to setup your cluster make sure to read the documentation
# available at http://redis.io web site.
########################## CLUSTER DOCKER/NAT support ########################
# In certain deployments, Redis Cluster nodes address discovery fails, because
# addresses are NAT-ted or because ports are forwarded (the typical case is
# Docker and other containers).
#
# In order to make Redis Cluster working in such environments, a static
# configuration where each node knows its public address is needed. The
# following two options are used for this scope, and are:
#
# * cluster-announce-ip
# * cluster-announce-port
# * cluster-announce-bus-port
#
# Each instruct the node about its address, client port, and cluster message
# bus port. The information is then published in the header of the bus packets
# so that other nodes will be able to correctly map the address of the node
# publishing the information.
#
# If the above options are not used, the normal Redis Cluster auto-detection
# will be used instead.
#
# Note that when remapped, the bus port may not be at the fixed offset of
# clients port + 10000, so you can specify any port and bus-port depending
# on how they get remapped. If the bus-port is not set, a fixed offset of
# 10000 will be used as usually.
#
# Example:
#
# cluster-announce-ip 10.1.1.5
# cluster-announce-port 6379
# cluster-announce-bus-port 6380
# 配置集群節點的公共地址（默認被注釋掉，使用redis群集自動檢測）
# 節點的ip地址，客戶端埠，集群訊息總線埠
三、Redis Cluster部署
0、基礎環境準備
所有redis節點采用相同硬體配置，相同密碼，清空所有資料
| IP | 角色 | slot | |
| node1 | 10.0.0.20 | M1 | 0-5460 |
| node2 | 10.0.0.21 | M2 | 5461-10922 |
| node3 | 10.0.0.22 | M3 | 10923-16383 |
| node4 | 10.0.0.23 | 集群擴容縮容備用（M4） | |
| node5 | 10.0.0.30 | S1 | |
| node6 | 10.0.0.31 | S2 | |
| node7 | 10.0.0.32 | S3 | |
| node8 | 10.0.0.33 | 集群擴容縮容備用（S4） |
初始化安裝6臺 redis服務，并啟用Cluster模式
[root@Redis-Ubuntu1804-node1:~]# apt install redis -y [root@Redis-Ubuntu1804-node1:~]# sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth redis' -e '/# requirepass/a requirepass redis' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file/a cluster-config-file nodes-6379.conf' -e '/# cluster-require-full-coverage yes/a cluster-require-full-coverage yes' /etc/redis/redis.conf [root@Redis-Ubuntu1804-node1:~]# systemctl restart redis [root@Redis-Ubuntu1804-node1:~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* LISTEN 0 128 0.0.0.0:16379 0.0.0.0:* LISTEN 0 128 0.0.0.0:6379 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 128 [::1]:6010 [::]:* [root@Redis-Ubuntu1804-node1:~]# ps -ef | grep redis redis 2098 1 0 01:38 ? 00:00:00 /usr/bin/redis-server 0.0.0.0:6379 [cluster] root 2112 1236 0 01:39 pts/0 00:00:00 grep --color=auto redis [root@Redis-Ubuntu1804-node1:~]#
1、基本模式手動部署
基本步驟
1、各節點服務器上安裝redis服務，啟用Cluster配置
2、通過meet 將節點添加至集群中
3、為各Master節點分配槽位 0~16383 ，必須全部分配完成，集群才可用
4、指定各節點主從關系
1.1、查看當前節點狀態資訊
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster nodes 98bb2740b622645ff51fd07bc994bb22e94feaaa :6379@16379 myself,master - 0 0 0 connected [root@Redis-Ubuntu1804-node1:~]#
1.2、將所有節點加入到當前集群中
Cluster節點操作相關命令
1、加入節點：將 ip 和 port 所指定的節點添加到集群當中，讓它成為集群的一份子，
1）CLUSTER MEET <ip:port>
2、移除節點：
1）CLUSTER FORGET <node_id>
2)、redis-trib.rb del-node <ip> <port> <node_id>
3、設定主從節點：
CLUSTER REPLICATE <node_id> 將當前節點設定為 node_id 指定的節點的從節點，
4、節點資料備份到硬碟：
CLUSTER SAVECONFIG 將節點的組態檔保存到硬碟里面，
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.20 6379 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.20 6379 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.21 6379 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.22 6379 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.30 6379 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.31 6379 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster meet 10.0.0.32 6379 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster nodes 98bb2740b622645ff51fd07bc994bb22e94feaaa 10.0.0.20:6379@16379 myself,master - 0 1681926433000 1 connected 9fe09d53d884c6c99c926a0e7213cf5386987ebf 10.0.0.32:6379@16379 master - 0 1681926434205 5 connected 55fd3a5d21e026b6503a89f1d88996b2a67c314a 10.0.0.31:6379@16379 master - 0 1681926435212 4 connected 6717c6b5a6170963ac251a1f14b0c600e917aa26 10.0.0.30:6379@16379 master - 0 1681926433000 3 connected d9c17ded7fb139f9eb7fd41e476bb8e17c8b9bed 10.0.0.21:6379@16379 master - 0 1681926434000 0 connected bdec3c92bcaa00c8ad1850cd477eb0a8f8a28aa2 10.0.0.22:6379@16379 master - 0 1681926433199 2 connected [root@Redis-Ubuntu1804-node1:~]#
1.3、查看當前集群資訊
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster info cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:0 cluster_current_epoch:5 cluster_my_epoch:1 cluster_stats_messages_ping_sent:70 cluster_stats_messages_pong_sent:89 cluster_stats_messages_meet_sent:6 cluster_stats_messages_sent:165 cluster_stats_messages_ping_received:88 cluster_stats_messages_pong_received:76 cluster_stats_messages_meet_received:1 cluster_stats_messages_received:165 [root@Redis-Ubuntu1804-node1:~]#
1.4、為集群節點分配槽位
Cluster 槽位相關命令
CLUSTER ADDSLOTS <slot> [slot ...] 將一個或多個槽（slot）指派（assign）給當前節點，CLUSTER DELSLOTS <slot> [slot ...] 移除一個或多個槽對當前節點的指派，
CLUSTER FLUSHSLOTS 移除指派給當前節點的所有槽，讓當前節點變成一個沒有指派任何槽的節點，
CLUSTER SETSLOT <slot> NODE <node_id> 將槽 slot 指派給 node_id 指定的節點，如果槽已經指派給另一個節點，那么先讓另一個節點洗掉該槽>，然后再進行指派，
CLUSTER SETSLOT <slot> MIGRATING <node_id> 將本節點的槽 slot 遷移到 node_id 指定的節點中，
CLUSTER SETSLOT <slot> IMPORTING <node_id> 從 node_id 指定的節點中匯入槽 slot 到本節點，
CLUSTER SETSLOT <slot> STABLE 取消對槽 slot 的匯入（import）或者遷移（migrate），
使用命令 redis-cli -h <目標節點IP> -p <目標節點埠> -a <目標節點密碼> --no-auth-warning cluster addslots <要分配的槽位號> 進行槽位分配
[root@Client-Ubuntu-1804-250:~/script/redis]# cat add_slots.sh #!/bin/bash # #******************************************************************** #Author: janzen #Date: 2023-04-20 #FileName： add_slots.sh #Description： The test script #Copyright (C): 2023 All rights reserved #******************************************************************** host=$1 port=$2 start=$3 end=$4 passwd=redis for slot in `seq $start $end`; do redis-cli -h $host -p $port -a $passwd --no-auth-warning cluster addslots $slot && echo $slot add to node $host:$port done [root@Client-Ubuntu-1804-250:~/script/redis]# [root@Client-Ubuntu-1804-250:~/script/redis]# ./add_slots.sh 10.0.0.20 6379 0 5460 >> addslot.log [root@Client-Ubuntu-1804-250:~/script/redis]# ./add_slots.sh 10.0.0.21 6379 5461 10922 >> addslot.log [root@Client-Ubuntu-1804-250:~/script/redis]# ./add_slots.sh 10.0.0.22 6379 10923 16383 >> addslot.log
##查看當前集群資訊，槽位已完成分配
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster nodes 98bb2740b622645ff51fd07bc994bb22e94feaaa 10.0.0.20:6379@16379 myself,master - 0 1681929286000 1 connected 0-5460 55fd3a5d21e026b6503a89f1d88996b2a67c314a 10.0.0.31:6379@16379 master - 0 1681929287000 4 connected 9fe09d53d884c6c99c926a0e7213cf5386987ebf 10.0.0.32:6379@16379 master - 0 1681929287000 5 connected 6717c6b5a6170963ac251a1f14b0c600e917aa26 10.0.0.30:6379@16379 master - 0 1681929288247 3 connected d9c17ded7fb139f9eb7fd41e476bb8e17c8b9bed 10.0.0.21:6379@16379 master - 0 1681929286000 0 connected 5461-10922 bdec3c92bcaa00c8ad1850cd477eb0a8f8a28aa2 10.0.0.22:6379@16379 master - 0 1681929289256 2 connected 10923-16383 [root@Redis-Ubuntu1804-node1:~]#
1.5、創建 Master-Slave 主從關系
使用命令 redis-cli -h <從節點IP> -p <從節點埠> -a <從節點密碼> --no-auth-warning cluster replicate <主節點node id>
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.30 cluster replicate 98bb2740b622645ff51fd07bc994bb22e94feaaa OK ##觀察主節點上的replication資訊 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 info Replication # Replication role:master connected_slaves:1 slave0:ip=10.0.0.30,port=6379,state=online,offset=350,lag=1 master_replid:6a823070fcbc13b4f4eb7b5d13bf1d2f4625a1a2 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:350 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:350 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 cluster nodes 98bb2740b622645ff51fd07bc994bb22e94feaaa 10.0.0.20:6379@16379 myself,master - 0 1681929989000 1 connected 0-5460 55fd3a5d21e026b6503a89f1d88996b2a67c314a 10.0.0.31:6379@16379 master - 0 1681929992372 4 connected 9fe09d53d884c6c99c926a0e7213cf5386987ebf 10.0.0.32:6379@16379 master - 0 1681929991000 5 connected 6717c6b5a6170963ac251a1f14b0c600e917aa26 10.0.0.30:6379@16379 slave 98bb2740b622645ff51fd07bc994bb22e94feaaa 0 1681929992000 3 connected d9c17ded7fb139f9eb7fd41e476bb8e17c8b9bed 10.0.0.21:6379@16379 master - 0 1681929989351 0 connected 5461-10922 bdec3c92bcaa00c8ad1850cd477eb0a8f8a28aa2 10.0.0.22:6379@16379 master - 0 1681929991364 2 connected 10923-16383 ##觀察從節點上的replication資訊 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.30 info Replication # Replication role:slave master_host:10.0.0.20 master_port:6379 master_link_status:up master_last_io_seconds_ago:0 master_sync_in_progress:0 slave_repl_offset:462 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:6a823070fcbc13b4f4eb7b5d13bf1d2f4625a1a2 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:462 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:462 [root@Redis-Ubuntu1804-node1:~]#
繼續完成主從組配置
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.31 cluster replicate d9c17ded7fb139f9eb7fd41e476bb8e17c8b9bed OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.32 cluster replicate bdec3c92bcaa00c8ad1850cd477eb0a8f8a28aa2 OK
##觀察集群節點狀態 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.30 cluster nodes 9fe09d53d884c6c99c926a0e7213cf5386987ebf 10.0.0.32:6379@16379 slave bdec3c92bcaa00c8ad1850cd477eb0a8f8a28aa2 0 1681930655876 5 connected 98bb2740b622645ff51fd07bc994bb22e94feaaa 10.0.0.20:6379@16379 master - 0 1681930654870 1 connected 0-5460 55fd3a5d21e026b6503a89f1d88996b2a67c314a 10.0.0.31:6379@16379 slave d9c17ded7fb139f9eb7fd41e476bb8e17c8b9bed 0 1681930656886 4 connected bdec3c92bcaa00c8ad1850cd477eb0a8f8a28aa2 10.0.0.22:6379@16379 master - 0 1681930655000 2 connected 10923-16383 6717c6b5a6170963ac251a1f14b0c600e917aa26 10.0.0.30:6379@16379 myself,slave 98bb2740b622645ff51fd07bc994bb22e94feaaa 0 1681930654000 3 connected d9c17ded7fb139f9eb7fd41e476bb8e17c8b9bed 10.0.0.21:6379@16379 master - 0 1681930653863 0 connected 5461-10922
1.6、驗證Cluster集群訪問
[root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.20 set node1 10.0.0.20 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.21 set node2 10.0.0.21 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.22 set node3 10.0.0.22 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.30 set node5 10.0.0.24 OK [root@Redis-Ubuntu1804-node1:~]# [root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.31 get node1 "10.0.0.20" [root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.31 get node2 "10.0.0.21" [root@Redis-Ubuntu1804-node1:~]# redis-cli -c -a redis -h 10.0.0.21 get node6 "10.0.0.31" [root@Redis-Ubuntu1804-node1:~]#
2、基于 Redis 5 Cluster集群部署
--cluster 常用命令
[root@Redis-Ubuntu1804-node3:~]# redis-cli -a redis --cluster help Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. Cluster Manager Commands: create host1:port1 ... hostN:portN ##創建集群 --cluster-replicas <arg> ##指定每個集群的slave節點數，通常為1 check host:port ##檢查集群資訊 --cluster-search-multiple-owners info host:port ##查看集群節點基本資訊 fix host:port ##修復集群 --cluster-search-multiple-owners reshard host:port ##在線熱機遷移指定的slots資料 --cluster-from <arg> --cluster-to <arg> --cluster-slots <arg> --cluster-yes --cluster-timeout <arg> --cluster-pipeline <arg> --cluster-replace rebalance host:port ##平衡集群在線節點的slots數量 --cluster-weight <node1=w1...nodeN=wN> --cluster-use-empty-masters --cluster-timeout <arg> --cluster-simulate --cluster-pipeline <arg> --cluster-threshold <arg> --cluster-replace add-node new_host:new_port existing_host:existing_port ##添加新的節點到集群中 --cluster-slave --cluster-master-id <arg> del-node host:port node_id ##洗掉集群中的節點 call host:port command arg arg .. arg ##在集群上的所有節點執行命令 set-timeout host:port milliseconds ##設定節點超時時間 import host:port ##匯入外部redis資料到當前集群 --cluster-from <arg> --cluster-copy --cluster-replace help For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
2.0、基本環境準備
[root@Redis-Ubuntu1804-node4:~]# redis-cli -a redis --version redis-cli 5.0.14 [root@Redis-Ubuntu1804-node4:~]# redis-cli -a redis --no-auth-warning cluster nodes e603091426e2ce29629ca51fb5e7dafb2f0c9524 :6379@16379 myself,master - 0 0 0 connected [root@Redis-Ubuntu1804-node4:~]#
2.1、創建Cluster集群
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --cluster create 10.0.0.20:6379 10.0.0.21:6379 10.0.0.22:6379 10.0.0.30:6379 10.0.0.31:6379 10.0.0.32:6379 --cluster-replicas 1 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. >>> Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 10.0.0.31:6379 to 10.0.0.20:6379 Adding replica 10.0.0.32:6379 to 10.0.0.21:6379 Adding replica 10.0.0.30:6379 to 10.0.0.22:6379 M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master M: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots:[10923-16383] (5461 slots) master S: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 replicates 07a36af61eb1f887419f2266b2360b4f795bb7a3 S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join . >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 S: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots: (0 slots) slave replicates 07a36af61eb1f887419f2266b2360b4f795bb7a3 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
2.2、查看集群狀態
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 master - 0 1681998239529 3 connected 10923-16383 a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1681998238520 2 connected 5461-10922 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1681998237000 6 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1681998238000 5 connected 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 slave 07a36af61eb1f887419f2266b2360b4f795bb7a3 0 1681998237510 4 connected 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 myself,master - 0 1681998236000 1 connected 0-5460 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis cluster info Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:290 cluster_stats_messages_pong_sent:282 cluster_stats_messages_sent:572 cluster_stats_messages_ping_received:277 cluster_stats_messages_pong_received:290 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:572 [root@Redis-Ubuntu1804-node1:~]# [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (07a36af6...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (07a36af6...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 S: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots: (0 slots) slave replicates 07a36af61eb1f887419f2266b2360b4f795bb7a3 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
2.3、驗證集群寫入
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning cluster nodes 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 master - 0 1681998730094 3 connected 10923-16383 a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1681998729087 2 connected 5461-10922 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1681998728000 6 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1681998728079 5 connected 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 slave 07a36af61eb1f887419f2266b2360b4f795bb7a3 0 1681998729000 4 connected 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 myself,master - 0 1681998726000 1 connected 0-5460 #寫入 key1：value1
###經過slot計算 key1 不屬于當前節點
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning set key1 value1 (error) MOVED 9189 10.0.0.21:6379
##計算 key 值所屬的slot號 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning cluster keyslot key1 (integer) 9189
##結合 nodes 資訊，切換到對應節點上進行寫入 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.21 --no-auth-warning set key1 value1 OK
##get命令也只可查看當前節點上的資料 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning get key1 (error) MOVED 9189 10.0.0.21:6379 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.21 --no-auth-warning get key1 "value1"
##可以用 -c 引數通過連接到集群進行資料操作，就無需反復切換節點（key1、key4 不在當前節點所屬的slot范圍，出現錯誤提示） [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning cluster keyslot key1 (integer) 9189 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning cluster keyslot key2 (integer) 4998 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning cluster keyslot key3 (integer) 935 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning cluster keyslot key4 (integer) 13120 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.21 --no-auth-warning set key1 value1 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning set key1 value1 (error) MOVED 9189 10.0.0.21:6379 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning set key2 value2 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning set key3 value3 OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --no-auth-warning set key4 value4 (error) MOVED 13120 10.0.0.22:6379 [root@Redis-Ubuntu1804-node1:~]#
##使用 -c 引數后執行成功
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning set key1 value1
OK
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning set key4 value4
OK
[root@Redis-Ubuntu1804-node1:~]#
2.4、模擬故障驗證集群故障切換
啟用腳本模擬客戶端資料讀寫
故障前集群資訊狀態
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning cluster nodes 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 master - 0 1681999895495 3 connected 10923-16383 a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1681999896502 2 connected 5461-10922 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1681999894488 6 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1681999895000 5 connected 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 slave 07a36af61eb1f887419f2266b2360b4f795bb7a3 0 1681999892000 4 connected 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 myself,master - 0 1681999893000 1 connected 0-5460 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 97 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (07a36af6...) -> 100 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 92 keys | 5462 slots | 1 slaves. [OK] 289 keys in 3 masters. 0.02 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 99 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (07a36af6...) -> 103 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 94 keys | 5462 slots | 1 slaves. [OK] 296 keys in 3 masters. 0.02 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 S: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots: (0 slots) slave replicates 07a36af61eb1f887419f2266b2360b4f795bb7a3 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
停用 10.0.0.22 節點
[root@Redis-Ubuntu1804-node3:~]# systemctl stop redis_6379.service
[root@Redis-Ubuntu1804-node3:~]#
觀察故障期間集群狀態及日志
##10.0.0.22節點標記 faild，并提升它的從節點 10.0.0.30 為master
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 Could not connect to Redis at 10.0.0.22:6379: Connection refused *** WARNING: 10.0.0.30:6379 claims to be slave of unknown node ID 07a36af61eb1f887419f2266b2360b4f795bb7a3. 10.0.0.20:6379 (6b142fe3...) -> 131 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 126 keys | 5462 slots | 1 slaves. [OK] 257 keys in 2 masters. 0.02 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 S: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots: (0 slots) slave replicates 07a36af61eb1f887419f2266b2360b4f795bb7a3 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [ERR] Not all 16384 slots are covered by nodes. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 Could not connect to Redis at 10.0.0.22:6379: Connection refused *** WARNING: 10.0.0.30:6379 claims to be slave of unknown node ID 07a36af61eb1f887419f2266b2360b4f795bb7a3. 10.0.0.20:6379 (6b142fe3...) -> 131 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 126 keys | 5462 slots | 1 slaves. [OK] 257 keys in 2 masters. 0.02 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning cluster nodes 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 master,fail - 1682000024732 1682000022415 3 disconnected a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1682000042000 2 connected 5461-10922 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1682000040567 6 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1682000042582 5 connected 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 master - 0 1682000041575 7 connected 10923-16383 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 myself,master - 0 1682000041000 1 connected 0-5460 [root@Redis-Ubuntu1804-node1:~]#
17519:M 20 Apr 2023 22:14:01.600 * Marking node 07a36af61eb1f887419f2266b2360b4f795bb7a3 as failing (quorum reached). 17519:M 20 Apr 2023 22:14:01.600 # Cluster state changed: fail 17519:M 20 Apr 2023 22:14:02.167 # Failover auth granted to 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 for epoch 7 17519:M 20 Apr 2023 22:14:02.177 # Cluster state changed: ok
客戶端日志出現了約4s 的服務不可用
恢復 10.0.0.22 節點
觀察集群節點資訊及日志
##10.0.0.22節點被添加為slave節點，并將master指向10.0.0.30
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 150 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 141 keys | 5462 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 154 keys | 5461 slots | 1 slaves. [OK] 445 keys in 3 masters. 0.03 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 150 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 141 keys | 5462 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 154 keys | 5461 slots | 1 slaves. [OK] 445 keys in 3 masters. 0.03 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
##日志記錄清除 10.0.0.22 節點上的faild 標記，并作為slave節點啟動 17519:M 20 Apr 2023 22:14:01.600 * Marking node 07a36af61eb1f887419f2266b2360b4f795bb7a3 as failing (quorum reached). 17519:M 20 Apr 2023 22:14:01.600 # Cluster state changed: fail 17519:M 20 Apr 2023 22:14:02.167 # Failover auth granted to 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 for epoch 7 17519:M 20 Apr 2023 22:14:02.177 # Cluster state changed: ok 17519:M 20 Apr 2023 22:17:23.792 * Clear FAIL state for node 07a36af61eb1f887419f2266b2360b4f795bb7a3: master without slots is reachable again.
3、基于Redis 3/4 Cluster集群部署
3.0、基本環境準備
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis cluster nodes 1c2fee65ee50cdf66c862d04710672286b305f55 :6379@16379 myself,master - 0 0 0 connected [root@Redis-Ubuntu1804-node1:~]# redis-cli --version redis-cli 4.0.9 [root@Redis-Ubuntu1804-node1:~]#
3.1、準備 redis-trib.rb 工具
查找 redis-trib.rb 工具路徑，將二進制檔案復制到 /usr/bin 目錄下
[root@Redis-Ubuntu1804-node1:~]# find / -name redis-trib.rb /usr/share/doc/redis-tools/examples/redis-trib.rb [root@Redis-Ubuntu1804-node1:~]# cp /usr/share/doc/redis-tools/examples/redis-trib.rb /usr/bin/
安裝ruby運行環境
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb /usr/bin/env: ‘ruby’: No such file or directory [root@Redis-Ubuntu1804-node1:~]# ruby Command 'ruby' not found, but can be installed with: snap install ruby # version 3.2.2, or apt install ruby See 'snap info ruby' for additional versions. [root@Redis-Ubuntu1804-node1:~]# apt-cache madison ruby ruby | 1:2.5.1 | http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 Packages [root@Redis-Ubuntu1804-node1:~]# apt install ruby -y Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: linux-headers-4.15.0-151 linux-headers-4.15.0-151-generic linux-image-4.15.0-151-generic linux-modules-4.15.0-151-generic linux-modules-extra-4.15.0-151-generic Use 'apt autoremove' to remove them. The following additional packages will be installed: fonts-lato javascript-common libjs-jquery libruby2.5 rake ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit ruby2.5 rubygems-integration unzip zip Suggested packages: apache2 | lighttpd | httpd ri ruby-dev bundler The following NEW packages will be installed: fonts-lato javascript-common libjs-jquery libruby2.5 rake ruby ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit ruby2.5 rubygems-integration unzip zip 0 upgraded, 15 newly installed, 0 to remove and 62 not upgraded. Need to get 6499 kB of archives. After this operation, 29.0 MB of additional disk space will be used. Get:1 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 fonts-lato all 2.0-2 [2698 kB] Get:2 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 javascript-common all 11 [6066 B] Get:3 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 libjs-jquery all 3.2.1-1 [152 kB] Get:4 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 rubygems-integration all 1.11 [4994 B] Get:5 http://cn.archive.ubuntu.com/ubuntu bionic-updates/main amd64 ruby2.5 amd64 2.5.1-1ubuntu1.13 [48.6 kB] Get:6 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 ruby amd64 1:2.5.1 [5712 B] Get:7 http://cn.archive.ubuntu.com/ubuntu bionic-updates/main amd64 rake all 12.3.1-1ubuntu0.1 [44.9 kB] Get:8 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 ruby-did-you-mean all 1.2.0-2 [9700 B] Get:9 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 ruby-minitest all 5.10.3-1 [38.6 kB] Get:10 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 ruby-net-telnet all 0.1.1-2 [12.6 kB] Get:11 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 ruby-power-assert all 0.3.0-1 [7952 B] Get:12 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 ruby-test-unit all 3.2.5-1 [61.1 kB] Get:13 http://cn.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libruby2.5 amd64 2.5.1-1ubuntu1.13 [3074 kB] Get:14 http://cn.archive.ubuntu.com/ubuntu bionic-updates/main amd64 unzip amd64 6.0-21ubuntu1.2 [168 kB] Get:15 http://cn.archive.ubuntu.com/ubuntu bionic/main amd64 zip amd64 3.0-11build1 [167 kB] Fetched 6499 kB in 9s (722 kB/s) Selecting previously unselected package fonts-lato. (Reading database ... 144903 files and directories currently installed.) Preparing to unpack .../00-fonts-lato_2.0-2_all.deb ... Unpacking fonts-lato (2.0-2) ... Selecting previously unselected package javascript-common. Preparing to unpack .../01-javascript-common_11_all.deb ... Unpacking javascript-common (11) ... Selecting previously unselected package libjs-jquery. Preparing to unpack .../02-libjs-jquery_3.2.1-1_all.deb ... Unpacking libjs-jquery (3.2.1-1) ... Selecting previously unselected package rubygems-integration. Preparing to unpack .../03-rubygems-integration_1.11_all.deb ... Unpacking rubygems-integration (1.11) ... Selecting previously unselected package ruby2.5. Preparing to unpack .../04-ruby2.5_2.5.1-1ubuntu1.13_amd64.deb ... Unpacking ruby2.5 (2.5.1-1ubuntu1.13) ... Selecting previously unselected package ruby. Preparing to unpack .../05-ruby_1%3a2.5.1_amd64.deb ... Unpacking ruby (1:2.5.1) ... Selecting previously unselected package rake. Preparing to unpack .../06-rake_12.3.1-1ubuntu0.1_all.deb ... Unpacking rake (12.3.1-1ubuntu0.1) ... Selecting previously unselected package ruby-did-you-mean. Preparing to unpack .../07-ruby-did-you-mean_1.2.0-2_all.deb ... Unpacking ruby-did-you-mean (1.2.0-2) ... Selecting previously unselected package ruby-minitest. Preparing to unpack .../08-ruby-minitest_5.10.3-1_all.deb ... Unpacking ruby-minitest (5.10.3-1) ... Selecting previously unselected package ruby-net-telnet. Preparing to unpack .../09-ruby-net-telnet_0.1.1-2_all.deb ... Unpacking ruby-net-telnet (0.1.1-2) ... Selecting previously unselected package ruby-power-assert. Preparing to unpack .../10-ruby-power-assert_0.3.0-1_all.deb ... Unpacking ruby-power-assert (0.3.0-1) ... Selecting previously unselected package ruby-test-unit. Preparing to unpack .../11-ruby-test-unit_3.2.5-1_all.deb ... Unpacking ruby-test-unit (3.2.5-1) ... Selecting previously unselected package libruby2.5:amd64. Preparing to unpack .../12-libruby2.5_2.5.1-1ubuntu1.13_amd64.deb ... Unpacking libruby2.5:amd64 (2.5.1-1ubuntu1.13) ... Selecting previously unselected package unzip. Preparing to unpack .../13-unzip_6.0-21ubuntu1.2_amd64.deb ... Unpacking unzip (6.0-21ubuntu1.2) ... Selecting previously unselected package zip. Preparing to unpack .../14-zip_3.0-11build1_amd64.deb ... Unpacking zip (3.0-11build1) ... Setting up libjs-jquery (3.2.1-1) ... Setting up unzip (6.0-21ubuntu1.2) ... Setting up zip (3.0-11build1) ... Setting up fonts-lato (2.0-2) ... Setting up ruby-did-you-mean (1.2.0-2) ... Setting up ruby-net-telnet (0.1.1-2) ... Setting up rubygems-integration (1.11) ... Setting up javascript-common (11) ... Setting up ruby-minitest (5.10.3-1) ... Setting up ruby-power-assert (0.3.0-1) ... Setting up ruby2.5 (2.5.1-1ubuntu1.13) ... Setting up ruby (1:2.5.1) ... Setting up ruby-test-unit (3.2.5-1) ... Setting up rake (12.3.1-1ubuntu0.1) ... Setting up libruby2.5:amd64 (2.5.1-1ubuntu1.13) ... Processing triggers for mime-support (3.60ubuntu1) ... Processing triggers for libc-bin (2.27-3ubuntu1.5) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
解決 redis-rtib.rb 執行報錯
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb Traceback (most recent call last): 2: from /usr/bin/redis-trib.rb:25:in `<main>' 1: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- redis (LoadError)
##使用 gem 工具安裝 ruby 的redis模塊 [root@Redis-Ubuntu1804-node1:~]# gem install redis Fetching: redis-5.0.6.gem (100%) Successfully installed redis-5.0.6 Parsing documentation for redis-5.0.6 Installing ri documentation for redis-5.0.6 Done installing documentation for redis after 0 seconds 1 gem installed
##離線模式安裝 gem
[root@Redis-Ubuntu1804-node1:~]# wget https://rubygems.org/downloads/redis-4.0.3.gem
[root@Redis-Ubuntu1804-node1:~]# gem install redis-4.0.3.gem
redis-trib.rb 常用命令
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>
create host1:port1 ... hostN:portN #創建集群
--replicas <arg> #指定每個master節點的副本數，即每組主從的slave節點數，通常為1
check host:port #檢查集群資訊
info host:port #查看集群主機資訊
fix host:port #修復集群
--timeout <arg>
reshard host:port #在線熱機遷移指定的slots資料
--from <arg>
--to <arg>
--slots <arg>
--yes
--timeout <arg>
--pipeline <arg>
rebalance host:port #平衡集群在線節點的slots數量
--weight <arg>
--auto-weights
--use-empty-masters
--timeout <arg>
--simulate
--pipeline <arg>
--threshold <arg>
add-node new_host:new_port existing_host:existing_port #添加新的節點到集群中
--slave
--master-id <arg>
del-node host:port node_id #洗掉集群中的指定節點
set-timeout host:port milliseconds #設定節點超時時間
call host:port command arg arg .. arg #在集群上所有節點上執行命令
import host:port #匯入外部redis服務器資料到當前集群
--from <arg>
--copy
--replace
help (show this help)
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
3.2、修改redis-trib.rb 添加連接redis密碼
[root@Redis-Ubuntu1804-node1:~]# sed -i 's/@r = Redis.new.*/@r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60, :password => "'redis'")/' /usr/bin/redis-trib.rb
[root@Redis-Ubuntu1804-node1:~]# vim /usr/bin/redis-trib.rb def connect(o={}) return if @r print "Connecting to node #{self}: " if $verbose STDOUT.flush begin @r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60, :password => 'redis') @r.ping rescue xputs "[ERR] Sorry, can't connect to node #{self}" exit 1 if o[:abort] @r = nil end xputs "OK" if $verbose end
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
3.3、創建Cluster集群
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb create --replicas 1 10.0.0.20:6379 10.0.0.21:6379 10.0.0.22:6379 10.0.0.30:6379 10.0.0.31:6379 10.0.0.32:6379 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 10.0.0.20:6379 10.0.0.21:6379 10.0.0.22:6379 Adding replica 10.0.0.31:6379 to 10.0.0.20:6379 Adding replica 10.0.0.32:6379 to 10.0.0.21:6379 Adding replica 10.0.0.30:6379 to 10.0.0.22:6379 M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 replicates 1c2fee65ee50cdf66c862d04710672286b305f55 S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join...... >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
集群創建失敗處理
#在所有節點上手動清空資料，并重置集群設定
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 --no-auth-warning flushall [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 --no-auth-warning cluster reset
3.4、查看集群狀態
查看集群資訊自動生產的組態檔
[root@Redis-Ubuntu1804-node1:~]# cat /etc/redis/redis.conf | grep cluster-conf # cluster-config-file nodes-6379.conf cluster-config-file nodes-6379.conf [root@Redis-Ubuntu1804-node1:~]# find / -name nodes-6379.conf /var/lib/redis/nodes-6379.conf [root@Redis-Ubuntu1804-node1:~]# cat /var/lib/redis/nodes-6379.conf f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379@16379 slave 709b09b96f19209e7f0a1e3cfc5c035678491e4b 0 1681979886000 4 connected 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379@16379 myself,master - 0 1681979886000 1 connected 0-5460 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379@16379 master - 0 1681979888000 2 connected 5461-10922 c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379@16379 slave 1c2fee65ee50cdf66c862d04710672286b305f55 0 1681979888959 5 connected 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379@16379 master - 0 1681979887000 3 connected 10923-16383 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379@16379 slave 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 0 1681979888000 6 connected vars currentEpoch 6 lastVoteEpoch 0 [root@Redis-Ubuntu1804-node1:~]#
查看集群狀態資訊
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 5462 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 5461 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:1000 cluster_stats_messages_pong_sent:1033 cluster_stats_messages_sent:2033 cluster_stats_messages_ping_received:1028 cluster_stats_messages_pong_received:1000 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:2033 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 cluster nodes f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379@16379 slave 709b09b96f19209e7f0a1e3cfc5c035678491e4b 0 1681980883096 4 connected 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379@16379 myself,master - 0 1681980882000 1 connected 0-5460 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379@16379 master - 0 1681980885110 2 connected 5461-10922 c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379@16379 slave 1c2fee65ee50cdf66c862d04710672286b305f55 0 1681980883000 5 connected 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379@16379 master - 0 1681980884000 3 connected 10923-16383 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379@16379 slave 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 0 1681980884103 6 connected [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 info replication # Replication role:master connected_slaves:1 slave0:ip=10.0.0.31,port=6379,state=online,offset=1428,lag=1 master_replid:33b67c12391d0787c01305ee41ccba34db5c843b master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1428 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:1428 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.31 info replication # Replication role:slave master_host:10.0.0.20 master_port:6379 master_link_status:up master_last_io_seconds_ago:6 master_sync_in_progress:0 slave_repl_offset:1554 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:33b67c12391d0787c01305ee41ccba34db5c843b master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1554 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:1554
3.5、模擬故障驗證集群故障切換
使用腳本寫入資料，查看集群資訊
[root@Client-Ubuntu-1804-250:~/script/redis]# ./redis_cluster_newData.py 100 [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 30 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 31 keys | 5462 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 39 keys | 5461 slots | 1 slaves. [OK] 100 keys in 3 masters. 0.01 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 -c get keys11 "value11" [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 get keys11 "value11" [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.20 get keys12 (error) MOVED 14046 10.0.0.22:6379 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -p 6379 -h 10.0.0.22 get keys12 "value12"
關閉 10.0.0.22 redis服務，模擬故障
[root@Redis-Ubuntu1804-node3:~]# systemctl stop redis [root@Redis-Ubuntu1804-node3:~]# ##初始資訊 [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. ##故障發生第一時間 [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 [ERR] Sorry, can't connect to node 10.0.0.22:6379 *** WARNING: 10.0.0.30:6379 claims to be slave of unknown node ID 709b09b96f19209e7f0a1e3cfc5c035678491e4b. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [ERR] Not all 16384 slots are covered by nodes. ##自動提升Msater [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots:10923-16383 (5461 slots) master 0 additional replica(s) M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
1021:M 20 Apr 17:45:33.030 * Marking node 709b09b96f19209e7f0a1e3cfc5c035678491e4b as failing (quorum reached). 1021:M 20 Apr 17:45:33.030 # Cluster state changed: fail 1021:M 20 Apr 17:45:34.067 # Failover auth granted to f9e73307b3146b25b51cec9a861463a616ed322b for epoch 7 1021:M 20 Apr 17:45:34.070 # Cluster state changed: ok
重新啟動 10.0.0.22 redis服務，將被自動設定為slave節點
[root@Redis-Ubuntu1804-node3:~]# systemctl start redis [root@Redis-Ubuntu1804-node3:~]# [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) M: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 S: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots: (0 slots) slave replicates f9e73307b3146b25b51cec9a861463a616ed322b S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
1014:M 20 Apr 17:45:33.030 * Marking node 709b09b96f19209e7f0a1e3cfc5c035678491e4b as failing (quorum reached). 1014:M 20 Apr 17:45:33.030 # Cluster state changed: fail 1014:M 20 Apr 17:45:34.067 # Failover auth granted to f9e73307b3146b25b51cec9a861463a616ed322b for epoch 7 1014:M 20 Apr 17:45:34.070 # Cluster state changed: ok 1014:M 20 Apr 17:51:47.852 * Clear FAIL state for node 709b09b96f19209e7f0a1e3cfc5c035678491e4b: master without slots is reachable again.
4、Cluster集群節點維護
4.1、集群維護之動態擴容
當生產環境集群負載達到上限，需要擴充Master節點以環境集群性能瓶頸，將涉及到集群擴容操作
*通常生產環境下建議集群Master節點數應設定為奇數（3、5、7），以避免出現腦裂現象
使用集群管理工具的 add-node 命令
add-node new_host:new_port existing_host:existing_port #添加新的節點到集群中 --slave --master-id <arg>
4.1.1、Redis5 集群節點擴容
查看當前集群節點資訊
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 150 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 141 keys | 5462 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 154 keys | 5461 slots | 1 slaves. [OK] 445 keys in 3 masters. 0.03 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
將節點 10.0.0.23 添加至集群中
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster add-node 10.0.0.23:6379 10.0.0.20:6379 >>> Adding node 10.0.0.23:6379 to cluster 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 10.0.0.23:6379 to make it join the cluster. [OK] New node added correctly. [root@Redis-Ubuntu1804-node1:~]#
##10.0.0.23 節點已經加入到集群中，但是未分配任何槽位資料，也沒有從節點
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 150 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 141 keys | 5462 slots | 1 slaves. 10.0.0.23:6379 (ac947d4f...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.30:6379 (5c9789e1...) -> 154 keys | 5461 slots | 1 slaves. [OK] 445 keys in 4 masters. 0.03 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 150 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 141 keys | 5462 slots | 1 slaves. 10.0.0.23:6379 (ac947d4f...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.30:6379 (5c9789e1...) -> 154 keys | 5461 slots | 1 slaves. [OK] 445 keys in 4 masters. 0.03 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots: (0 slots) master S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
備份并清空集群所有Master節點上的資料，為新節點分配槽位
reshard host:port #在線熱機遷移指定的slots資料 --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg>
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.20 --no-auth-warning flushall OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.21 --no-auth-warning flushall OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.30 --no-auth-warning flushall OK
##為新節點分配槽位
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots: (0 slots) master S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 4096 ##需要進行重新分配的槽位數 What is the receiving node ID? ac947d4fc970b79c2364ce305441906b462f8b65 ##接收分配槽位的節點 node id Please enter all the source node IDs. ##提供重新分配槽位的節點node id Type 'all' to use all the nodes as source nodes for the hash slots. ##輸入all將從擁有槽位的所有節點進行自動分配 Type 'done' once you entered all the source nodes IDs. ##輸入提供槽位的節點node id，輸入done結束 Source node #1: all ... ... ... Moving slot 12280 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12281 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12282 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12283 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12284 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12285 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12286 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Moving slot 12287 from 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Do you want to proceed with the proposed reshard plan (yes/no)? yes ##確認槽位分配方案 ... ... ... Moving slot 12280 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12281 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12282 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12283 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12284 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12285 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12286 from 10.0.0.30:6379 to 10.0.0.23:6379: Moving slot 12287 from 10.0.0.30:6379 to 10.0.0.23:6379:
查看槽位分配最終結果
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.23:6379 (ac947d4f...) -> 0 keys | 4096 slots | 0 slaves. 10.0.0.30:6379 (5c9789e1...) -> 0 keys | 4096 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
將 10.0.0.33 添加至集群中，并設定為 10.0.0.23 的從節點
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster add-node 10.0.0.33:6379 10.0.0.20:6379 --cluster-slave --cluster-master-id ac947d4fc970b79c2364ce305441906b462f8b65 >>> Adding node 10.0.0.33:6379 to cluster 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 10.0.0.33:6379 to make it join the cluster. Waiting for the cluster to join >>> Configure node as replica of 10.0.0.23:6379. [OK] New node added correctly. [root@Redis-Ubuntu1804-node1:~]#
查看集群節點資訊
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.23:6379 (ac947d4f...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 0 keys | 4096 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) S: 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379 slots: (0 slots) slave replicates ac947d4fc970b79c2364ce305441906b462f8b65 S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
4.1.2、Redis 3/4 集群節點擴容
當前集群節點資訊
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
將節點 10.0.0.23 添加至集群中
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb add-node 10.0.0.23:6379 10.0.0.20:6379 >>> Adding node 10.0.0.23:6379 to cluster 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 10.0.0.23:6379 to make it join the cluster. [OK] New node added correctly. [root@Redis-Ubuntu1804-node1:~]#
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.23:6379 (0e504f59...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots: (0 slots) master 0 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
備份并清空集群所有Master節點上的資料，為新節點分配槽位
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb call 10.0.0.20:6379 flushall >>> Calling FLUSHALL 10.0.0.20:6379: OK 10.0.0.31:6379: READONLY You can't write against a read only slave. 10.0.0.22:6379: OK 10.0.0.32:6379: READONLY You can't write against a read only slave. 10.0.0.23:6379: OK 10.0.0.30:6379: READONLY You can't write against a read only slave. 10.0.0.21:6379: OK [root@Redis-Ubuntu1804-node1:~]#
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots: (0 slots) master 0 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 4096 What is the receiving node ID? 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:all Do you want to proceed with the proposed reshard plan (yes/no)? yes ... ... ... Moving slot 12281 from 10.0.0.22:6379 to 10.0.0.23:6379: Moving slot 12282 from 10.0.0.22:6379 to 10.0.0.23:6379: Moving slot 12283 from 10.0.0.22:6379 to 10.0.0.23:6379: Moving slot 12284 from 10.0.0.22:6379 to 10.0.0.23:6379: Moving slot 12285 from 10.0.0.22:6379 to 10.0.0.23:6379: Moving slot 12286 from 10.0.0.22:6379 to 10.0.0.23:6379: Moving slot 12287 from 10.0.0.22:6379 to 10.0.0.23:6379:
查看槽位分配最終結果
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.23:6379 (0e504f59...) -> 0 keys | 4096 slots | 0 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 4096 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:1365-5460 (4096 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:0-1364,5461-6826,10923-12287 (4096 slots) master 0 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
將 10.0.0.33 添加至集群中，并設定為 10.0.0.23 的從節點
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb add-node --slave --master-id 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.33:6379 10.0.0.20:6379 >>> Adding node 10.0.0.33:6379 to cluster 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:1365-5460 (4096 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:0-1364,5461-6826,10923-12287 (4096 slots) master 0 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 10.0.0.33:6379 to make it join the cluster. Waiting for the cluster to join. >>> Configure node as replica of 10.0.0.23:6379. [OK] New node added correctly. [root@Redis-Ubuntu1804-node1:~]#
查看集群節點資訊
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.23:6379 (0e504f59...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 4096 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:1365-5460 (4096 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: 6e761ee02c9c17838e4643b9030ddfe327f1c742 10.0.0.33:6379 slots: (0 slots) slave replicates 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:0-1364,5461-6826,10923-12287 (4096 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
4.2、集群維護之動態縮容
當生產環境集群中的某臺服務器存在故障風險 或 集群性能閑置需要對集群節點進行服務器替換或資源回收操作時，需要涉及到動態縮容操作
下線Master節點10.0.0.23 及對應的 Slave節點 10.0.0.33
del-node host:port node_id
4.2.1、Redis 5 集群節點縮容
查看當前集群狀態
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster check 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.23:6379 (ac947d4f...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 0 keys | 4096 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) S: 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379 slots: (0 slots) slave replicates ac947d4fc970b79c2364ce305441906b462f8b65 S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
將洗掉Master節點的槽位分配至其他可用Master節點
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) S: 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379 slots: (0 slots) slave replicates ac947d4fc970b79c2364ce305441906b462f8b65 S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1365 What is the receiving node ID? 6b142fe3438c3ea488c34366e5ea5a298f89f518 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: ac947d4fc970b79c2364ce305441906b462f8b65 Source node #2: done Do you want to proceed with the proposed reshard plan (yes/no)? yes [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379 slots: (0 slots) slave replicates ac947d4fc970b79c2364ce305441906b462f8b65 S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots:[5461-6826],[10923-12287] (2731 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1366 What is the receiving node ID? a4b387d82ac06e5cbf212e48c781e1c27b50320f Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: ac947d4fc970b79c2364ce305441906b462f8b65 Source node #2: done Do you want to proceed with the proposed reshard plan (yes/no)? yes [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379 slots: (0 slots) slave replicates ac947d4fc970b79c2364ce305441906b462f8b65 S: 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379 slots: (0 slots) slave replicates 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 M: a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379 slots:[10923-12287] (1365 slots) master 1 additional replica(s) S: 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379 slots: (0 slots) slave replicates a4b387d82ac06e5cbf212e48c781e1c27b50320f S: 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379 slots: (0 slots) slave replicates 6b142fe3438c3ea488c34366e5ea5a298f89f518 M: 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1365 What is the receiving node ID? 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: ac947d4fc970b79c2364ce305441906b462f8b65 Source node #2: done Do you want to proceed with the proposed reshard plan (yes/no)? yes
查看當前槽位分配結果
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 5462 slots | 1 slaves. 10.0.0.23:6379 (ac947d4f...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.30:6379 (5c9789e1...) -> 0 keys | 5461 slots | 2 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
洗掉下線節點
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning cluster nodes 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379@16379 slave 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 0 1682007712000 15 connected 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 slave 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 0 1682007713000 15 connected a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1682007713079 14 connected 5461-10922 ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379@16379 master - 0 1682007710000 12 connected 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1682007711000 14 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1682007712000 13 connected 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 master - 0 1682007714086 15 connected 10923-16383 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 myself,master - 0 1682007709000 13 connected 0-5460 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster del-node 10.0.0.20:6379 ac947d4fc970b79c2364ce305441906b462f8b65 >>> Removing node ac947d4fc970b79c2364ce305441906b462f8b65 from cluster 10.0.0.20:6379 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster del-node 10.0.0.20:6379 829eef61e20def1a36e02a99f1650c9267be2108 >>> Removing node 829eef61e20def1a36e02a99f1650c9267be2108 from cluster 10.0.0.20:6379 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node.
查看縮容后的集群狀態
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning cluster nodes 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 slave 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 0 1682008285000 15 connected a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1682008285000 14 connected 5461-10922 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1682008286181 14 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1682008283157 13 connected 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 master - 0 1682008285173 15 connected 10923-16383 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 myself,master - 0 1682008283000 13 connected 0-5460 [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 0 keys | 5462 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 0 keys | 5461 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
清理下線節點node配置
被移除集群的redis節點行程將被自動關閉，需要先清理原node配置后，重新啟動服務，否則節點將誤以為自身仍在集群中
[root@Redis-Ubuntu1804-node4:~]# cat /app/redis/data/nodes-6379.conf 6b142fe3438c3ea488c34366e5ea5a298f89f518 10.0.0.20:6379@16379 master - 0 1682007475000 13 connected 0-5460 6a9dd64b6bf1a7eb336ec9d320d86ae7450d5e2b 10.0.0.32:6379@16379 slave a4b387d82ac06e5cbf212e48c781e1c27b50320f 0 1682007476000 14 connected 829eef61e20def1a36e02a99f1650c9267be2108 10.0.0.33:6379@16379 slave 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 0 1682007474000 15 connected ac947d4fc970b79c2364ce305441906b462f8b65 10.0.0.23:6379@16379 myself,master - 0 1682007470000 12 connected a4b387d82ac06e5cbf212e48c781e1c27b50320f 10.0.0.21:6379@16379 master - 0 1682007475000 14 connected 5461-10922 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 10.0.0.30:6379@16379 master - 0 1682007477017 15 connected 10923-16383 07a36af61eb1f887419f2266b2360b4f795bb7a3 10.0.0.22:6379@16379 slave 5c9789e105f8d8a0f1dc5ac2a6d51831482fbd34 0 1682007476000 15 connected 60aec73a6b0d3929cda3a284bb513898d2e73657 10.0.0.31:6379@16379 slave 6b142fe3438c3ea488c34366e5ea5a298f89f518 0 1682007476010 13 connected vars currentEpoch 15 lastVoteEpoch 0 [root@Redis-Ubuntu1804-node4:~]# rm -rf /app/redis/data/nodes-6379.conf
4.2.2、Redis 3/4 集群節點縮容
查看當前集群狀態
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.23:6379 (0e504f59...) -> 0 keys | 4096 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 4096 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:1365-5460 (4096 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: 6e761ee02c9c17838e4643b9030ddfe327f1c742 10.0.0.33:6379 slots: (0 slots) slave replicates 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:0-1364,5461-6826,10923-12287 (4096 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
將洗掉Master節點的槽位分配至其他可用Master節點
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:1365-5460 (4096 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: 6e761ee02c9c17838e4643b9030ddfe327f1c742 10.0.0.33:6379 slots: (0 slots) slave replicates 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:0-1364,5461-6826,10923-12287 (4096 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1365 What is the receiving node ID? 1c2fee65ee50cdf66c862d04710672286b305f55 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a Source node #2:done Do you want to proceed with the proposed reshard plan (yes/no)? yes [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: 6e761ee02c9c17838e4643b9030ddfe327f1c742 10.0.0.33:6379 slots: (0 slots) slave replicates 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:5461-6826,10923-12287 (2731 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:6827-10922 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1366 What is the receiving node ID? 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a Source node #2:done Do you want to proceed with the proposed reshard plan (yes/no)? yes [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb reshard 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: 6e761ee02c9c17838e4643b9030ddfe327f1c742 10.0.0.33:6379 slots: (0 slots) slave replicates 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots:10923-12287 (1365 slots) master 1 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1365 What is the receiving node ID? 709b09b96f19209e7f0a1e3cfc5c035678491e4b Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a Source node #2:done Do you want to proceed with the proposed reshard plan (yes/no)? yes
查看當前槽位分配結果
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 5461 slots | 2 slaves. 10.0.0.23:6379 (0e504f59...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 4 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 2 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: 6e761ee02c9c17838e4643b9030ddfe327f1c742 10.0.0.33:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a 10.0.0.23:6379 slots: (0 slots) master 0 additional replica(s) S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@Redis-Ubuntu1804-node1:~]#
洗掉下線節點
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb del-node 10.0.0.20:6379 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a >>> Removing node 0e504f596a5e8b55d0a38f5eacd3bd4b5c8f462a from cluster 10.0.0.20:6379 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb del-node 10.0.0.20:6379 6e761ee02c9c17838e4643b9030ddfe327f1c742 >>> Removing node 6e761ee02c9c17838e4643b9030ddfe327f1c742 from cluster 10.0.0.20:6379 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. [root@Redis-Ubuntu1804-node1:~]#
查看縮容后的集群狀態
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb check 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
4.3、集群維護之資料匯入
當企業Redis服務由 原先的單機運行或主從架構 升級為Cluster模式后，需要將原資料遷移至當前Cluster集群中，需要使用集群資料匯入
從歷史Redis服務器node8 10.0.0.33 上將資料匯入至當前集群中
[root@Redis-Ubuntu1804-node8:~]# redis-cli -a redis --no-auth-warning info keyspace # Keyspace db0:keys=20000,expires=0,avg_ttl=0[root@Redis-Ubuntu1804-node8:~]# redis-cli -a redis --no-auth-warning get key_2931 "value-2931" [root@Redis-Ubuntu1804-node8:~]#
*集群匯入資料需要確保新匯入資料不能與Redis Cluster已有資料出現重復的key名稱，否則將導致匯入失敗或中斷（建議在匯入資料前清空當前集群資料）
import host:port
--cluster-from <arg> ##外部redis的連接地址<ip:port>
--cluster-copy ##單獨使用時，匯入資料與當前集群中key值不可沖突
--cluster-replace ##強制替換key值
4.3.1、在 Redis 5 上執行資料匯入
清理集群和外部redis所有節點密碼
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning --cluster call 10.0.0.20:6379 config set requirepass '' >>> Calling config set requirepass 10.0.0.20:6379: OK 10.0.0.22:6379: OK 10.0.0.21:6379: OK 10.0.0.32:6379: OK 10.0.0.31:6379: OK 10.0.0.30:6379: OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.33 --no-auth-warning config set requirepass '' OK
執行資料匯入
[root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis --cluster info 10.0.0.20:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.20:6379 (6b142fe3...) -> 1 keys | 5461 slots | 1 slaves.
10.0.0.21:6379 (a4b387d8...) -> 1 keys | 5462 slots | 1 slaves.
10.0.0.30:6379 (5c9789e1...) -> 2 keys | 5461 slots | 1 slaves.
[OK] 4 keys in 3 masters.
0.00 keys per slot on average.
[root@Redis-Ubuntu1804-node1:~]# redis-cli --cluster import 10.0.0.20:6379 --cluster-from 10.0.0.33:6379 --cluster-copy --cluster-replace ... ... ... Migrating key_7867 to 10.0.0.20:6379: OK Migrating key_6810 to 10.0.0.20:6379: OK Migrating key_1232 to 10.0.0.21:6379: OK Migrating key_10171 to 10.0.0.21:6379: OK Migrating key_15561 to 10.0.0.30:6379: OK Migrating key_13723 to 10.0.0.21:6379: OK Migrating key_18761 to 10.0.0.21:6379: OK Migrating key_14471 to 10.0.0.20:6379: OK Migrating key_12496 to 10.0.0.30:6379: OK Migrating key_10559 to 10.0.0.21:6379: OK Migrating key_15282 to 10.0.0.21:6379: OK Migrating key_19021 to 10.0.0.21:6379: OK Migrating key_12831 to 10.0.0.21:6379: OK Migrating key_13260 to 10.0.0.20:6379: OK Migrating key_15767 to 10.0.0.30:6379: OK
查看資料
[root@Redis-Ubuntu1804-node1:~]# redis-cli --cluster info 10.0.0.20:6379 10.0.0.20:6379 (6b142fe3...) -> 6662 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (a4b387d8...) -> 6666 keys | 5462 slots | 1 slaves. 10.0.0.30:6379 (5c9789e1...) -> 6676 keys | 5461 slots | 1 slaves. [OK] 20004 keys in 3 masters. 1.22 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -c --no-auth-warning get key_2981 "value-2981" [root@Redis-Ubuntu1804-node1:~]#
4.3.2、在 Redis 3/4 上執行資料匯入
清理集群和外部redis所有節點密碼
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb call 10.0.0.20:6379 config set requirepass '' >>> Calling CONFIG set requirepass 10.0.0.20:6379: OK 10.0.0.31:6379: OK 10.0.0.22:6379: OK 10.0.0.32:6379: OK 10.0.0.30:6379: OK 10.0.0.21:6379: OK [root@Redis-Ubuntu1804-node1:~]# redis-cli -a redis -h 10.0.0.33 config set requirepass '' OK [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 [ERR] Sorry, can't connect to node 10.0.0.20:6379
##清理 redis-trib.rb 工具配置的密碼
[root@Redis-Ubuntu1804-node1:~]# sed -i 's/@r = Redis.new.*/@r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60)/' /usr/bin/redis-trib.rb [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]#
執行資料匯入
由于gem redis 版本與客戶端版本不一致出現資料匯入錯誤 Unsupported command argument type: NilClass (TypeError)
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb import --from 10.0.0.33:6379 --copy --replace 10.0.0.20:6379 >>> Importing data from 10.0.0.33:6379 to cluster >>> Performing Cluster Check (using node 10.0.0.20:6379) M: 1c2fee65ee50cdf66c862d04710672286b305f55 10.0.0.20:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: c567eea1d3fda7b37c42b741bc32d14385bbe872 10.0.0.31:6379 slots: (0 slots) slave replicates 1c2fee65ee50cdf66c862d04710672286b305f55 M: 709b09b96f19209e7f0a1e3cfc5c035678491e4b 10.0.0.22:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 0428f9a71c369004c0ca01c0a403f9c2ee5fa41c 10.0.0.32:6379 slots: (0 slots) slave replicates 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e S: f9e73307b3146b25b51cec9a861463a616ed322b 10.0.0.30:6379 slots: (0 slots) slave replicates 709b09b96f19209e7f0a1e3cfc5c035678491e4b M: 337bf2a15dd9b0ae8caaee08c5ec41beb754b78e 10.0.0.21:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Connecting to the source Redis instance *** Importing 20000 keys from DB 0 Traceback (most recent call last): 11: from /usr/bin/redis-trib.rb:1830:in `<main>' 10: from /usr/bin/redis-trib.rb:1604:in `import_cluster_cmd' 9: from /var/lib/gems/2.5.0/gems/redis-5.0.6/lib/redis/commands/keys.rb:26:in `scan' 8: from /var/lib/gems/2.5.0/gems/redis-5.0.6/lib/redis/commands/keys.rb:433:in `_scan' 7: from /var/lib/gems/2.5.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command' 6: from /usr/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize' 5: from /var/lib/gems/2.5.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command' 4: from /var/lib/gems/2.5.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v' 3: from /var/lib/gems/2.5.0/gems/redis-client-0.14.1/lib/redis_client.rb:223:in `call_v' 2: from /var/lib/gems/2.5.0/gems/redis-client-0.14.1/lib/redis_client/command_builder.rb:68:in `generate' 1: from /var/lib/gems/2.5.0/gems/redis-client-0.14.1/lib/redis_client/command_builder.rb:68:in `map!' /var/lib/gems/2.5.0/gems/redis-client-0.14.1/lib/redis_client/command_builder.rb:75:in `block in generate': Unsupported command argument type: NilClass (TypeError)
gem卸載當前版本 redis工具，重新安裝4.0版本gem
[root@Redis-Ubuntu1804-node1:~]# gem uninstall redis Successfully uninstalled redis-5.0.6 [root@Redis-Ubuntu1804-node1:~]# wget https://rubygems.org/downloads/redis-4.0.3.gem --2023-04-21 03:36:14-- https://rubygems.org/downloads/redis-4.0.3.gem Resolving rubygems.org (rubygems.org)... 151.101.1.227, 151.101.65.227, 151.101.129.227, ... Connecting to rubygems.org (rubygems.org)|151.101.1.227|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 116224 (114K) [application/octet-stream] Saving to: ‘redis-4.0.3.gem’ redis-4.0.3.gem 100%[===================================================================>] 113.50K --.-KB/s in 0.1s 2023-04-21 03:36:14 (1.11 MB/s) - ‘redis-4.0.3.gem’ saved [116224/116224] [root@Redis-Ubuntu1804-node1:~]# gem install redis-4.0.3.gem Successfully installed redis-4.0.3 Parsing documentation for redis-4.0.3 Installing ri documentation for redis-4.0.3 Done installing documentation for redis after 0 seconds 1 gem installed [root@Redis-Ubuntu1804-node1:~]#
由于gem版本過高導致出現 ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
Migrating key_2307 to 10.0.0.22:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_5080 to 10.0.0.20:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_15899 to 10.0.0.21:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_6538 to 10.0.0.22:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_1665 to 10.0.0.20:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_14837 to 10.0.0.21:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_1757 to 10.0.0.20:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY) Migrating key_13190 to 10.0.0.20:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
卸載當前版本，安裝gem-redis 3.5 版本
[root@Redis-Ubuntu1804-node1:~]# gem uninstall redis Successfully uninstalled redis-4.0.3 [root@Redis-Ubuntu1804-node1:~]# wget https://rubygems.org/downloads/redis-3.3.5.gem --2023-04-21 03:46:33-- https://rubygems.org/downloads/redis-3.3.5.gem Resolving rubygems.org (rubygems.org)... 151.101.193.227, 151.101.129.227, 151.101.65.227, ... Connecting to rubygems.org (rubygems.org)|151.101.193.227|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 92672 (90K) [application/octet-stream] Saving to: ‘redis-3.3.5.gem’ redis-3.3.5.gem 100%[===================================================================>] 90.50K 497KB/s in 0.2s 2023-04-21 03:46:34 (497 KB/s) - ‘redis-3.3.5.gem’ saved [92672/92672] [root@Redis-Ubuntu1804-node1:~]# gem install redis-3.3.5.gem Successfully installed redis-3.3.5 Parsing documentation for redis-3.3.5 Installing ri documentation for redis-3.3.5 Done installing documentation for redis after 0 seconds 1 gem installed
執行資料匯入
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb import --from 10.0.0.33:6379 --copy --replace 10.0.0.20:6379 ... ... ... Migrating key_19321 to 10.0.0.20:6379: OK Migrating key_3394 to 10.0.0.20:6379: OK Migrating key_11273 to 10.0.0.22:6379: OK Migrating key_15899 to 10.0.0.21:6379: OK Migrating key_14837 to 10.0.0.21:6379: OK Migrating key_1665 to 10.0.0.20:6379: OK Migrating key_5080 to 10.0.0.20:6379: OK Migrating key_13190 to 10.0.0.20:6379: OK Migrating key_1757 to 10.0.0.20:6379: OK Migrating key_6538 to 10.0.0.22:6379: OK Migrating key_2307 to 10.0.0.22:6379: OK [root@Redis-Ubuntu1804-node1:~]#
查看資料
[root@Redis-Ubuntu1804-node1:~]# redis-trib.rb info 10.0.0.20:6379 10.0.0.20:6379 (1c2fee65...) -> 6661 keys | 5461 slots | 1 slaves. 10.0.0.22:6379 (709b09b9...) -> 6674 keys | 5461 slots | 1 slaves. 10.0.0.21:6379 (337bf2a1...) -> 6665 keys | 5462 slots | 1 slaves. [OK] 20000 keys in 3 masters. 1.22 keys per slot on average. [root@Redis-Ubuntu1804-node1:~]# redis-cli -c get key_12332 "value-12332" [root@Redis-Ubuntu1804-node1:~]#
4.4、集群偏斜
Redis Cluser 多節點在運行一段時間以后，會出現傾斜現象，某個節點資料偏多，記憶體消耗更大，或者接收到的用戶請求更多，發生偏移現象的可能性，如下：
- 節點和槽位分配不均
- 不同槽位對應的鍵值數量差異較大
- 包含bigkey（建議少用）
- 記憶體相關配置不一致
- 熱點資料不均衡：一致性不高時，可使用本地快取或者MQ
#查看指定槽位中的鍵值數 [root@Redis-Ubuntu1804-node1:~]# redis-cli cluster countkeysinslot 0 (integer) 3 [root@Redis-Ubuntu1804-node1:~]# redis-cli cluster countkeysinslot 16383 (integer) 0 [root@Redis-Ubuntu1804-node1:~]# redis-cli cluster countkeysinslot 1 (integer) 2 [root@Redis-Ubuntu1804-node1:~]#
#槽位自動平衡分布 [root@Redis-Ubuntu1804-node1:~]# redis-trib.rb rebalance 10.0.0.20:6379 >>> Performing Cluster Check (using node 10.0.0.20:6379) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. *** No rebalancing needed! All nodes are within the 2.0% threshold. [root@Redis-Ubuntu1804-node1:~]#
#查找bigkeys,建議在slave上執行 [root@Redis-Ubuntu1804-node1:~]# redis-cli --bigkeys # Scanning the entire keyspace to find biggest keys as well as # average sizes per key type. You can use -i 0.1 to sleep 0.1 sec # per 100 SCAN commands (not usually needed). [00.00%] Biggest string found so far 'key_13312' with 11 bytes -------- summary ------- Sampled 6661 keys in the keyspace! Total key length in bytes is 56245 (avg len 8.44) Biggest string found 'key_13312' has 11 bytes 6661 strings with 69567 bytes (100.00% of keys, avg size 10.44) 0 lists with 0 items (00.00% of keys, avg size 0.00) 0 sets with 0 members (00.00% of keys, avg size 0.00) 0 hashs with 0 fields (00.00% of keys, avg size 0.00) 0 zsets with 0 members (00.00% of keys, avg size 0.00) 0 streams with 0 entries (00.00% of keys, avg size 0.00) [root@Redis-Ubuntu1804-node1:~]#
四、集群操作自動化
1、初始環境自動化配置腳本
-shell Ubuntu編譯安裝
[root@Client-Ubuntu-1804-250:~/script/redis]# cat redis_install.sh #!/bin/bash # #******************************************************************** #Author: janzen #Date: 2023-04-13 #FileName： redis_install.sh #Description： Install Redis #Copyright (C): 2023 All rights reserved #******************************************************************** DIR='/app/redis' DIR_S='\/app\/redis' PASSWD='redis' PORT=() echo -e "啟用架構：\n 1、single node \n 2、Replicae_node \n 3、Cluster_node\n" read -p "選擇節點型別：" node_type read -p "創建的副本數：" num case $num in 0) echo 不可創建0副本 exit ;; [1-9]*) echo 創建$num副本 ;; *) echo $num 不是數字 exit ;; esac for i in $(seq $num); do rds_port=`echo $i-1+6379|bc` PORT+=($rds_port) done wget https://download.redis.io/releases/redis-5.0.14.tar.gz tar xf redis-5.0.14.tar.gz useradd redis -s /sbin/nologin mkdir $DIR/{etc,log,data,run} -p chown redis.redis $DIR/ -R apt-get install build-essential tree -y cd redis-5.0.14 make PREFIX=$DIR install ln -s $DIR/bin/* /usr/bin/ echo net.core.somaxconn=1024 >> /etc/sysctl.conf echo vm.overcommit_memory=1 >> /etc/sysctl.conf sysctl -p echo never > /sys/kernel/mm/transparent_hugepage/enabled for i in ${PORT[*]};do case $node_type in 1) sed -e 's/^bind 127.0.0.1/bind 0.0.0.0/' -e 's/^port 6379/port '"$i"'/' -e 's/^pidfile \/var\/run\/redis_6379.pid/pidfile '"$DIR_S"'\/run\/redis_'"$i"'.pid/' -e 's/^logfile ""/logfile "'"$DIR_S"'\/log\/redis_'"$i"'.log"/g' -e 's/^always-show-logo yes/always-show-logo no/g' -e 's/^appendonly no/appendonly yes/' -e 's/^appendfilename "appendonly.aof"/appendfilename "appendonly_'"$i"'.aof"/g' -e 's/^dbfilename dump.rdb/dbfilename dump_'"$i"'.rdb/g' -e 's/^dir \.\//dir '"$DIR_S"'\/data/g' -e 's/^# masterauth <master-password>$/masterauth '"$PASSWD"'/' -e 's/^# requirepass foobared/requirepass '"$PASSWD"'/' redis.conf > $DIR/etc/redis_$i.conf ;; 2) read -p "node $i Master：<IP> <port>: " Master sed -e 's/^bind 127.0.0.1/bind 0.0.0.0/' -e 's/^port 6379/port '"$i"'/' -e 's/^pidfile \/var\/run\/redis_6379.pid/pidfile '"$DIR_S"'\/run\/redis_'"$i"'.pid/' -e 's/^logfile ""/logfile "'"$DIR_S"'\/log\/redis_'"$i"'.log"/g' -e 's/^always-show-logo yes/always-show-logo no/g' -e 's/^appendonly no/appendonly yes/' -e 's/^appendfilename "appendonly.aof"/appendfilename "appendonly_'"$i"'.aof"/g' -e 's/^dbfilename dump.rdb/dbfilename dump_'"$i"'.rdb/g' -e 's/^dir \.\//dir '"$DIR_S"'\/data/g' -e 's/^# masterauth <master-password>$/masterauth '"$PASSWD"'/' -e 's/^# requirepass foobared/requirepass '"$PASSWD"'/' -e '/# replicaof/a replicaof '"$Master"'' redis.conf > $DIR/etc/redis_$i.conf ;; 3) sed -e 's/^bind 127.0.0.1/bind 0.0.0.0/' -e 's/^port 6379/port '"$i"'/' -e 's/^pidfile \/var\/run\/redis_6379.pid/pidfile '"$DIR_S"'\/run\/redis_'"$i"'.pid/' -e 's/^logfile ""/logfile "'"$DIR_S"'\/log\/redis_'"$i"'.log"/g' -e 's/^always-show-logo yes/always-show-logo no/g' -e 's/^appendonly no/appendonly yes/' -e 's/^appendfilename "appendonly.aof"/appendfilename "appendonly_'"$i"'.aof"/g' -e 's/^dbfilename dump.rdb/dbfilename dump_'"$i"'.rdb/g' -e 's/^dir \.\//dir '"$DIR_S"'\/data/g' -e 's/^# masterauth <master-password>$/masterauth '"$PASSWD"'/' -e 's/^# requirepass foobared/requirepass '"$PASSWD"'/' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file/a cluster-config-file nodes-'"$i"'.conf' -e '/# cluster-require-full-coverage yes/a cluster-require-full-coverage yes' redis.conf > $DIR/etc/redis_$i.conf ;; *) exit ;; esac cat > /etc/systemd/system/redis_$i.service << EOF [Unit] Description=Redis persistent key-value database After=network.target [Service] ExecStart=$DIR/bin/redis-server $DIR/etc/redis_$i.conf --supervised systemd ExecStop=/bin/kill -s QUIT $MAINPID Type=notify User=redis Group=redis RuntimeDirectory=redis RuntimeDirectoryMode=0755 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF chown redis.redis $DIR/ -R systemctl daemon-reload systemctl enable --now redis_$i.service redis-cli -a $PASSWD -P $I --no-auth-warning info keyspace done cat > /etc/init.d/disable_transparent_hugepage << EOF #!/bin/bash echo never > /sys/kernel/mm/transparent_hugepage/enabled EOF chmod 755 /etc/init.d/disable_transparent_hugepage cd /etc/rcS.d/ ln -s ../init.d/disable_transparent_hugepage disable_transparent_hugepage ss -ntl ps -ef | grep redis-server # reboot
-shell Ubuntu 二進制包安裝
#!/bin/bash # #******************************************************************** #Author: janzen #Date: 2023-04-20 #FileName： redis_install_apt.sh #Description： The test script #Copyright (C): 2023 All rights reserved #******************************************************************** apt install redis -y sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth redis' -e '/# requirepass/a requirepass redis' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file/a cluster-config-file nodes-6379.conf' -e '/# cluster-require-full-coverage yes/a cluster-require-full-coverage yes' /etc/redis/redis.conf systemctl restart redis ss -ntl
2、批量分配slot槽位
[root@Client-Ubuntu-1804-250:~/script/redis]# cat add_slots.sh #!/bin/bash # #******************************************************************** #Author: janzen #Date: 2023-04-20 #FileName： add_slots.sh #Description： The test script #Copyright (C): 2023 All rights reserved #******************************************************************** host=$1 port=$2 start=$3 end=$4 passwd=redis for slot in `seq $start $end`; do redis-cli -h $host -p $port -a $passwd --no-auth-warning cluster addslots $slot && echo $slot add to node $host:$port done
3、python腳本實作自動創建資料
pip 安裝 redis-py-cluster 模塊
[root@Client-Ubuntu-1804-250:~/script/redis]# pip3 install redis-py-cluster Collecting redis-py-cluster Downloading https://files.pythonhosted.org/packages/b2/96/153bbcf5dee29b52b2674e77a87ce864d381f72151737317529b7de4f337/redis_py_cluster-2.1.3-py2.py3-none-any.whl (42kB) 100% |████████████████████████████████| 51kB 403kB/s Collecting redis<4.0.0,>=3.0.0 (from redis-py-cluster) Downloading https://files.pythonhosted.org/packages/a7/7c/24fb0511df653cf1a5d938d8f5d19802a88cef255706fdda242ff97e91b7/redis-3.5.3-py2.py3-none-any.whl (72kB) 100% |████████████████████████████████| 81kB 1.5MB/s Installing collected packages: redis, redis-py-cluster Found existing installation: redis 2.10.6 Not uninstalling redis at /usr/lib/python3/dist-packages, outside environment /usr Successfully installed redis-3.5.3 redis-py-cluster-2.1.3
python腳本連接集群創建資料
#!/usr/bin/python3 # -*- coding: UTF-8 -*- #******************************************************************** #Author: janzen #Date: 2023-04-20 #FileName： redis_cluster_newData.py #Description： The python3 script #Copyright (C): 2023 All rights reserved #******************************************************************** from rediscluster import RedisCluster import sys num=int(sys.argv[1]) passwd='redis' startup_nodes = [ {"host":"10.0.0.20","port":"6379"}, {"host":"10.0.0.21","port":"6379"}, {"host":"10.0.0.22","port":"6379"}, {"host":"10.0.0.30","port":"6379"}, {"host":"10.0.0.31","port":"6379"}, {"host":"10.0.0.32","port":"6379"} ] redis_conn = RedisCluster(startup_nodes = startup_nodes,password = passwd,decode_responses=True) for i in range(0,num): redis_conn.set("keys%s" % i,"value%s" % i) print("keys%s:" % i,redis_conn.get("keys%s" % i))
#!/usr/bin/python3 # -*- coding: UTF-8 -*- #******************************************************************** #Author: janzen #Date: 2023-04-20 #FileName： redis_cluster_newData.py #Description： The python3 script #Copyright (C): 2023 All rights reserved #******************************************************************** from rediscluster import RedisCluster import sys num=int(sys.argv[1]) passwd='redis' startup_nodes = [ {"host":"10.0.0.20","port":"6379"}, {"host":"10.0.0.21","port":"6379"}, {"host":"10.0.0.22","port":"6379"}, {"host":"10.0.0.30","port":"6379"}, {"host":"10.0.0.31","port":"6379"}, {"host":"10.0.0.32","port":"6379"} ] redis_conn = RedisCluster(startup_nodes = startup_nodes,password = passwd,decode_responses=True) while 1: time.sleep(1) try: redis_conn.set("keys%s" % num,"value%s" % num) print("set data %s:%s" % (num,redis_conn.get("keys%s" % num))) except Exception as err: print(err) continue num+=1
python腳本連接集群讀取資料
#!/usr/bin/python3 # -*- coding: UTF-8 -*- #******************************************************************** #Author: janzen #Date: 2023-04-20 #FileName： redis_cluster_newData.py #Description： The python3 script #Copyright (C): 2023 All rights reserved #******************************************************************** from rediscluster import RedisCluster import sys,time num=int(sys.argv[1]) passwd='redis' startup_nodes = [ {"host":"10.0.0.20","port":"6379"轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/550990.html
標籤：NoSQL
上一篇：讀《mysql是怎樣運行的》有感
下一篇：返回列表