主从数据全量迁移到分片集群测试

一、背景与目的

测试与验证HA模式数据全量迁移到分片集群，

工具： redisShake4.3.1

二、准备

1. 原集群

   主： 172.25.242.158
   从： 172.25.242.154

   免密
   SOX主机
   端口：6379
   redis-cli -h 172.25.242.158 -n 1 dbsize

2. 目的集群

# 
172.25.39.54:6379
172.25.39.51:6379
172.25.39.52:6379
172.25.39.48:6379
172.25.39.55:6379
172.25.39.49:6379
172.25.39.53:6379
172.25.39.50:6379



组名： redis-cluster-newlps-ksa
密码：  2ghlmvl_ST
端口：6379


# 
redis-cli -h 172.25.39.48 -a 2ghlmvl_MT cluster nodes


# 测试分片集群
172.25.190.13
172.25.189.209
172.25.189.211
172.25.190.12 
172.25.190.11 
172.25.189.212

密码
123456

redis-cli -h 172.25.190.13 -a 123456 dbsize 
# redis-cli -h 172.25.189.209 -a 123456 dbsize 
redis-cli -h 172.25.189.211 -a 123456 dbsize 
redis-cli -h 172.25.190.12 -a 123456 dbsize 
redis-cli -h 172.25.190.11 -a 123456 dbsize 
redis-cli -h 172.25.189.212 -a 123456 dbsize 

3. 工具

/root/redis/redisshake431

三、全量数据同步源

原集群数据量检查

[root@KSSYYVI01352 redisshake431]# redis-cli -h 172.25.242.158 -n 1 dbsize
(integer) 106992                                                                                                                                             

[root@KSSYYVI01352 redisshake431]# redis-cli -h 172.25.242.158 -n 10 dbsize
(integer) 2486   

2. redisShake配置文件调整shake.toml

[sync_reader] 
cluster = false 
address = "172.25.252.154:6379" # 源 Redis 地址 
password = ""          # 源 Redis 密码 
sync_rdb = false               # 启用 RDB 同步 
sync_aof = false               # 关闭 AOF 同步 
 
[redis_writer] 
cluster = true 
address = "172.25.190.13:6379" # 目标 Redis 地址 
password = "123456"          # 目标 Redis 密码 
skip_existing_keys = true  
 
[filter] 
allow_db = [01,10]                # 只同步 1和10 号库的数据 
#allow_keys = ["ycms:sm:draw:lottery:prize:current:count:2e2f3471006540194639:total"] 
 
[advanced] 
log_file = "shake.log"        # 日志文件路径 
log_level = "info"            # 日志级别：info 
pipeline_count_limit = 1024   # 默认管道大小 
 
#rdb_restore_command_behavior = "skip" # 遇到重复键时覆盖

3. 执行

# 
./redis-shake shake.toml


# 
tail -f shake.log

shake.log 中显示如下：

执行的输出：

[root@KSSYYVI01352 redisshake431]# ./redis-shake shake.toml
2025-04-17 19:52:07 INF load config from file: shake.toml
2025-04-17 19:52:07 INF log_level: [info], log_file: [/root/redis/redisshake431/data/shake.log]
2025-04-17 19:52:07 INF changed work dir. dir=[/root/redis/redisshake431/data]
2025-04-17 19:52:07 INF GOMAXPROCS defaults to the value of runtime.NumCPU [4]
2025-04-17 19:52:07 INF not set pprof port
2025-04-17 19:52:07 INF create SyncStandaloneReader
2025-04-17 19:52:07 INF * address: 172.25.242.158:6379
2025-04-17 19:52:07 INF * username: 
2025-04-17 19:52:07 INF * password: 
2025-04-17 19:52:07 INF * tls: false
2025-04-17 19:52:07 INF create RedisClusterWriter
2025-04-17 19:52:07 INF * address (should be the address of one node in the Redis cluster): 172.25.190.13:6379
2025-04-17 19:52:07 INF * username: 
2025-04-17 19:52:07 INF * password: ******
2025-04-17 19:52:07 INF * tls: false
2025-04-17 19:52:07 INF address=172.25.190.13:6379, reply=f99fe6a48492bd388474f33771a18b04fbf3c3d5 172.25.190.13:6379@16379 myself,master - 0 1744890726000 8 connected 5462-10922
c5815dabf9896b15ceda8dbd15742ab15d3e9fe6 172.25.190.11:6379@16379 slave 82a78b1cb3aaf11343704fb39f32e28f702fbfcc 0 1744890725039 6 connected
2150fe1f0122b2b03f2ab75ccd9516288800a2c8 172.25.189.211:6379@16379 master - 0 1744890727042 9 connected 10923-16383
82a78b1cb3aaf11343704fb39f32e28f702fbfcc 172.25.189.212:6379@16379 master - 0 1744890726039 6 connected 0-5461
5ec837e77676d921e135f7879c044786f7754b30 172.25.190.12:6379@16379 slave f99fe6a48492bd388474f33771a18b04fbf3c3d5 0 1744890725000 8 connected
e49b1fe2d24c3cd4452b32c9d78f99a2e9300c37 172.25.189.209:6379@16379 slave 2150fe1f0122b2b03f2ab75ccd9516288800a2c8 0 1744890725000 9 connected
2025-04-17 19:52:07 INF redisClusterWriter connected to redis cluster successful. addresses=[172.25.190.13:6379 172.25.189.211:6379 172.25.189.212:6379]
2025-04-17 19:52:07 INF start syncing...
2025-04-17 19:52:07 INF [reader_172.25.242.158_6379] source db is not doing bgsave! continue.


2025-04-17 19:52:11 ERR [writer_172.25.189.211_6379] receive reply failed. cmd=[select 1], error=[ERR SELECT is not allowed in cluster mode]
                        RedisShake/internal/writer/redis_standalone_writer.go:154 -> (*redisStandaloneWriter).processReply()
                        runtime/asm_amd64.s:1650 -> goexit()

说明:

主从模式向cluster迁移数据的时候，由于cluster模式不支持切换db，导致迁移失败。

cluster不支持select 选择db， 而源redis集群数据并不在db01中，所以失败。

建议：

如果源 Redis 使用了特定的数据库编号（如 1 或 10），而目标 Redis 集群不支持数据库编号，那么需要在源 Redis 中将数据迁移到默认数据库（通常是数据库 0）。

观察

# 
redis-cli -h 172.25.190.13 -a 123456 dbsize 
redis-cli -h 172.25.189.209 -a 123456 dbsize 
redis-cli -h 172.25.189.211 -a 123456 dbsize 
redis-cli -h 172.25.190.12 -a 123456 dbsize 
redis-cli -h 172.25.190.11 -a 123456 dbsize 
redis-cli -h 172.25.189.212 -a 123456 dbsize 

对比新旧集群的数据量

# 新集群
redis-cli -h 172.25.39.54 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.51 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.52 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.48 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.55 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.49 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.53 -p 6379 -a 2ghlmvl_MT dbsize
redis-cli -h 172.25.39.50 -p 6379 -a 2ghlmvl_MT dbsize

完成

四、补充信息

1. 文档与参考

tair-opensource.github.io/RedisShake/...

2. 相关方案 lue