一、基础信息介绍
0、架构介绍
0x1 部署结构
| 服务器 | 端口7001 | 端口7002 | 端口7003 |
|---|---|---|---|
| 192.168.1.83 | master1 | slave2 | slave3 |
| 192.168.1.84 | master2 | slave3 | slave1 |
| 192.168.1.85 | master3 | slave1 | slave2 |
主节点分布:
- master1:192.168.1.83:7001
- master2:192.168.1.84:7001
- master3:192.168.1.85:7001
从节点分布:
- slave1:192.168.1.84:7003, 192.168.1.85:7002
- slave2:192.168.1.83:7002, 192.168.1.85:7003
- slave3:192.168.1.83:7003, 192.168.1.84:7002
说明:
- 每个主节点的从节点都分布在不同服务器上,每台服务器的7001通常为主节点,负责部分slot;
- 7002、7003分别为其它主节点的从节点,实现主从分布在不同机器,提升高可用;
- 单台服务器宕机不会导致数据不可用,集群仍有主节点和从节点保证数据完整和服务可用;
0x2 端口及角色
7001端口
- 在本机(比如192.168.1.83)为主节点(master1)。
- 在其它机器(比如192.168.1.84/85)为从节点(slave1),同步master1的数据。
7002端口
- 在本机为从节点(slave2),同步其它机器上的master2。
- 在其它机器上可能是master2或slave2。
7003端口
- 在本机为从节点(slave3),同步其它机器上的master3。
- 在其它机器上可能是master3或slave3。
总结
- 3台服务器,每台3个端口 ,通常是3主3从结构。
- 每台服务器上的7001/7002/7003分别承担不同主/从角色,保证高可用。
- 集群自动分配slot,主节点负责分片存储,从节点负责同步和故障切换。
角色端口表
| 节点 | 端口 | 角色 | 主节点 | 备注 |
|---|---|---|---|---|
| 192.168.1.83 | 7001 | master | self | Master1 |
| 192.168.1.83 | 7002 | slave | 192.168.1.84:7001 | Master2的从 |
| 192.168.1.83 | 7003 | slave | 192.168.1.85:7001 | Master3的从 |
| 192.168.1.84 | 7001 | master | self | Master2 |
| 192.168.1.84 | 7002 | slave | 192.168.1.85:7001 | Master3的从 |
| 192.168.1.84 | 7003 | slave | 192.168.1.83:7001 | Master1的从 |
| 192.168.1.85 | 7001 | master | self | Master3 |
| 192.168.1.85 | 7002 | slave | 192.168.1.83:7001 | Master1的从 |
| 192.168.1.85 | 7003 | slave | 192.168.1.84:7001 | Master2的从 |
1、基础环境
OS 信息:
- OS: openEuler 22.03 (LTS-SP4)
- Kernel: Linux 5.10.0-216.0.0.115.oe2203sp4.x86_64
- Architecture: x86-64
- CPU/MEM: 12C64G
- Disk: 300G
Server信息:
- srv01:192.168.1.83
- srv02:192.168.1.84
- srv03:192.168.1.85
目录规划(所有节点一致):
- Redis源码目录:/opt/software/redis-7.4.5/
- Redis集群数据目录:/opt/redis-cluster/7001、7002、7003
- 配置文件目录:/etc/redis/
- 服务配置目录:/etc/systemd/system
- 日志目录:/var/log/redis/
- PID文件目录:/var/run/redis/
二、目录创建
所有节点都执行
bash
# 创建数据目录
mkdir -p /opt/redis-cluster/7001 /opt/redis-cluster/7002 /opt/redis-cluster/7003
# 创建日志目录
mkdir -p /var/log/redis
# 创建 PID 目录(两种路径都创建,兼容所有配置)
mkdir -p /var/run/redis
mkdir -p /run/redis
# 设置权限
chmod 755 /var/log/redis /var/run/redis /run/redis
chown root:root /var/log/redis /var/run/redis /run/redis
三、内核参数优化
所有节点都执行
bash
# 开启内存 overcommit
echo "vm.overcommit_memory=1" >> /etc/sysctl.conf
sysctl -p //刷新加载
四、配置文件清单
1、 节点1
vi /etc/redis/7001.conf //编辑配置文件,输入以下内容
shell
# Redis 7001 Configuration for 192.168.1.83
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7001.pid
logfile "/var/log/redis/7001.log"
dir /opt/redis-cluster/7001
bind 192.168.1.83
port 7001
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7001.aof"
dbfilename dump-7001.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
vi /etc/redis/7002.conf //编辑配置文件,输入以下内容
shell
# Redis 7002 Configuration for 192.168.1.83
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7002.pid
logfile "/var/log/redis/7002.log"
dir /opt/redis-cluster/7002
bind 192.168.1.83
port 7002
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7002.aof"
dbfilename dump-7002.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
vi /etc/redis/7003.conf //编辑配置文件,输入以下内容
shell
# Redis 7003 Configuration for 192.168.1.83
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7003.pid
logfile "/var/log/redis/7003.log"
dir /opt/redis-cluster/7003
bind 192.168.1.83
port 7003
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7003.aof"
dbfilename dump-7003.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
2、 节点2
vi /etc/redis/7001.conf //编辑配置文件,输入以下内容
shell
# Redis 7001 Configuration for 192.168.1.84
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7001.pid
logfile "/var/log/redis/7001.log"
dir /opt/redis-cluster/7001
bind 192.168.1.84
port 7001
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7001.aof"
dbfilename dump-7001.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
vi /etc/redis/7002.conf //编辑配置文件,输入以下内容
shell
# Redis 7002 Configuration for 192.168.1.84
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7002.pid
logfile "/var/log/redis/7002.log"
dir /opt/redis-cluster/7002
bind 192.168.1.84
port 7002
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7002.aof"
dbfilename dump-7002.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
vi /etc/redis/7003.conf //编辑配置文件,输入以下内容
shell
# Redis 7003 Configuration for 192.168.1.84
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7003.pid
logfile "/var/log/redis/7003.log"
dir /opt/redis-cluster/7003
bind 192.168.1.84
port 7003
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7003.aof"
dbfilename dump-7003.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
3、节点3
vi /etc/redis/7001.conf //编辑配置文件,输入以下内容
shell
# Redis 7001 Configuration for 192.168.1.85
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7001.pid
logfile "/var/log/redis/7001.log"
dir /opt/redis-cluster/7001
bind 192.168.1.85
port 7001
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7001.aof"
dbfilename dump-7001.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
vi /etc/redis/7002.conf //编辑配置文件,输入以下内容
shell
# Redis 7002 Configuration for 192.168.1.85
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7002.pid
logfile "/var/log/redis/7002.log"
dir /opt/redis-cluster/7002
bind 192.168.1.85
port 7002
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7002.aof"
dbfilename dump-7002.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
vi /etc/redis/7003.conf //编辑配置文件,输入以下内容
shell
# Redis 7003 Configuration for 192.168.1.85
# Basic Settings
daemonize yes
pidfile /var/run/redis/redis_7003.pid
logfile "/var/log/redis/7003.log"
dir /opt/redis-cluster/7003
bind 192.168.1.85
port 7003
# Security Settings
requirepass "RedisCluster@Secure2023!"
masterauth "RedisCluster@Secure2023!"
protected-mode no
# Cluster Configuration
# Persistence
appendonly yes
appendfilename "appendonly-7003.aof"
dbfilename dump-7003.rdb
save 900 1
save 300 10
save 60 10000
# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Performance Tuning
tcp-backlog 511
timeout 0
tcp-keepalive 300
三、systemd服务文件
每台服务器依次配置,可以在一台服务器上创建完毕后通过scp传输到其他服务器节点。
1、7001.service
vi /etc/systemd/system/redis-7001.service //编辑服务文件,输入以下内容
shell
[Unit]
Description=Redis Cluster 7001
After=network.target
[Service]
Type=forking
User=root
Group=root
PIDFile=/var/run/redis/redis_7001.pid
ExecStart=/opt/software/redis-7.4.5/src/redis-server /etc/redis/7001.conf
StandardOutput=journal+console
4StandardError=journal+console
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
2、7002.service
vi /etc/systemd/system/redis-7002.service //编辑服务文件,输入以下内容
shell
[Unit]
Description=Redis Cluster 7002
After=network.target
[Service]
Type=forking
User=root
Group=root
PIDFile=/var/run/redis/redis_7002.pid
ExecStart=/opt/software/redis-7.4.5/src/redis-server /etc/redis/7002.conf
StandardOutput=journal+console
4StandardError=journal+console
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
3、7003.service
vi /etc/systemd/system/redis-7003.service //编辑服务文件,输入以下内容
shell
[Unit]
Description=Redis Cluster 7003
After=network.target
[Service]
Type=forking
User=root
Group=root
PIDFile=/var/run/redis/redis_7003.pid
ExecStart=/opt/software/redis-7.4.5/src/redis-server /etc/redis/7003.conf
StandardOutput=journal+console
4StandardError=journal+console
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
四、启动服务命令
每台节点服务器依次分别执行
shell
systemctl daemon-reload
systemctl enable redis-7001 redis-7002 redis-7003
systemctl start redis-7001.service redis-7002.service redis-7003
systemctl status redis-7001 redis-7003 redis-7002
五、集群创建
1、集群初始化
任选一个节点服务器执行即可
shell
/opt/software/redis-7.4.5/src/redis-cli --cluster create \
192.168.1.83:7001 192.168.1.83:7002 192.168.1.83:7003 \
192.168.1.84:7001 192.168.1.84:7002 192.168.1.84:7003 \
192.168.1.85:7001 192.168.1.85:7002 192.168.1.85:7003 \
--cluster-replicas 1 -a "RedisCluster@Secure2023!"
可能会出错:
ERR\] Not all 16384 slots are covered by nodes.
说明:根据提示,哪些主节点没有分配到对应数量的 slots,到指定节点上执行修复
2、原因分析
通常是因为节点顺序或 redis-cli 算法分配不均导致。初始化时指定了 9 个节点,其中 3 个是 master(每台机一个 master),但实际 slot 分配不均,有的 master 没有分配到 slots。
3、修复 slots 覆盖
你只需要在任意一个节点执行如下命令,Redis 会自动补齐 slot 分配:
shell
/opt/software/redis-7.4.5/src/redis-cli --cluster fix 192.168.1.83:7001 -a "RedisCluster@Secure2023!"
这个命令会自动把未分配的 slots 补给合适的 master 节点。
六、集群状态确认
1、集群检查
用
redis-cli --cluster check检查
shell
/opt/software/redis-7.4.5/src/redis-cli --cluster check 192.168.1.83:7001 -a "RedisCluster@Secure2023!"
输出中应有:
shell
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
如果看到这些,说明集群分片和主从都正常。
2、 连接测试 & 数据读写
2.1 写入测试
shell
/opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" set testkey "hello cluster"
返回:
OK
2.2 读取测试
shell
/opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.84 -p 7002 -a "RedisCluster@Secure2023!" get testkey
返回:
"hello cluster"
2.3 跨节点测试
带 -c 参数,redis-cli 会自动跟随集群重定向:
shell
/opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.85 -p 7003 -a "RedisCluster@Secure2023!" get testkey
返回同样是:
shell
"hello cluster"
3、监控主从状态
可以在任意节点执行:
shell
/opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" info replication
查看主从角色、同步状态。
4、节点角色分布
shell
/opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" cluster nodes
5、查看节点运行状态
shell
/opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" info
七、日常运维
1、服务启停
shell
#生产环境避免数据丢失,建议停止用shutdown
systemctl start redis-7001 redis-7002 redis-7003
/opt/software/redis-7.4.5/src/redis-cli -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" shutdown
2、进程端口检查
bash
ps -ef | grep redis-server
bash
netstat -ntlp | grep 7001
netstat -ntlp | grep 7002
netstat -ntlp | grep 7003
3、日志排查
3.1 服务日志文件路径
shell
/var/logs/redis/redis-7001.log
/var/logs/redis/redis-7002.log
/var/logs/redis/redis-7003.log
3.2 查看服务日志内容
bash
tail -n 100 /var/logs/redis/redis-7001.log
tail -n 100 /var/logs/redis/redis-7002.log
tail -n 100 /var/logs/redis/redis-7003.log
3.3 服务报错日志
shell
journalctl -u redis7001 -n 100 -xe
journalctl -u redis7002 -n 100 -xe
journalctl -u redis7003 -n 100 -xe
-n 100:显示最近100行日志(可调整行数)。-xe:显示详细(扩展)和报错信息。
3.4 实时服务日志
(启动过程中看)
shell
journalctl -u redis7001 -f
4、常见报错及排查
| 报错内容 | 处理建议 |
|---|---|
MISCONF Redis is configured to save RDB snapshots |
检查磁盘空间,清理后重启 |
CLUSTERDOWN The cluster is down |
检查所有 master 是否在线,slot 是否全覆盖,必要时用 --cluster fix 修复 |
NOAUTH Authentication required |
命令需加 -a "RedisCluster@Secure2023!" |
ERR Slot not covered |
用 --cluster fix 192.168.1.83:7001 -a "RedisCluster@Secure2023!" 修复 |
Unable to connect to Redis |
检查网络、防火墙、配置文件监听地址 |
Out Of Memory |
增加内存或优化配置(如 maxmemory),查看 info memory |
八、运维命令速查表
| 操作 | 命令 |
|---|---|
| 启动节点 | systemctl start redis-7001 redis-7002 redis-7003 |
| 停止节点 | /opt/software/redis-7.4.5/src/redis-cli -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" shutdown |
| 查看集群状态 | /opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" cluster nodes |
| 查看slot分布 | /opt/software/redis-7.4.5/src/redis-cli --cluster check 192.168.1.83:7001 -a "RedisCluster@Secure2023!" |
| 主从信息 | /opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" info replication |
| 集群健康 | /opt/software/redis-7.4.5/src/redis-cli -c -h 192.168.1.83 -p 7001 -a "RedisCluster@Secure2023!" info |
| slot修复 | /opt/software/redis-7.4.5/src/redis-cli --cluster fix 192.168.1.83:7001 -a "RedisCluster@Secure2023!" |
| 集群扩容 | /opt/software/redis-7.4.5/src/redis-cli --cluster add-node 10.45.161.86:7001 192.168.1.83:7001 -a "RedisCluster@Secure2023!" |
| 集群缩容 | /opt/software/redis-7.4.5/src/redis-cli --cluster del-node 192.168.1.83:7001 <NODE_ID> -a "RedisCluster@Secure2023!" |