主从复制实现
- 当master出现故障后,可以自动提升一个slave节点变成新的Mster,因此Redis Slave 需要设置和master相同的连接密码，此外当一个Slave提升为新的master 通过持久化实现数据的恢复
- 当配置Redis复制功能时，强烈建议打开主服务器的持久化功能。否则的话，由于延迟等问题，部署的主节点Redis服务应该要避免自动启动。
- 特性
- 一个master可以有多个slave
- 一个slave只能有一个master
- 数据流向是从master到slave单向的
- master 可读可写
- slave 只读
- 首次全量复制,之后的复制基本增量复制实现
- Redis 的主从同步是非阻塞的，即同步过程不会影响主服务器的正常访问.
案例：一主两从实现Redis主从复制
#主节点安装redis并且准备了测试数据
#注意：配置从节点前保证从节点数据库干净，或者转移数据，配置主从同步生效后从节点数据被清空
[root@master ~]#redis-cli
127.0.0.1:6379> dbsize
(error) NOAUTH Authentication required.
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> dbsize
(integer) 1000
127.0.0.1:6379> role
1) "master"
2) (integer) 1022
3) 1) 1) "192.168.122.3"
2) "6379"
3) "1022"
127.0.0.1:6379> info
...
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.122.3,port=6379,state=online,offset=1092,lag=1
master_failover_state:no-failover
master_replid:8fe17fd654c200e38dc6cf58edd86f660348186d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1092
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1092
...
127.0.0.1:6379> exit
[root@master ~]#ls
install_redis.sh redis-7.0.7.tar.gz redis_test.sh
#配置从节点
[root@slave1 ~]#vim /apps/redis/etc/redis.conf
[root@slave1 ~]#cat /apps/redis/etc/redis.conf |grep replicaof
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport> #指定master的IP和端口号
replicaof 192.168.122.2 6379
[root@slave1 ~]#cat /apps/redis/etc/redis.conf |grep masterauth
# masterauth <master-password> #如果密码需要设置
masterauth 123456
[root@slave1 ~]#cat /apps/redis/etc/redis.conf |grep requirepass
# If the master is password protected (using the "requirepass" configuration
# IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatibility
# The requirepass is not compatible with aclfile option and the ACL LOAD
# command, these will cause requirepass to be ignored.
# requirepass foobared #和masterauth保持一致，用于将来从节点提升主后使用
requirepass 123456
#重启从节点redis服务后，实现主从同步
[root@slave1 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> dbsize
(integer) 0
127.0.0.1:6379> exit
[root@slave1 ~]#systemctl restart redis
[root@slave1 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> dbsize
(integer) 1000
127.0.0.1:6379> role
1) "slave"
2) "192.168.122.2"
3) (integer) 6379
4) "connected"
5) (integer) 966
127.0.0.1:6379> info
...
# Replication
role:slave
master_host:192.168.122.2
master_port:6379
master_link_status:up
master_last_io_seconds_ago:4
master_sync_in_progress:0
slave_read_repl_offset:1246
slave_repl_offset:1246
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:8fe17fd654c200e38dc6cf58edd86f660348186d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1246
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1246
127.0.0.1:6379> exit
#尝试断开主节点通信
[root@slave1 ~]#iptables -A INPUT -s 192.168.122.2 -j REJECT
[root@slave1 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info
...
# Replication
role:slave
master_host:192.168.122.2
master_port:6379
master_link_status:down -->状态
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_read_repl_offset:2030
slave_repl_offset:2030
master_link_down_since_seconds:1
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:8fe17fd654c200e38dc6cf58edd86f660348186d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2030
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:2030
总结：主从同步完整过程
- slave发起连接master，验证通过后,发送PSYNC命令
- master接收到PSYNC命令后，执行BGSAVE命令将全部数据保存至RDB文件中,并将后续发生的写操作记录至buffer中
- master向所有slave发送RDB文件
- master向所有slave发送后续记录在buffer中写操作
- slave收到快照文件后丢弃所有旧数据
- slave加载收到的RDB到内存
- slave 执行来自master接收到的buffer写操作
- 当slave完成全量复制后,后续master只会先发送slave_repl_offset信息，以后slave比较自身和master的差异,只会进行增量复制的数据即可
常见主从复制故障
- 主从硬件和软件配置不一致
- 主从节点的maxmemory不一致,主节点内存大于从节点内存,主从复制可能丢失数据
- rename-command 命令不一致,如在主节点启用flushdb,从节点禁用此命令,结果在master节点执行flushdb后,导致slave节点不同步
- master密码错误
- 如果slave节点配置的master密码错误，导致验证不通过,自然将无法建立主从同步关系
- Redis版本不一致
- 不同的redis 版本之间尤其是大版本间可能会存在兼容性问题，如：Redis 3,4,5,6之间，因此主从复制的所有节点应该使用相同的版本
- 安全模式下无法远程连接
- 如果开启了安全模式，并且没有设置bind地址和密码,会导致无法远程连接
集群搭建-哨兵Sentinel-实现主从复制结构的故障转移
工作模式：在客户端和服务器之间加Sentinel作为配置中心
原理：
- 专门的Sentinel 服务进程是用于监控redis集群中Master工作的状态，当Master主服务器发生故障的时候，可以实现Master和Slave的角色的自动切换，从而实现系统的高可用性
- Sentinel是一个分布式系统,即需要在多个节点上各自同时运行一个sentinel进程，Sentienl 进程通过流言协议(gossip protocols)来接收关于Master是否下线状态，并使用投票协议(Agreement Protocols)来决定是否执行自动故障转移,并选择合适的Slave作为新的Master
- 每个Sentinel进程会向其它Sentinel、Master、Slave定时发送消息，来确认对方是否存活，如果发现某个节点在指定配置时间内未得到响应，则会认为此节点已离线，即为主观宕机Subjective Down，简称为 SDOWN
- 如果哨兵集群中的多数Sentinel进程认为Master存在SDOWN，共同利用 is-master-down-by-addr 命令互相通知后，则认为客观宕机Objectively Down， 简称 ODOWN
- 接下来利用投票算法，从所有slave节点中，选一台合适的slave将之提升为新Master节点，然后自动修改其它slave相关配置，指向新的master节点,最终实现故障转移failover
- Redis Sentinel中的Sentinel节点个数应该为大于等于3且最好为奇数
- 客户端初始化时连接的是Sentinel节点集合，不再是具体的Redis节点，即 Sentinel只是配置中心不是代理。
- Redis Sentinel 节点与普通 Redis 没有区别,要实现读写分离依赖于客户端程序
- Sentinel 机制类似于MySQL中的MHA功能,只解决master和slave角色的自动故障转移问题，但单个Master 的性能瓶颈问题并没有解决
- Redis 3.0 之前版本中,生产环境一般使用哨兵模式较多,Redis 3.0后推出Redis cluster功能,可以支持更大规模的高并发环境
Sentinel中的三个定时任务：
- 每10 秒每个sentinel 对master和slave执行info
- 发现slave节点
- 确认主从关系
- 消息队列订阅者模式，每2秒每个sentinel通过master节点的channel交换信息(pub/sub)
- 通过sentinel__:hello频道交互
- 交互对节点的“看法”和自身信息
- 每1秒每个sentinel对其他sentinel和redis执行ping
- 测试是否存活，类似keepalived的心跳线
案例：复用Redis主从服务器搭建Sentinel
- 实现主从复制
#哨兵的前提是已经实现了Redis的主从复制
#主节点
[root@master ~]#cat /apps/redis/etc/redis.conf |grep replicaof ; cat /apps/redis/etc/redis.conf |grep masterauth ; cat /apps/redis/etc/redis.conf |grep requirepass
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport>
# masterauth <master-password>
# If the master is password protected (using the "requirepass" configuration
# IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatibility
# The requirepass is not compatible with aclfile option and the ACL LOAD
# command, these will cause requirepass to be ignored.
# requirepass foobared
requirepass 123456
#从节点
[root@slave1 ~]#vim /apps/redis/etc/redis.conf
[root@slave1 ~]#cat /apps/redis/etc/redis.conf |grep replicaof ; cat /apps/redis/etc/redis.conf |grep masterauth ; cat /apps/redis/etc/redis.conf |grep requirepass
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport>
replicaof 192.168.122.2 6379
# masterauth <master-password>
masterauth 123456
# If the master is password protected (using the "requirepass" configuration
# IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatibility
# The requirepass is not compatible with aclfile option and the ACL LOAD
# command, these will cause requirepass to be ignored.
# requirepass foobared
requirepass 123456
[root@slave2 ~]#vim /apps/redis/etc/redis.conf
[root@slave2 ~]#cat /apps/redis/etc/redis.conf |grep replicaof ; cat /apps/redis/etc/redis.conf |grep masterauth ; cat /apps/redis/etc/redis.conf |grep requirepass
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport>
replicaof 192.168.122.2 6379
# masterauth <master-password>
masterauth 123456
# If the master is password protected (using the "requirepass" configuration
# IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatibility
# The requirepass is not compatible with aclfile option and the ACL LOAD
# command, these will cause requirepass to be ignored.
# requirepass foobared
requirepass 123456
- 搭建Sentinel
- Sentinel实际上是一个特殊的redis服务器,有些redis指令支持,但很多指令并不支持.默认监听在26379/tcp端口.
- 哨兵服务可以和Redis服务器分开部署在不同主机，但为了节约成本一般会部署在一起
#所有redis节点使用相同的以下示例的配置文件
#1. copy安装时自带的sentinel配置文件到redis安装目录下
[root@master ~]#cd /usr/local/src/redis-7.0.7/
[root@master redis-7.0.7]#ls
00-RELEASENOTES CODE_OF_CONDUCT.md COPYING MANIFESTO README.md TLS.md redis.conf runtest-cluster runtest-sentinel src utils
BUGS CONTRIBUTING.md INSTALL Makefile SECURITY.md deps runtest runtest-moduleapi sentinel.conf tests
[root@master redis-7.0.7]#cp sentinel.conf /apps/redis/etc/sentinel.conf
[root@master redis-7.0.7]#chown redis.redis /apps/redis/etc/sentinel.conf
[root@master redis-7.0.7]#cd /apps/redis/etc/
[root@master etc]#ll
total 132
drwxr-xr-x 2 redis redis 4096 Feb 1 16:20 ./
drwxr-xr-x 7 redis redis 4096 Feb 1 13:53 ../
-rw-r--r-- 1 redis redis 106610 Feb 1 13:53 redis.conf
-rw-r--r-- 1 redis redis 14005 Feb 1 16:20 sentinel.conf
#2. 更改sentinel配置文件
[root@master etc]#vim sentinel.conf
[root@master etc]#grep -Ev "#|^$" /apps/redis/etc/sentinel.conf
protected-mode no
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile "/apps/redis/log/sentinel.log" #生成日志文件
dir /tmp
-----------------------------------
#2为法定人数限制(quorum)，即有几个sentinel认为master down了就进行故障转移，一般此值是所有sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2,是master的ODOWN客观下线的依据
------------------------------------
sentinel monitor mymaster 192.168.122.2 6379 2
sentinel auth-pass mymaster 123456 #mymaster集群中master的密码，注意此行要在上面行的下面
sentinel down-after-milliseconds mymaster 3000 #判断mymaster集群中所有节点的主观下线(SDOWN)的时间，单位：毫秒，建议3000
acllog-max-len 128
sentinel parallel-syncs mymaster 1 #发生故障转移后，可以同时向新master同步数据的slave的数量，数字越小总同步时间越长，但可以减轻新master的负载压力
sentinel failover-timeout mymaster 180000 #所有slaves指向新的master所需的超时时间，单位：毫秒
sentinel deny-scripts-reconfig yes #禁止修改脚本
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no
SENTINEL master-reboot-down-after-period mymaster 0
#3. 复制配置文件到各个节点
[root@master etc]#rsync -a sentinel.conf 192.168.122.3:/apps/redis/etc/
[root@master etc]#rsync -a sentinel.conf 192.168.122.4:/apps/redis/etc/
#4. 在所有节点准备Service文件
[root@master etc]#vim /lib/systemd/system/redis-sentinel.service
[root@master etc]#cat /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target
[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectory
Mode=0755
[Install]
WantedBy=multi-user.target
[root@master etc]#systemctl daemon-reload
[root@master etc#systemctl enable --now redis-sentinel.service
Created symlink /etc/systemd/system/multi-user.target.wants/redis-sentinel.service → /lib/systemd/system/redis-sentinel.service.
#复制到其他节点并生成服务文件
[root@master etc]#scp /lib/systemd/system/redis-sentinel.service 192.168.122.3:/lib/systemd/system/redis-sentinel.service
root@192.168.122.3's password:
redis-sentinel.service 100% 305 173.8KB/s 00:00
[root@master etc]#scp /lib/systemd/system/redis-sentinel.service 192.168.122.4:/lib/systemd/system/redis-sentinel.service
root@192.168.122.4's password:
redis-sentinel.service 100% 305 195.9KB/s 00:00
#5. 确保所有节点的目录权限
[root@master ~]#chown -R redis.redis /apps/redis/
[root@slave1 ~]#chown -R redis.redis /apps/redis/
[root@slave2 ~]#chown -R redis.redis /apps/redis/
- 验证哨兵服务
#查看哨兵服务端口状态
[root@master ~]#ss -ntlp|grep redis
LISTEN 0 511 0.0.0.0:26379 0.0.0.0:* users:(("redis-sentinel",pid=7225,fd=6))
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:(("redis-server",pid=5914,fd=6))
LISTEN 0 511 [::]:26379 [::]:* users:(("redis-sentinel",pid=7225,fd=7))
LISTEN 0 511 [::1]:6379 [::]:* users:(("redis-server",pid=5914,fd=7))
[root@slave1 ~]#ss -ntlp|grep redis
LISTEN 0 511 0.0.0.0:26379 0.0.0.0:* users:(("redis-sentinel",pid=6763,fd=6))
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:(("redis-server",pid=6228,fd=6))
LISTEN 0 511 [::]:26379 [::]:* users:(("redis-sentinel",pid=6763,fd=7))
LISTEN 0 511 [::1]:6379 [::]:* users:(("redis-server",pid=6228,fd=7))
[root@slave2 ~]#ss -ntlp|grep redis
LISTEN 0 511 0.0.0.0:26379 0.0.0.0:* users:(("redis-sentinel",pid=6227,fd=6))
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:(("redis-server",pid=5946,fd=6))
LISTEN 0 511 [::]:26379 [::]:* users:(("redis-sentinel",pid=6227,fd=7))
LISTEN 0 511 [::1]:6379 [::]:* users:(("redis-server",pid=5946,fd=7))
#查看哨兵日志
[root@master ~]#tail -f /apps/redis/log/sentinel.log
7225:X 01 Feb 2023 16:53:50.939 # systemd supervision error: NOTIFY_SOCKET not found!
7225:X 01 Feb 2023 16:53:50.939 # systemd supervision error: NOTIFY_SOCKET not found!
7225:X 01 Feb 2023 16:53:50.941 * +slave slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.2 6379
7225:X 01 Feb 2023 16:53:50.968 * Sentinel new configuration saved on disk
7225:X 01 Feb 2023 16:53:50.968 * +slave slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
7225:X 01 Feb 2023 16:53:51.026 * Sentinel new configuration saved on disk
7225:X 01 Feb 2023 16:53:58.391 * +sentinel sentinel a75b6523c1efefba98341c5bb9ec766e1d9b78bf 192.168.122.3 26379 @ mymaster 192.168.122.2 6379
7225:X 01 Feb 2023 16:53:58.435 * Sentinel new configuration saved on disk
7225:X 01 Feb 2023 16:54:01.231 * +sentinel sentinel f0866463210b4142e372b4fc61084b4a4eff7ab0 192.168.122.4 26379 @ mymaster 192.168.122.2 6379
7225:X 01 Feb 2023 16:54:01.267 * Sentinel new configuration saved on disk
[root@slave1 ~]#tail -f /apps/redis/log/sentinel.log
6763:X 01 Feb 2023 16:53:56.379 # systemd supervision error: NOTIFY_SOCKET not found!
6763:X 01 Feb 2023 16:53:56.379 # systemd supervision error: NOTIFY_SOCKET not found!
6763:X 01 Feb 2023 16:53:56.381 * +slave slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.2 6379
6763:X 01 Feb 2023 16:53:56.431 * Sentinel new configuration saved on disk
6763:X 01 Feb 2023 16:53:56.431 * +slave slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
6763:X 01 Feb 2023 16:53:56.452 * Sentinel new configuration saved on disk
6763:X 01 Feb 2023 16:53:57.090 * +sentinel sentinel 4c1c50da9885685a85f653107c5166cb650123dd 192.168.122.2 26379 @ mymaster 192.168.122.2 6379
6763:X 01 Feb 2023 16:53:57.132 * Sentinel new configuration saved on disk
6763:X 01 Feb 2023 16:54:01.227 * +sentinel sentinel f0866463210b4142e372b4fc61084b4a4eff7ab0 192.168.122.4 26379 @ mymaster 192.168.122.2 6379
6763:X 01 Feb 2023 16:54:01.261 * Sentinel new configuration saved on disk
[root@slave2 ~]#tail -f /apps/redis/log/sentinel.log
6227:X 01 Feb 2023 16:53:59.287 # systemd supervision error: NOTIFY_SOCKET not found!
6227:X 01 Feb 2023 16:53:59.287 # systemd supervision error: NOTIFY_SOCKET not found!
6227:X 01 Feb 2023 16:53:59.289 * +slave slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 16:53:59.312 * Sentinel new configuration saved on disk
6227:X 01 Feb 2023 16:53:59.312 * +slave slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 16:53:59.344 * Sentinel new configuration saved on disk
6227:X 01 Feb 2023 16:54:00.460 * +sentinel sentinel a75b6523c1efefba98341c5bb9ec766e1d9b78bf 192.168.122.3 26379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 16:54:00.493 * Sentinel new configuration saved on disk
6227:X 01 Feb 2023 16:54:01.180 * +sentinel sentinel 4c1c50da9885685a85f653107c5166cb650123dd 192.168.122.2 26379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 16:54:01.246 * Sentinel new configuration saved on disk
#确认sentien配置文件中生成了集成信息
[root@master ~]#cat /apps/redis/etc/sentinel.conf
....
# Generated by CONFIG REWRITE
supervised systemd
latency-tracking-info-percentiles 50 99 99.9
user default on nopass ~* &* +@all
sentinel myid 4c1c50da9885685a85f653107c5166cb650123dd --> 确保id不一致，一般是自动生成
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 192.168.122.4 6379
sentinel known-replica mymaster 192.168.122.3 6379
sentinel known-sentinel mymaster 192.168.122.4 26379 f0866463210b4142e372b4fc61084b4a4eff7ab0
sentinel known-sentinel mymaster 192.168.122.3 26379 a75b6523c1efefba98341c5bb9ec766e1d9b78bf
[root@slave1 ~]#cat /apps/redis/etc/sentinel.conf
....
# Generated by CONFIG REWRITE
supervised systemd
latency-tracking-info-percentiles 50 99 99.9
user default on nopass ~* &* +@all
sentinel myid a75b6523c1efefba98341c5bb9ec766e1d9b78bf
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 192.168.122.4 6379
sentinel known-replica mymaster 192.168.122.3 6379
sentinel known-sentinel mymaster 192.168.122.4 26379 f0866463210b4142e372b4fc61084b4a4eff7ab0
sentinel known-sentinel mymaster 192.168.122.2 26379 4c1c50da9885685a85f653107c5166cb650123dd
[root@slave2 ~]#cat /apps/redis/etc/sentinel.conf
....
# Generated by CONFIG REWRITE
supervised systemd
latency-tracking-info-percentiles 50 99 99.9
user default on nopass ~* &* +@all
sentinel myid f0866463210b4142e372b4fc61084b4a4eff7ab0
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 192.168.122.3 6379
sentinel known-replica mymaster 192.168.122.4 6379
sentinel known-sentinel mymaster 192.168.122.3 26379 a75b6523c1efefba98341c5bb9ec766e1d9b78bf
sentinel known-sentinel mymaster 192.168.122.2 26379 4c1c50da9885685a85f653107c5166cb650123dd
#当前sentinel状态
[root@master ~]#redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.122.2:6379,slaves=2,sentinels=3
[root@slave1 ~]#redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.122.2:6379,slaves=2,sentinels=3
[root@slave2 ~]#redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.122.2:6379,slaves=2,sentinels=3
- 开始测试故障转移功能
- 停止 Master 节点，观察日志
[root@master ~]#killall redis-server
[root@slave1 ~]#tail -f /apps/redis/log/sentinel.log
...
6763:X 01 Feb 2023 17:17:42.652 # +sdown master mymaster 192.168.122.2 6379
6763:X 01 Feb 2023 17:17:42.788 * Sentinel new configuration saved on disk
6763:X 01 Feb 2023 17:17:42.788 # +new-epoch 1
6763:X 01 Feb 2023 17:17:42.821 * Sentinel new configuration saved on disk
6763:X 01 Feb 2023 17:17:42.821 # +vote-for-leader f0866463210b4142e372b4fc61084b4a4eff7ab0 1
6763:X 01 Feb 2023 17:17:42.822 # +odown master mymaster 192.168.122.2 6379 #quorum 3/2
6763:X 01 Feb 2023 17:17:42.822 # Next failover delay: I will not start a failover before Wed Feb 1 17:23:42 2023
6763:X 01 Feb 2023 17:17:43.804 # +config-update-from sentinel f0866463210b4142e372b4fc61084b4a4eff7ab0 192.168.122.4 26379 @ mymaster 192.168.122.2 6379
6763:X 01 Feb 2023 17:17:43.804 # +switch-master mymaster 192.168.122.2 6379 192.168.122.4 6379
6763:X 01 Feb 2023 17:17:43.804 * +slave slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.4 6379
6763:X 01 Feb 2023 17:17:43.804 * +slave slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
6763:X 01 Feb 2023 17:17:43.898 * Sentinel new configuration saved on disk
6763:X 01 Feb 2023 17:17:46.864 # +sdown slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
[root@slave2 ~]#tail -f /apps/redis/log/sentinel.log
...
6227:X 01 Feb 2023 17:17:42.635 # +sdown master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:42.692 # +odown master mymaster 192.168.122.2 6379 #quorum 2/2
6227:X 01 Feb 2023 17:17:42.692 # +new-epoch 1
6227:X 01 Feb 2023 17:17:42.692 # +try-failover master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:42.738 * Sentinel new configuration saved on disk
6227:X 01 Feb 2023 17:17:42.739 # +vote-for-leader f0866463210b4142e372b4fc61084b4a4eff7ab0 1
6227:X 01 Feb 2023 17:17:42.804 # 4c1c50da9885685a85f653107c5166cb650123dd voted for f0866463210b4142e372b4fc61084b4a4eff7ab0 1
6227:X 01 Feb 2023 17:17:42.820 # a75b6523c1efefba98341c5bb9ec766e1d9b78bf voted for f0866463210b4142e372b4fc61084b4a4eff7ab0 1
6227:X 01 Feb 2023 17:17:42.868 # +elected-leader master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:42.868 # +failover-state-select-slave master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:42.970 # +selected-slave slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:42.970 * +failover-state-send-slaveof-noone slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:43.029 * +failover-state-wait-promotion slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:43.765 * Sentinel new configuration saved on disk
6227:X 01 Feb 2023 17:17:43.765 # +promoted-slave slave 192.168.122.4:6379 192.168.122.4 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:43.765 # +failover-state-reconf-slaves master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:43.801 * +slave-reconf-sent slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:43.919 # -odown master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:44.797 * +slave-reconf-inprog slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:44.797 * +slave-reconf-done slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:44.849 # +failover-end master mymaster 192.168.122.2 6379
6227:X 01 Feb 2023 17:17:44.849 # +switch-master mymaster 192.168.122.2 6379 192.168.122.4 6379
6227:X 01 Feb 2023 17:17:44.850 * +slave slave 192.168.122.3:6379 192.168.122.3 6379 @ mymaster 192.168.122.4 6379
6227:X 01 Feb 2023 17:17:44.850 * +slave slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
6227:X 01 Feb 2023 17:17:44.913 * Sentinel new configuration saved on disk
6227:X 01 Feb 2023 17:17:47.860 # +sdown slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
- 故障转移后redis.conf中的replicaof行的master IP会被修改
[root@slave1 ~]#grep ^replicaof /apps/redis/etc/redis.conf
replicaof 192.168.122.4 6379
[root@slave2 ~]#grep replicaof /apps/redis/etc/redis.conf
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport>
- 哨兵配置文件的sentinel monitor IP 同样也会被修改
[root@slave1 ~]#grep -Ev "#|^$" /apps/redis/etc/sentinel.conf
protected-mode no
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.122.4 6379 2 --> ip变更
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
supervised systemd
latency-tracking-info-percentiles 50 99 99.9
user default on nopass ~* &* +@all
sentinel myid a75b6523c1efefba98341c5bb9ec766e1d9b78bf
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.122.2 6379
sentinel known-replica mymaster 192.168.122.3 6379
sentinel known-sentinel mymaster 192.168.122.4 26379 f0866463210b4142e372b4fc61084b4a4eff7ab0
sentinel known-sentinel mymaster 192.168.122.2 26379 4c1c50da9885685a85f653107c5166cb650123dd
- 确认slave 提升成master
[root@slave2 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> role
1) "master"
2) (integer) 546715
3) 1) 1) "192.168.122.3"
2) "6379"
3) "546715"
- 原master重新加入Redis集群
#原Master的变化
[root@master ~]#systemctl start redis
[root@master ~]#tail -f /apps/redis/log/sentinel.log
...
7225:X 01 Feb 2023 17:40:06.028 # -sdown slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
7225:X 01 Feb 2023 17:40:16.032 * +convert-to-slave slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
[root@master ~]#grep -Ev "#|^$" /apps/redis/etc/sentinel.conf
protected-mode no
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.122.4 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
supervised systemd
latency-tracking-info-percentiles 50 99 99.9
user default on nopass ~* &* +@all
sentinel myid 4c1c50da9885685a85f653107c5166cb650123dd
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.122.3 6379
sentinel known-replica mymaster 192.168.122.2 6379
sentinel known-sentinel mymaster 192.168.122.4 26379 f0866463210b4142e372b4fc61084b4a4eff7ab0
sentinel known-sentinel mymaster 192.168.122.3 26379 a75b6523c1efefba98341c5bb9ec766e1d9b78bf
[root@master ~]#redis-cli -a 123456 role
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1) "slave"
2) "192.168.122.4"
3) (integer) 6379
4) "handshake"
5) (integer) -1
#新Master日志
[root@slave2 ~]#tail -f /apps/redis/log/sentinel.log
...
6227:X 01 Feb 2023 17:40:06.122 # -sdown slave 192.168.122.2:6379 192.168.122.2 6379 @ mymaster 192.168.122.4 6379
我是Moore,新的一年，希望大家都好运！深夜写文章还是很痛快的....