Redis主从复制、哨兵实现，并模拟master故障场景

Redis是一个开源（BSD许可），内存中数据结构存储，用作数据库，缓存和消息代理。它支持数据结构：如字符串、哈希、列表、集、带范围查询的排序集、位图、超日志、带半径查询和流的地理空间索引。Redis 具有内置的复制、Lua 脚本、LRU 逐出、事务和不同级别的磁盘持久性，并通过 Redis Sentinel 和 Redis Cluster 的自动分区提供高可用性。

本文将实践redis的主从复制和哨兵。

1. 架构拓扑及主机

IP

主机名

角色

软件

192.168.250.18

Redis-IP18

Redis 主

Redis6.2.6

192.168.250.28

Redis-IP28

Redis 从1

Redis6.2.6

192.168.250.38

Redis-IP38

Redis 从2

Redis6.2.6

环境准备：
时间同步，确保NTP或Chrony服务正常运行。
禁用SELinux和防火墙（或放通需要的端口）

2. 准备三台Redis主机

任务内容：初始化CentOS、修改规划好的主机名、通过脚本一键安装好Redis。

#### 只列出 IP18的操作过程，同样的操作步骤完成 IP28和IP38的Redis安装
[root@CentOS84-IP18 ]#hostnamectl set-hostname Redis-IP18
[root@CentOS84-IP18 ]#exit
[root@Redis-IP18 ]#cd /data/
[root@Redis-IP18 ]#
[root@Redis-IP18 ]#bash install_redis.sh
[root@Redis-IP18 ]#

[root@Redis-IP18 ]#rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.
Transferring install_redis.sh...
100% 2 KB 2 KB/sec 00:00:01 0 Errors

[root@Redis-IP18 ]#ll /data/
total 4
-rw-r--r-- 1 root root 2854 Apr 26 19:01 install_redis.sh
# 验证安装
[root@Redis-IP18 ]#ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:*
LISTEN 0 511 [::1]:6379 [::]:*
[root@Redis-IP18 ]#

3. 配置Redis主从复制

3.1 概述

虽然Redis可以实现单机的数据持久化，但无论是RDB也好或者AOF也好，都解决不了单点宕机问题，即一旦单台 redis服务器本身出现系统故障、硬件故障等问题后，就会直接造成数据的丢失。此外，单机的性能也是有极限的，因此需要使用另外的技术，比如复制、哨兵、集群等来解决单点故障和性能扩展的问题。

默认所有的redis 安装完成后的状态都为master，需要将规划用来作为slave的redis主机，进行角色修改并指向master服务器的IP+PORT+Password。在从节点执行REPLICAOF MASTER_IP PORT 指令可以启用主从同步复制功能,早期版本使用 SLAVEOF 。

基本操作命令：

# 这样的操作命令
127.0.0.1:6379> REPLICAOF MASTER_IP PORT
127.0.0.1:6379> CONFIG SET masterauth <masterpass>

3.2 主从Redis中复制架构的配置

哨兵的前提是已经实现了一个redis的主从复制的运行环境，从而实现一个一主两从基于哨兵的高可用redis架构

注意: master 的配置文件中masterauth 和slave 都必须相同。所有主从节点的redis.conf中关健配置如下：

所有主从节点都要执行的配置

## 因为是编译安装的，redis.conf 文件的路径和 yum安装（ /etc/redis.conf ）的不同。
[root@Redis ]#vim /apps/redis/etc/redis.conf
bind 0.0.0.0 # 修改redis监听，下面的密码可以自行定义，但必须对应好
masterauth "123456"
requirepass "123456"

# yum 安装一般只需要修改上面的项，因本实践是编译安装，按照自定义的目录结构修改下面的几项。
pidfile "/apps/redis/run/redis_6379.pid"
logfile "/apps/redis/log/redis-6379.log"
dir "/apps/redis/data"

所有从节点都要执行的配置

# 将所有的从redis都指向主redis，同样也可以自行定义下从redis的优先级别 replica-priority 100 这个值可以修改下，实现从节点的优先级自定义
[root@Redis ]#echo "replicaof 192.168.250.18 6379" >> /etc/redis.conf

3.2.1 主redis-IP18节点的配置及验证复制状态

经验和技巧：在学习一个新的软件，我们曾经经常提醒一般都先yum安装，读懂其文件结构和关键性配置文件的内容，这样有利于编译自定义安装后借鉴，在官方提供的配置文件内做了很多注释性的解读，非常人性化，内容很多很长，我们仅通过命令提取出实际生效行。

# redis.conf 内实际上校的行 cat /apps/redis/etc/redis.conf 列出的文件内容过于冗长。特别要注意的是复制和借鉴的文件，要查看下权限，避免不必要的错误。

[root@Redis-IP18 ]#egrep -v "^#|^$" /apps/redis/etc/redis.conf
bind 0.0.0.0 -::1
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
pidfile "/apps/redis/run/redis_6379.pid"
loglevel notice
logfile "/apps/redis/log/redis-6379.log"
databases 16
always-show-logo no
set-proc-title yes
proc-title-template "{title} {listen-addr} {server-mode}"
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
rdb-del-sync-files no
dir "/apps/redis/data"
masterauth "123456"
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-diskless-load disabled
repl-disable-tcp-nodelay no
replica-priority 100
acllog-max-len 128
requirepass "123456"
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no
lazyfree-lazy-user-flush no
oom-score-adj no
oom-score-adj-values 0 200 800
disable-thp yes
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4kb
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
supervised systemd
save 3600 1
save 300 100
save 60 10000
user default on sanitize-payload #8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92 ~* &* +@all
[root@Redis-IP18 ]#

##############################################################################
## 验证主从复制，主节点上的状态
[root@Redis-IP18 ]#redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.250.38,port=6379,state=online,offset=51770731,lag=1
slave1:ip=192.168.250.28,port=6379,state=online,offset=51770731,lag=1
master_failover_state:no-failover
master_replid:f14a4fed51b884834035a13d89829ff621bfeef7
master_replid2:0137cb1fa513d18c18a764ac95cbbc2916585710
master_repl_offset:51770874
second_repl_offset:51638431
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:50722299
repl_backlog_histlen:1048576
127.0.0.1:6379>

3.2.2 从redis-IP28节点的配置及验证复制状态

#### 过滤出有效的配置行
[root@Redis-IP28 ]#egrep -v "^#|^$" /apps/redis/etc/redis.conf
bind 0.0.0.0 -::1
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
pidfile "/apps/redis/run/redis_6379.pid"
loglevel notice
logfile "/apps/redis/log/redis-6379.log"
databases 16
always-show-logo no
set-proc-title yes
proc-title-template "{title} {listen-addr} {server-mode}"
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
rdb-del-sync-files no
dir "/apps/redis/data"
replicaof 192.168.250.18 6379
masterauth "123456"
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-diskless-load disabled
repl-disable-tcp-nodelay no
replica-priority 100
acllog-max-len 128
requirepass "123456"
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no
lazyfree-lazy-user-flush no
oom-score-adj no
oom-score-adj-values 0 200 800
disable-thp yes
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4kb
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
supervised systemd
save 3600 1
save 300 100
save 60 10000
user default on #8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92 ~* &* +@all
[root@Redis-IP28 ]#

##############################################################################
## 验证主从复制，从节点上的状态
[root@Redis-IP28 ]#redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.250.18
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:51860994
slave_repl_offset:51860994
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:f14a4fed51b884834035a13d89829ff621bfeef7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:51860994
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:51729425
repl_backlog_histlen:131570
127.0.0.1:6379>

3.2.3 从redis-IP38节点的配置及验证复制状态

#### 过滤出有效配置行
[root@Redis-IP38 ]#egrep -v "^#|^$" /apps/redis/etc/redis.conf
bind 0.0.0.0 -::1
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
pidfile "/apps/redis/run/redis_6379.pid"
loglevel notice
logfile "/apps/redis/log/redis-6379.log"
databases 16
always-show-logo no
set-proc-title yes
proc-title-template "{title} {listen-addr} {server-mode}"
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
rdb-del-sync-files no
dir "/apps/redis/data"
masterauth "123456"
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-diskless-load disabled
repl-disable-tcp-nodelay no
replica-priority 80
acllog-max-len 128
requirepass "123456"
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no
lazyfree-lazy-user-flush no
oom-score-adj no
oom-score-adj-values 0 200 800
disable-thp yes
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4kb
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
supervised systemd
save 3600 1
save 300 100
save 60 10000
user default on sanitize-payload #8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92 ~* &* +@all
replicaof 192.168.250.18 6379
[root@Redis-IP38 ]#

##############################################################################
## 验证主从复制，从节点上的状态
[root@Redis-IP38 ]#redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.250.18
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:51891220
slave_repl_offset:51891220
slave_priority:80
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:f14a4fed51b884834035a13d89829ff621bfeef7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:51891220
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:51729259
repl_backlog_histlen:161962
127.0.0.1:6379>

结语：redis的主从复制有很多可以根据业务优化的选项，内容比较多，也相对比较重要，在实际生产中可以根据需要摸索并实践利用好。

4. redis哨兵模式

4.1 概述

Sentinel实际上是一个特殊的redis服务器,有些redis指令支持,但很多指令并不支持.默认监听在26379/tcp端口。哨兵可以不和Redis服务器部署在一起，但一般部署在一起以节约成本。

4.2 配置Redis哨兵

#### 编译安装，没sentinel.conf 文件，从源码目录复制一个模板 sentinel.conf文件到安装目录进行修改
# 查找 sentinel.conf 模板文件
[root@Redis ]#find / -name sentinel.conf
/root/redis-6.2.6/sentinel.conf
/root/redis-6.2.6/tests/sentinel/tests/includes/sentinel.conf
[root@Redis ]#

# 默认的sentinel.conf有效配置行
[root@Redis ]#egrep -v "^#|^$" /root/redis-6.2.6/sentinel.conf
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile ""
dir /tmp
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no
[root@Redis-IP18 ]#

# 下面是yum安装后的
[root@centos ]# egrep -v "^#|^$" /etc/redis-sentinel.conf
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile /var/log/redis/sentinel.log
dir /tmp
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no

4.2.1 配置主Redis 哨兵sentinel.conf

#### 从源码目录复制一个模板 sentinel.conf文件到安装目录进行修改；修改后的配置文件

[root@Redis-IP18 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile /apps/redis/log/sentinel.log
dir /tmp
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no
[root@Redis-IP18 ]#


####################################################################################
# 将主redis上的 /apps/redis/etc/redis-sentinel.conf 文件复制到两个从redis上去，被挪用。
# 记录一个容易犯错的点：开始在从redis上启动哨兵服务无法启动，报错；后来发现是这个文件复制后的权属是root，需要修改成redis 。
[root@Redis-IP18 ]#
[root@Redis-IP18 ]#scp /apps/redis/etc/redis-sentinel.conf 192.168.250.28:/apps/redis/etc/
The authenticity of host '192.168.250.28 (192.168.250.28)' can't be established.
ECDSA key fingerprint is SHA256:WGibMK0eLfGqzsaTJEHUwYyD+RwjH6hlC0ZBURwn7ns.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.250.28' (ECDSA) to the list of known hosts.
root@192.168.250.28's password:
redis-sentinel.conf 100% 14KB 1.5MB/s 00:00

[root@Redis-IP18 ]#scp /apps/redis/etc/redis-sentinel.conf 192.168.250.38:/apps/redis/etc/
The authenticity of host '192.168.250.38 (192.168.250.38)' can't be established.
ECDSA key fingerprint is SHA256:WGibMK0eLfGqzsaTJEHUwYyD+RwjH6hlC0ZBURwn7ns.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.250.38' (ECDSA) to the list of known hosts.
root@192.168.250.38's password:
redis-sentinel.conf 100% 14KB 1.6MB/s 00:00
[root@Redis-IP18 ]#

####################################################################################
# 编写 /lib/systemd/system/redis-sentinel.service 文件
[root@Redis-IP18 ]#cat /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target

[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
[root@Redis-IP18 ]#

# 权属修改成redis
[root@Redis-IP18 ]#chown -R redis.redis /apps/redis/
[root@Redis-IP18 ]#
####################################################################################
[root@Redis-IP18 ]#systemctl enable --now redis-sentinel.service
[root@Redis-IP18 ]#systemctl status redis-sentinel.service
● redis-sentinel.service - Redis Sentinel
Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-04-29 19:31:21 CST; 11s ago
Main PID: 85116 (redis-sentinel)
Tasks: 5 (limit: 23544)
Memory: 1.9M
CGroup: /system.slice/redis-sentinel.service
└─85116 /apps/redis/bin/redis-sentinel 0.0.0.0:26379 [sentinel]

[root@Redis-IP18 ]#
# 查看端口监听
[root@Redis-IP18 ]#ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:26379 0.0.0.0:*
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:*
[root@Redis-IP18 ]#

# 启动哨兵服务文件后看到的/apps/redis/etc/redis-sentinel.conf 内信息。
[root@Redis-IP18 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass <sub>* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 192.168.250.38 6379
sentinel known-replica mymaster 192.168.250.28 6379
[root@Redis-IP18 ]#

####################################################################################
#### 将服务启动文件传送给两台从redis IP28 / IP38 备用
[root@Redis-IP18 ]#scp /lib/systemd/system/redis-sentinel.service 192.168.250.28:/lib/systemd/system/
root@192.168.250.28's password:
redis-sentinel.service 100% 310 201.1KB/s 00:00
[root@Redis-IP18 ]#scp /lib/systemd/system/redis-sentinel.service 192.168.250.38:/lib/systemd/system/
root@192.168.250.38's password:
redis-sentinel.service 100% 310 208.6KB/s 00:00
[root@Redis-IP18 ]#

#### 至此主redis上的 哨兵和复制都配置好了

####################################################################################
## 哨兵配置全部配置好后的/apps/redis/etc/redis-sentinel.conf 会在下面自动添加几行
# 特别提醒：sentinel myid 三个主机必须不一样，否则也无法启动
[root@Redis-IP18 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload </sub>* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
sentinel config-epoch mymaster 2
sentinel leader-epoch mymaster 2
sentinel current-epoch 2
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-replica mymaster 192.168.250.38 6379
sentinel known-sentinel mymaster 192.168.250.28 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
sentinel known-sentinel mymaster 192.168.250.38 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f838

4.2.2 配置从Redis-IP28 哨兵sentinel.conf

#### 在配置主redis节点，已经将修改好的redis-sentinel.conf 和redis-sentinel.service传输给两台从redis主机，查看信息如下
[root@Redis-IP28 ]#ll /apps/redis/etc/
total 108
-rw-r--r-- 1 redis redis 94016 May 2 15:54 redis.conf
-rw-r--r-- 1 redis redis 14287 May 2 15:44 redis-sentinel.conf
[root@Redis-IP28 ]#

## 修改 /apps/redis/etc/redis-sentinel.conf 后的配置信息，有效行如下
[root@Redis-IP28 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile /apps/redis/log/sentinel.log
dir /tmp
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no

# 下面这个权属必须修改成redis，这样才能成功启动redis-sentinel 服务
[root@Redis-IP28 ]#chown -R redis.redis /apps/redis/

[root@Redis-IP28 ]#systemctl enable --now redis-sentinel.service
[root@Redis-IP28 ]#systemctl status redis-sentinel.service
● redis-sentinel.service - Redis Sentinel
Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-04-29 20:12:15 CST; 2s ago
Main PID: 3968 (redis-sentinel)
Tasks: 5 (limit: 23544)
Memory: 5.3M
CGroup: /system.slice/redis-sentinel.service
└─3968 /apps/redis/bin/redis-sentinel 0.0.0.0:26379 [sentinel]

[root@Redis-IP28 ]#

####################################################################################
## 哨兵配置全部配置好后的/apps/redis/etc/redis-sentinel.conf 会在下面自动添加几行
# 特别提醒：sentinel myid 三个主机必须不一样，否则也无法启动
[root@Redis-IP28 ]#systemctl enable --now redis-sentinel.service
[root@Redis-IP28 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 192.168.250.38 6379
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-sentinel mymaster 192.168.250.18 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
sentinel known-sentinel mymaster 192.168.250.38 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
[root@Redis-IP28 ]#

####################################################################################

4.2.3 配置从Redis-IP38 哨兵sentinel.conf

#### 在配置主redis节点，已经将修改好的redis-sentinel.conf 和redis-sentinel.service传输给两台从redis主机，查看信息如下
[root@Redis-IP38 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload <sub>* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-replica mymaster 192.168.250.38 6379
[root@Redis-IP38 ]#

# 修改权属
[root@Redis-IP38 ]#chown -R redis.redis /apps/redis/
[root@Redis-IP38 ]#systemctl enable --now redis-sentinel.service
[root@Redis-IP38 ]#systemctl status redis-sentinel.service
● redis-sentinel.service - Redis Sentinel
Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-04-29 22:20:41 CST; 9s ago
Process: 5058 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 5087 (redis-sentinel)
Tasks: 5 (limit: 23544)
Memory: 1.9M
CGroup: /system.slice/redis-sentinel.service
└─5087 /apps/redis/bin/redis-sentinel 0.0.0.0:26379 [sentinel]

####################################################################################
## 哨兵配置全部配置好后的/apps/redis/etc/redis-sentinel.conf 会在下面自动添加几行
# 特别提醒：sentinel myid 三个主机必须不一样，否则也无法启动
[root@Redis-IP38 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.18 6379 2
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload </sub>* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
sentinel config-epoch mymaster 2
sentinel leader-epoch mymaster 2
sentinel current-epoch 2
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-replica mymaster 192.168.250.38 6379
sentinel known-sentinel mymaster 192.168.250.18 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
sentinel known-sentinel mymaster 192.168.250.28 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
[root@Redis-IP38 ]#

4.3 验证状态及查看日志

4.3.1 主redis-IP18哨兵状态及日志

## 主redis
[root@Redis-IP18 ]#ll /apps/redis/log
-rw-r--r-- 1 redis redis 31886 May 2 15:54 redis-6379.log
-rw-r--r-- 1 redis redis 11046 May 2 15:54 sentinel.log

[root@Redis-IP18 ]#tail -f /apps/redis/log/sentinel.log
85116:X 29 Apr 2022 19:31:21.273 # Configuration loaded
85116:X 29 Apr 2022 19:31:21.275 * Increased maximum number of open files to 10032 (it was originally set to 1024).
85116:X 29 Apr 2022 19:31:21.275 * monotonic clock: POSIX clock_gettime
85116:X 29 Apr 2022 19:31:21.276 * Running mode=sentinel, port=26379.
85116:X 29 Apr 2022 19:31:21.280 # Sentinel ID is 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
85116:X 29 Apr 2022 19:31:21.280 # +monitor master mymaster 192.168.250.18 6379 quorum 2
85116:X 29 Apr 2022 19:31:21.281 * +slave slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.18 6379
85116:X 29 Apr 2022 19:31:21.283 * +slave slave 192.168.250.38:6379 192.168.250.38 6379 @ mymaster 192.168.250.18 6379

####################################################################################
## 在sentinel状态中尤其是最后一行，涉及到masterIP是多少，有几个slave，有几个sentinels，必须是符合全部服务器数量
[root@Redis-IP18 ]#redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.250.18:6379,slaves=2,sentinels=3
127.0.0.1:26379>
#两个slave,三个sentinel服务器,如果sentinels值不符合,检查myid可能冲突

4.3.2 从redis-IP28哨兵状态及日志

# 日志信息
[root@Redis-IP28 ]#tail -f /apps/redis/log/sentinel.log
6229:X 30 Apr 2022 00:06:36.983 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
6229:X 30 Apr 2022 00:06:36.983 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6229:X 30 Apr 2022 00:06:36.983 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=6229, just started
6229:X 30 Apr 2022 00:06:36.983 # Configuration loaded
6229:X 30 Apr 2022 00:06:36.985 * Increased maximum number of open files to 10032 (it was originally set to 1024).
6229:X 30 Apr 2022 00:06:36.985 * monotonic clock: POSIX clock_gettime
6229:X 30 Apr 2022 00:06:36.986 * Running mode=sentinel, port=26379.
6229:X 30 Apr 2022 00:06:36.987 # Sentinel ID is 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
6229:X 30 Apr 2022 00:06:36.987 # +monitor master mymaster 192.168.250.18 6379 quorum 2

####################################################################################
[root@Redis-IP28 ]#redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.250.18:6379,slaves=2,sentinels=3
127.0.0.1:26379>

4.3.3 从redis-IP38哨兵状态及日志

# 日志信息
[root@Redis-IP38 ]#tail -f /apps/redis/log/sentinel.log
5087:X 29 Apr 2022 22:21:51.125 * +sentinel sentinel 8a3b79f833d85c73f046cdc5e31dfa266aa4f828 192.168.250.28 26379 @ mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:08:33.789 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
6111:X 30 Apr 2022 00:08:33.789 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6111:X 30 Apr 2022 00:08:33.789 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=6111, just started
6111:X 30 Apr 2022 00:08:33.789 # Configuration loaded
6111:X 30 Apr 2022 00:08:33.791 * Increased maximum number of open files to 10032 (it was originally set to 1024).
6111:X 30 Apr 2022 00:08:33.791 * monotonic clock: POSIX clock_gettime
6111:X 30 Apr 2022 00:08:33.794 * Running mode=sentinel, port=26379.
6111:X 30 Apr 2022 00:08:33.794 # Sentinel ID is 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
6111:X 30 Apr 2022 00:08:33.794 # +monitor master mymaster 192.168.250.18 6379 quorum 2

####################################################################################
[root@Redis-IP38 ]#redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.250.18:6379,slaves=2,sentinels=3
127.0.0.1:26379>

5. 模拟主Redis 发生故障

停止Redis Master 节点测试故障转移，查看两个从redis的选举过程，并最总确定一个新的主redis。

5.1 主redis 故障转移及监测切换过程

#### 模拟18故障转移 IP38 的 replica-priority 80优先级高些，被提升为主redis
## 停止Redis Master 节点 redis-server
[root@Redis-IP18 ]#killall redis-server
[root@Redis-IP18 ]#

# 观测主日志的信息变化过程，发现IP38会被选举成为新的主redis
[root@Redis-IP18 ]#tail -f /apps/redis/log/sentinel.log
85396:X 30 Apr 2022 00:16:19.497 # +sdown master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:19.553 # +odown master mymaster 192.168.250.18 6379 #quorum 2/2
85396:X 30 Apr 2022 00:16:19.553 # +new-epoch 1
85396:X 30 Apr 2022 00:16:19.553 # +try-failover master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:19.614 # +vote-for-leader 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 1
85396:X 30 Apr 2022 00:16:19.615 # 8a3b79f833d85c73f046cdc5e31dfa266aa4f838 voted for 8a3b79f833d85c73f046cdc5e31dfa266aa4f838 1
85396:X 30 Apr 2022 00:16:19.619 # 8a3b79f833d85c73f046cdc5e31dfa266aa4f828 voted for 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 1
85396:X 30 Apr 2022 00:16:19.691 # +elected-leader master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:19.691 # +failover-state-select-slave master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:19.754 # +selected-slave slave 192.168.250.38:6379 192.168.250.38 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:19.754 * +failover-state-send-slaveof-noone slave 192.168.250.38:6379 192.168.250.38 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:19.816 * +failover-state-wait-promotion slave 192.168.250.38:6379 192.168.250.38 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:20.383 # +promoted-slave slave 192.168.250.38:6379 192.168.250.38 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:20.383 # +failover-state-reconf-slaves master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:20.440 * +slave-reconf-sent slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:20.717 # -odown master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:21.483 * +slave-reconf-inprog slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:21.483 * +slave-reconf-done slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:21.559 # +failover-end master mymaster 192.168.250.18 6379
85396:X 30 Apr 2022 00:16:21.559 # +switch-master mymaster 192.168.250.18 6379 192.168.250.38 6379
85396:X 30 Apr 2022 00:16:21.560 * +slave slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.38 6379
85396:X 30 Apr 2022 00:16:21.560 * +slave slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379
85396:X 30 Apr 2022 00:16:51.603 # +sdown slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379

####################################################################################
####故障转移后的redis配置文件会被自动修改

[root@Redis-IP28 ]#grep ^replicaof /apps/redis/etc/redis.conf
replicaof 192.168.250.38 6379
[root@Redis-IP28 ]#

####################################################################################
#### 哨兵配置文件的sentinel monitor IP 同样也会被修改

[root@Redis-IP28 ]#grep "^[a-Z]" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.38 6379 2 #自动修改此行
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.250.18 6379
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-sentinel mymaster 192.168.250.38 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
sentinel known-sentinel mymaster 192.168.250.18 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
[root@Redis-IP28 ]#

####################################################################################
## 进入redis 查看 info replication
[root@Redis-IP28 ]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.250.38
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:3345393
slave_repl_offset:3345393
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:0137cb1fa513d18c18a764ac95cbbc2916585710
master_replid2:f1dac3206a25ad187f119c32a58f477bd0700ab1
master_repl_offset:3345393
second_repl_offset:3243312
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2296818
repl_backlog_histlen:1048576
127.0.0.1:6379>

5.2 从redis-ip38 被选举新主过程监测

# 实时日志记录的变化过程
[root@Redis-IP38 ]#tail -f /apps/redis/log/sentinel.log
5087:X 29 Apr 2022 22:21:51.125 * +sentinel sentinel 8a3b79f833d85c73f046cdc5e31dfa266aa4f828 192.168.250.28 26379 @ mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:08:33.789 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
6111:X 30 Apr 2022 00:08:33.789 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6111:X 30 Apr 2022 00:08:33.789 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=6111, just started
6111:X 30 Apr 2022 00:08:33.789 # Configuration loaded
6111:X 30 Apr 2022 00:08:33.791 * Increased maximum number of open files to 10032 (it was originally set to 1024).
6111:X 30 Apr 2022 00:08:33.791 * monotonic clock: POSIX clock_gettime
6111:X 30 Apr 2022 00:08:33.794 * Running mode=sentinel, port=26379.
6111:X 30 Apr 2022 00:08:33.794 # Sentinel ID is 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
6111:X 30 Apr 2022 00:08:33.794 # +monitor master mymaster 192.168.250.18 6379 quorum 2
6111:X 30 Apr 2022 00:16:19.514 # +sdown master mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:16:19.567 # +odown master mymaster 192.168.250.18 6379 #quorum 3/2
6111:X 30 Apr 2022 00:16:19.567 # +new-epoch 1
6111:X 30 Apr 2022 00:16:19.567 # +try-failover master mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:16:19.614 # +vote-for-leader 8a3b79f833d85c73f046cdc5e31dfa266aa4f838 1
6111:X 30 Apr 2022 00:16:19.614 # 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 voted for 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 1
6111:X 30 Apr 2022 00:16:19.619 # 8a3b79f833d85c73f046cdc5e31dfa266aa4f828 voted for 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 1
6111:X 30 Apr 2022 00:16:20.440 # +config-update-from sentinel 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 192.168.250.18 26379 @ mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:16:20.440 # +switch-master mymaster 192.168.250.18 6379 192.168.250.38 6379
6111:X 30 Apr 2022 00:16:20.440 * +slave slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.38 6379
6111:X 30 Apr 2022 00:16:20.440 * +slave slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379
6111:X 30 Apr 2022 00:16:50.486 # +sdown slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379

####################################################################################
# 新主看不到 replicaof配置了
[root@Redis-IP38 ]#grep ^replicaof /apps/redis/etc/redis.conf

####################################################################################
# 新主redis上的信息
[root@Redis-IP38 ]#grep "^[a-Z]" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.38 6379 2
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f838
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.250.18 6379
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-sentinel mymaster 192.168.250.18 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
sentinel known-sentinel mymaster 192.168.250.28 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
[root@Redis-IP38 ]#

####################################################################################
[root@Redis-IP38 ]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.250.28,port=6379,state=online,offset=3337043,lag=1
master_failover_state:no-failover
master_replid:0137cb1fa513d18c18a764ac95cbbc2916585710
master_replid2:f1dac3206a25ad187f119c32a58f477bd0700ab1
master_repl_offset:3337329
second_repl_offset:3243312
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2288754
repl_backlog_histlen:1048576
127.0.0.1:6379>

5.3 恢复故障的redis-ip18监测日志

恢复后redis-ip18 只能作为从redis在哨兵模式下去运行，新选举的主在没发生故障或者人为干预下将一直承担主redis角色。

#### 恢复故障的 IP18 机器 将作为 从redis 加入到 哨兵模式中

[root@Redis-IP18 ]#systemctl start redis redis-sentinel.service
[root@Redis-IP18 ]#systemctl status redis redis-sentinel.service
● redis.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2022-04-30 00:27:52 CST; 11s ago
Process: 87861 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=1/FAILURE)
Main PID: 87972 (redis-server)
Tasks: 6 (limit: 23544)
Memory: 2.4M
CGroup: /system.slice/redis.service
└─87972 /apps/redis/bin/redis-server 0.0.0.0:6379

Apr 30 00:27:52 Redis-IP18 systemd[1]: Started Redis persistent key-value database.

● redis-sentinel.service - Redis Sentinel
Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-04-29 19:52:04 CST; 4h 35min ago
Process: 85392 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 85396 (redis-sentinel)
Tasks: 5 (limit: 23544)
Memory: 1.9M
CGroup: /system.slice/redis-sentinel.service
└─85396 /apps/redis/bin/redis-sentinel 0.0.0.0:26379 [sentinel]

Apr 29 19:52:04 Redis-IP18 systemd[1]: Started Redis Sentinel.
[root@Redis-IP18 ]#ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:*
LISTEN 0 511 0.0.0.0:26379 0.0.0.0:*

####################################################################################
[root@Redis-IP18 ]#egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.250.38 6379 2 #这样自定添加了新主redis
sentinel auth-pass mymaster 123456
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 8a3b79f833d85c73f046cdc5e31dfa266aa4f819
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.250.28 6379
sentinel known-replica mymaster 192.168.250.18 6379
sentinel known-sentinel mymaster 192.168.250.28 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f828
sentinel known-sentinel mymaster 192.168.250.38 26379 8a3b79f833d85c73f046cdc5e31dfa266aa4f838

####################################################################################
# 自动添加复制 replicaof 配置行
[root@Redis-IP18 ]#grep ^replicaof /apps/redis/etc/redis.conf
replicaof 192.168.250.38 6379

####################################################################################
# 进入redis后查看 info replication 信息，角色被确定为 role:slave
[root@Redis-IP18 ]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.250.38
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:3419980
slave_repl_offset:3419980
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:0137cb1fa513d18c18a764ac95cbbc2916585710
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:3419980
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:3392320
repl_backlog_histlen:27661
127.0.0.1:6379> quit

####################################################################################
# 进入哨兵查看信息
[root@Redis-IP18 ]#redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.250.38:6379,slaves=2,sentinels=3
127.0.0.1:26379>#### 现在的新主redis-IP38上看到的日志
[root@Redis-IP38 ]#tail -f /apps/redis/log/sentinel.log
6111:X 30 Apr 2022 00:16:19.567 # +try-failover master mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:16:19.614 # +vote-for-leader 8a3b79f833d85c73f046cdc5e31dfa266aa4f838 1
6111:X 30 Apr 2022 00:16:19.614 # 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 voted for 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 1
6111:X 30 Apr 2022 00:16:19.619 # 8a3b79f833d85c73f046cdc5e31dfa266aa4f828 voted for 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 1
6111:X 30 Apr 2022 00:16:20.440 # +config-update-from sentinel 8a3b79f833d85c73f046cdc5e31dfa266aa4f819 192.168.250.18 26379 @ mymaster 192.168.250.18 6379
6111:X 30 Apr 2022 00:16:20.440 # +switch-master mymaster 192.168.250.18 6379 192.168.250.38 6379
6111:X 30 Apr 2022 00:16:20.440 * +slave slave 192.168.250.28:6379 192.168.250.28 6379 @ mymaster 192.168.250.38 6379
6111:X 30 Apr 2022 00:16:20.440 * +slave slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379
6111:X 30 Apr 2022 00:16:50.486 # +sdown slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379
6111:X 30 Apr 2022 00:27:52.926 # -sdown slave 192.168.250.18:6379 192.168.250.18 6379 @ mymaster 192.168.250.38 6379

附A：Redis安装脚本

[root@Redis-IP18 ]#cat /data/install_redis.sh
#!/bin/bash
#
#*****************************************************************
#Date: 2022-04-26
#FileName： install_redis_for_centos.sh
#Description： The test script
#Copyright (C): 2022 All rights reserved
#*****************************************************************
VERSION=redis-6.2.6
PASSWORD=123456
INSTALL_DIR=/apps/redis

color () {
RES_COL=60
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
SETCOLOR_SUCCESS="echo -en \\033[1;32m"
SETCOLOR_FAILURE="echo -en \\033[1;31m"
SETCOLOR_WARNING="echo -en \\033[1;33m"
SETCOLOR_NORMAL="echo -en \E[0m"
echo -n "$1" && $MOVE_TO_COL
echo -n "["
if [ $2 = "success" -o $2 = "0" ] ;then
${SETCOLOR_SUCCESS}
echo -n $" OK "
elif [ $2 = "failure" -o $2 = "1" ] ;then
${SETCOLOR_FAILURE}
echo -n $"FAILED"
else
${SETCOLOR_WARNING}
echo -n $"WARNING"
fi
${SETCOLOR_NORMAL}
echo -n "]"
echo
}


install() {
yum -y install gcc jemalloc-devel || { color "安装软件包失败，请检查网络配置" 1 ; exit; }

wget http://download.redis.io/releases/${VERSION}.tar.gz || { color "Redis 源码下载失败" 1 ; exit; }



tar xf ${VERSION}.tar.gz
cd ${VERSION}
make -j 4 PREFIX=${INSTALL_DIR} install && color "Redis 编译安装完成" 0 || { color "Redis 编译安装失败" 1 ;exit ; }


ln -s ${INSTALL_DIR}/bin/redis-* /usr/bin/
mkdir -p ${INSTALL_DIR}/{etc,log,data,run}
cp redis.conf ${INSTALL_DIR}/etc/

sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e "/# requirepass/a requirepass $PASSWORD" -e "/^dir .*/c dir ${INSTALL_DIR}/data/" -e "/logfile .*/c logfile ${INSTALL_DIR}/log/redis-6379.log" -e "/^pidfile .*/c pidfile ${INSTALL_DIR}/run/redis_6379.pid" ${INSTALL_DIR}/etc/redis.conf


if id redis &> /dev/null ;then
color "Redis 用户已存在" 1
else
useradd -r -s /sbin/nologin redis
color "Redis 用户创建成功" 0
fi

chown -R redis.redis ${INSTALL_DIR}

cat >> /etc/sysctl.conf <<EOF
net.core.somaxconn = 1024
vm.overcommit_memory = 1
EOF
sysctl -p

echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local
/etc/rc.d/rc.local

cat > /usr/lib/systemd/system/redis.service <<EOF
[Unit]
Description=Redis persistent key-value database
After=network.target

[Service]
ExecStart=${INSTALL_DIR}/bin/redis-server ${INSTALL_DIR}/etc/redis.conf --supervised systemd
ExecStop=/bin/kill -s QUIT \$MAINPID
#Type=notify
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

EOF
systemctl daemon-reload
systemctl enable --now redis &> /dev/null && color "Redis 服务启动成功,Redis信息如下:" 0 || { color "Redis 启动失败" 1 ;exit; }
sleep 2
redis-cli -a $PASSWORD INFO Server 2> /dev/null

}

install

[root@Redis-IP18 ]#

附B：主从同步优化配置

Redis在2.8版本之前没有提供增量部分复制的功能，当网络闪断或者slave Redis重启之后会导致主从之间的全量同步，即从2.8版本开始增加了部分复制的功能。

相关配置解读：

repl-diskless-sync no # 是否使用无盘同步RDB文件，默认为no，no为不使用无盘，需要将RDB文件保存到磁盘后再发送给slave，yes为支持无盘，支持无盘就是RDB文件不需要保存至本地磁盘，而且直接通过socket文件发送给slave。
repl-diskless-sync-delay 5 #diskless时复制的服务器等待的延迟时间。
repl-ping-slave-period 10 #slave端向server端发送ping的时间间隔，默认为10秒。
repl-timeout 60 #设置主从ping连接超时时间,超过此值无法连接,master_link_status显示为down,并记录错误日志。
repl-disable-tcp-nodelay no #是否启用TCP_NODELAY，如设置成yes，则redis会合并小的TCP包从而节省带宽， 但会增加同步延迟（40ms），造成master与slave数据不一致，假如设置成no，则redismaster会立即发送同步数据，没有延迟，yes关注网络性能，no关注redis服务中的数据一致性。
repl-backlog-size 1mb #master的写入数据缓冲区，用于记录自上一次同步后到下一次同步过程中间的写入命令，计算公式：repl-backlog-size = 允许从节点最大中断时长 * 主实例offset每秒写入量，比如master每秒最大写入64mb，最大允许60秒，那么就要设置为64mb*60秒=3840MB(3.8G),建议此值是设置的足够大。
repl-backlog-ttl 3600 #如果一段时间后没有slave连接到master，则backlog size的内存将会被释放。如果值为0则 表示永远不释放这部份内存。
slave-priority 100 #slave端的优先级设置，值是一个整数，数字越小表示优先级越高。当master故障时将会按照优先级来选择slave端进行恢复，如果值设置为0，则表示该slave永远不会被选择。
min-replicas-to-write 1 #设置一个master的可用slave不能少于多少个，否则master无法执行写。
min-slaves-max-lag 20 #设置至少有上面数量的slave延迟时间都大于多少秒时，master不接收写操作(拒绝写入)。

附C：哨兵sentinel配置文件解读

[root@centos8 ]#vim /etc/redis-sentinel.conf
bind 0.0.0.0 # 监听IP定义
port 26379 # 哨兵工作端口
daemonize yes # 后台运行
pidfile "redis-sentinel.pid"
logfile "sentinel_26379.log"
dir "/tmp" # 工作目录
sentinel monitor mymaster 10.0.0.8 6379 2
#mymaster是集群的名称，此行指定当前mymaster集群中master服务器的地址和端口 #2为法定人数限制(quorum)，即有几个sentinel认为master down了就进行故障转移，一般此值是所有 sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2,是master的ODOWN客观下线的依据
sentinel auth-pass mymaster 123456 # mymaster集群中master的密码，注意此行要在上面行的下面
sentinel down-after-milliseconds mymaster 30000 # (SDOWN)判断mymaster集群中所有节点的主观下线的时间，单位：毫秒，建议3000
sentinel parallel-syncs mymaster 1 # 发生故障转移后，可以同时向新master同步数据的slave的数量，数字越小总同步时间越长，但可以减轻新master的负载压力
sentinel failover-timeout mymaster 180000 # 所有slaves指向新的master所需的超时时间，单位：毫秒
sentinel deny-scripts-reconfig yes # 禁止修改脚本
logfile /var/log/redis/sentinel.log