Keepalived+LVS实战案例（二）： 双主架构实现WEB和MariaDB负载均衡及高可用

​概要说明：双主是指两台Keepalived / LVS服务器，互为冗余，在其中一台上（IP:192.168.250.18）以master方式 VIP 192.168.250.100 代理转发后端WEB服务，同时这台以slave方式代理转发mariadb的服务请求；在另外一台上（IP:192.168.250.28）以master方式 VIP 192.168.250.200 代理转发mariadb的服务请求，同时这台以slave方式代理转发WEB；当任何一台发生故障都会将全部转发请求转移到另外一台上，这样充分地利用了运算资源，也提供了高可靠性。​

​ 本文以比较简要的方式进行描述，详细解读可以参考前文《​Keepalived+LVS实战案例（一）： 单主架构实现WEB负载均衡及高可用​》​

1. 拓扑结构及主机环境

# 七台主机
1 2台web服务器 ：
主机名：WebServer-IP17
CentOS 7.9
IP:192.168.250.17

主机名：WebServer-IP27
CentOS 7.9
IP:192.168.250.27

2 2台 MariaDB 数据库服务器 ：
主机名：MariaDB-IP37
CentOS 7.9
IP:192.168.250.37

主机名：MariaDB-IP47
CentOS 7.9
IP:192.168.250.47


3 2台 keepalived 服务器 ：
主机名： KA-IP18
CentOS 8.4
IP:192.168.250.18/24
Keepalived v2.1.5 (07/13,2020)

主机名： KA-IP28
CentOS 8.4
IP:192.168.250.28/24
Keepalived v2.1.5 (07/13,2020)

4 1台client主机 ：
主机名：Client-IP172-8
CentOS 8.4
IP:172.16.0.8/24 NAT成192.168.250.254 访问192.168.250.X网段

2. 配置后端WEB服务器及数据库RS服务

​简要说明：按照架构图，需要IP17 / IP27 两台WEB服务器，安装好httpd及定义好首页内容，并按照LVS-DR对后端RS服务器的配置要求（关闭ARP宣告和接受；绑定VIP地址等）完成配置。同样需要准备两台 IP37 /IP47 两台 MariaDB 数据库服务器。​

2.1 配置WEB服务器

# 基础环境包括CentOS操作系统的优化、防火墙关闭、同步时间等都要做好，我们按照规划的架构图对四台服务器进行分组并重新命名
# 修改服务器名称
[root@centos79 _{]# hostnamectl set-hostname WebServer-IP17
[root@centos79}]# exit
# 修改NTP服务器地址为阿里云的NTP 启用时钟同步服务
[root@webserver-ip17 _{]#timedatectl set-timezone Asia/Shanghai
[root@webserver-ip17}]#sed -i '/^server/cserver ntp.aliyun.com iburst' /etc/chrony.conf
[root@webserver-ip17 _{]#systemctl enable --now chronyd.service
# 安装Apache
[root@webserver-ip17}]#yum -y install httpd
# 定义web主页文件
[root@webserver-ip17 <sub>]#yum -y install httpd;hostname > /var/www/html/indexTmp.html;hostname -I >> /var/www/html/indexTmp.html;cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html;rm -rf /var/www/html/indexTmp.html;systemctl enable --now httpd

# 启动Apache服务，并开机自启
[root@webserver-ip17 </sub>]#systemctl enable --now httpd

# 验证
[root@webserver-ip17 _{]# curl 192.168.250.17
webserver-ip17 192.168.250.17
[root@webserver-ip17}]#

################################################################################
#### 同样的方式完成另外一台 webserver-ip27 192.168.250.27 的Apache的配置和调试

####################################################################################
#### 用脚本修改于LVS相关的配置 lvs_dr_rs.sh 内容,在VS-Code内修改好上传到两个WEB-RS服务器上运行

#!/bin/bash
vip=192.168.250.100
mask='255.255.255.255'
dev=lo:1

case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac

## IP192.168.250.17 上的配置过程
[root@webserver-ip17 <sub>]# rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.
Transferring lvs_dr_rs.sh...
100% 728 bytes 728 bytes/sec 00:00:01 0 Errors

[root@webserver-ip17 </sub>]# bash
[root@webserver-ip17 _{]# bash lvs_dr_rs.sh
Usage: lvs_dr_rs.sh start|stop
[root@webserver-ip17}]# bash lvs_dr_rs.sh start
The RS Server is Ready!
# 查看绑定是否成功
[root@webserver-ip17 <sub>]# ip a


## 同样的方式在 IP192.168.250.27 上的配置
[root@webserver-ip27 </sub>]# bash lvs_dr_rs.sh start
The RS Server is Ready!
# 查看绑定是否成功
[root@webserver-ip27 ~]# ip a

2.2 配置MariaDB数据库服务器

#### 准备好基础的环境、按照拓扑修改主机名、
[root@centos79 _{]# hostnamectl set-hostname MariaDB-IP37
[root@centos79}]# exit
# 安装mariadb，并启动、开机自启、授权
[root@mariadb-ip37 _{]# yum -y install mariadb-server
[root@mariadb-ip37}]# mysql -e 'grant all on *.* to test@"%.%.%.%" identified by "shone8888"'
[root@mariadb-ip37 <sub>]# systemctl enable --now mariadb.service

# 用脚本修改和LVS相关的内容，这个VIP是 192.168.250.200 WEB绑定的是192.168.250.100 这就是我们说的双主
[root@mariadb-ip37 </sub>]#vim lvs_dr_rs.sh
#!/bin/bash
vip=192.168.250.200
mask='255.255.255.255'
dev=lo:1

case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac

[root@mariadb-ip37 _{]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@mariadb-ip37}]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.250.200/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:ef:ca brd ff:ff:ff:ff:ff:ff
inet 192.168.250.37/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea3:efca/64 scope link
valid_lft forever preferred_lft forever

# 在IP192.168.250.47 上重复上面的步骤，完成LVS-DR的lo-IP地址设置，并关闭arp宣告和接收

# 配置完成后再本机上测试访问
[root@CentOS84-IP172-08 ]#mysql -utest -pshone8888 -h192.168.250.37 -e 'select @@hostname'
+--------------------------+
| @@hostname |
+--------------------------+
| mariadb-ip192.168.250.37 |
+--------------------------+
[root@CentOS84-IP172-08 ]#mysql -utest -pshone8888 -h192.168.250.47 -e 'select @@hostname'
+--------------------------+
| @@hostname |
+--------------------------+
| mariadb-ip192.168.250.47 |
+--------------------------+
[root@CentOS84-IP172-08 ]#

2.3 keepalived 基础配置

​简要说明：配置好keepalived 地全局配置、vrrp的配置。 本次我们沿用的是单播方式，如果需要启用多播方式，只需要注释掉单播的配置，开启多播的配置行（在下面的配置文件中也有，只需要去掉 # 注释行头）。​

2.3.1 Keepalived-IP18的配置

[root@Keepalived-IP18 ]#cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@shone.cn
}
notification_email_from admin@shone.cn
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA-IP18
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
#vrrp_mcast_group4 224.0.0.18
}


vrrp_instance VI_IP100 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.100 dev eth0 label eth0:1
}
unicast_src_ip 192.168.250.18
unicast_peer {
192.168.250.28
}
}

vrrp_instance VI_IP200 {
state BACKUP
interface eth0
virtual_router_id 200
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.200 dev eth0 label eth0:2
}
unicast_src_ip 192.168.250.18
unicast_peer {
192.168.250.28
}
}




[root@Keepalived-IP18 ]#systemctl restart keepalived
[root@Keepalived-IP18 ]#

2.3.2 Keepalived-IP28的配置

[root@Keepalived-IP28 ]#cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@shone.cn
}
notification_email_from admin@shone.cn
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA-IP28
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
#vrrp_mcast_group4 224.0.0.18
}


vrrp_instance VI_IP100 {
state BACKUP
interface eth0
virtual_router_id 100
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.100 dev eth0 label eth0:1
}
unicast_src_ip 192.168.250.28
unicast_peer {
192.168.250.18
}
}

vrrp_instance VI_IP200 {
state MASTER
interface eth0
virtual_router_id 200
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.200 dev eth0 label eth0:2
}
unicast_src_ip 192.168.250.28
unicast_peer {
192.168.250.18
}
}



[root@Keepalived-IP28 ]#systemctl restart keepalived

# 配置完成后在Keepalived 上抓包查看工作状态是否正常，并模拟故障交叉关停下18、28上的Keepalived的服务，看是否切换成功，确保没问题再进入下一个环节配置
[root@Keepalived-IP28 ]#tcpdump -i eth0 -nn src host 192.168.250.18 and dst 192.168.250.28
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
05:21:35.782367 IP 192.168.250.18 > 192.168.250.28: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
05:21:36.782470 IP 192.168.250.18 > 192.168.250.28: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
05:21:37.782605 IP 192.168.250.18 > 192.168.250.28: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
05:21:38.782653 IP 192.168.250.18 > 192.168.250.28: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
05:21:39.782792 IP 192.168.250.18 > 192.168.250.28: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
05:21:40.010292 ARP, Reply 192.168.250.18 is-at 00:50:56:a3:e8:6b, length 46
^C
6 packets captured
7 packets received by filter
0 packets dropped by kernel
[root@Keepalived-IP28 ]#tcpdump -i eth0 -nn src host 192.168.250.28 and dst 192.168.250.18
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
05:21:45.625001 IP 192.168.250.28 > 192.168.250.18: VRRPv2, Advertisement, vrid 200, prio 100, authtype simple, intvl 1s, length 20
05:21:46.625105 IP 192.168.250.28 > 192.168.250.18: VRRPv2, Advertisement, vrid 200, prio 100, authtype simple, intvl 1s, length 20
05:21:47.625262 IP 192.168.250.28 > 192.168.250.18: VRRPv2, Advertisement, vrid 200, prio 100, authtype simple, intvl 1s, length 20
05:21:48.625361 IP 192.168.250.28 > 192.168.250.18: VRRPv2, Advertisement, vrid 200, prio 100, authtype simple, intvl 1s, length 20
05:21:49.625467 IP 192.168.250.28 > 192.168.250.18: VRRPv2, Advertisement, vrid 200, prio 100, authtype simple, intvl 1s, length 20
^C
5 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@Keepalived-IP28 ]#

2.4 keepalived LVS相关的配置

​简要说明：完成2.3章节还需要完成与LVS相关的配置，实现和后端的RS服务器的代理转发。​

2.4.1 Keepalived-IP18的配置

[root@Keepalived-IP18 ]#cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@shone.cn
}
notification_email_from admin@shone.cn
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA-IP18
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
#vrrp_mcast_group4 224.0.0.18
}


vrrp_instance VI_IP100 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.100 dev eth0 label eth0:1
}
unicast_src_ip 192.168.250.18
unicast_peer {
192.168.250.28
}
}

vrrp_instance VI_IP200 {
state BACKUP
interface eth0
virtual_router_id 200
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.200 dev eth0 label eth0:2
}
unicast_src_ip 192.168.250.18
unicast_peer {
192.168.250.28
}
}


virtual_server 192.168.250.100 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.250.17 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.250.27 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}

virtual_server 192.168.250.200 3306 {
idelay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.250.37 3306 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 192.168.250.47 3306 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
[root@Keepalived-IP18 ]#systemctl restart keepalived

[root@Keepalived-IP18 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.250.100:80 rr
-> 192.168.250.17:80 Route 1 0 0
-> 192.168.250.27:80 Route 1 0 0
TCP 192.168.250.200:3306 rr
-> 192.168.250.37:3306 Route 1 0 0
-> 192.168.250.47:3306 Route 1 0 0
[root@Keepalived-IP18 ]#
[root@Keepalived-IP18 ]#hostname -I
192.168.250.18 192.168.250.100
[root@Keepalived-IP18 ]#

[root@Keepalived-IP18 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e8:6b brd ff:ff:ff:ff:ff:ff
inet 192.168.250.18/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 192.168.250.100/32 scope global eth0:1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea3:e86b/64 scope link
valid_lft forever preferred_lft forever

2.4.2 Keepalived-IP28的配置

[root@Keepalived-IP28 ]#cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@shone.cn
}
notification_email_from admin@shone.cn
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA-IP28
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
#vrrp_mcast_group4 224.0.0.18
}


vrrp_instance VI_IP100 {
state BACKUP
interface eth0
virtual_router_id 100
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.100 dev eth0 label eth0:1
}
unicast_src_ip 192.168.250.28
unicast_peer {
192.168.250.18
}
}

vrrp_instance VI_IP200 {
state MASTER
interface eth0
virtual_router_id 200
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass shone888
}
virtual_ipaddress {
192.168.250.200 dev eth0 label eth0:2
}
unicast_src_ip 192.168.250.28
unicast_peer {
192.168.250.18
}
}


virtual_server 192.168.250.100 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.250.17 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.250.27 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}


virtual_server 192.168.250.200 3306 {
idelay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.250.37 3306 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 192.168.250.47 3306 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}

[root@Keepalived-IP28 ]#
[root@Keepalived-IP28 ]#systemctl restart keepalived



[root@Keepalived-IP28 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.250.100:80 rr
-> 192.168.250.17:80 Route 1 0 0
-> 192.168.250.27:80 Route 1 0 0
TCP 192.168.250.200:3306 rr
-> 192.168.250.37:3306 Route 1 0 0
-> 192.168.250.47:3306 Route 1 0 0
[root@Keepalived-IP28 ]#hostname -I
192.168.250.28 192.168.250.200
[root@Keepalived-IP28 ]#


[root@Keepalived-IP28 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e2:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.250.28/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 192.168.250.200/32 scope global eth0:2
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea3:e2bf/64 scope link
valid_lft forever preferred_lft forever
[root@Keepalived-IP28 ]#

3. 测试验证

​简要说明：下面仅列出全正常状态下的访问的测试信息。模拟故障各自组合出几个方案自行试验。​

[root@CentOS84-IP172-08 ]#while :;do curl 192.168.250.100;sleep 1;done
webserver-ip27 192.168.250.27
webserver-ip17 192.168.250.17
webserver-ip27 192.168.250.27
webserver-ip17 192.168.250.17
webserver-ip27 192.168.250.27
webserver-ip17 192.168.250.17
webserver-ip27 192.168.250.27
^C
[root@CentOS84-IP172-08 ]#while :;do mysql -utest -pshone8888 -h192.168.250.200 -e 'select @@hostname';sleep 1;done
+--------------------------+
| @@hostname |
+--------------------------+
| mariadb-ip192.168.250.37 |
+--------------------------+
+--------------------------+
| @@hostname |
+--------------------------+
| mariadb-ip192.168.250.47 |
+--------------------------+
+--------------------------+
| @@hostname |
+--------------------------+
| mariadb-ip192.168.250.37 |
+--------------------------+
+--------------------------+
| @@hostname |
+--------------------------+
| mariadb-ip192.168.250.47 |
+--------------------------+
^C
[root@CentOS84-IP172-08 ]#