LVS:Linux Virtual Server,负载调度器,Linux内核集成,是章文嵩(花名 正明)博士主导的开源负载均衡项目。该项目在Linux内核中实现了基于IP的数据请求负载均衡调度方案,用户从外部发起访问,Web请求会发送给LVS调度器,调度器根据自己预设的算法决定将该请求发送给后端的某台Web服务器,服务器处理好请求将结果反馈给用户。LVS有多种工作模式:lvs-nat lvs-dr lvs-tun。
本文在多网段内实现lvs-dr模式,可以配合网络设备的调测,将LVS规划得层次更清晰。
本次实验为了验证测试出效果采用了rr调度算法,实际生产中根据实际情况优选算法。
1. 架构和主机
整体思路:client发起请求,第一阶段到达路由器eth1;第二阶段从路由器eth0到达LVS;第三阶段LVS将请求按照设定的算法调度到后端的RS上;第四阶段RS1直接将回应的数据包通过lo-VIP发送给路由器eth0;第五阶段返回的回应数据包从路由器eth1传输到client,完成整个的数据传输过程。
这个过程中是通过修改各RS内核参数,来限制arp响应和通告,避免VIP地址冲突问题。
1 2台RS服务器 :
主机名:RS1-IP18
CentOS 8.4
eth0:RIP:192.168.250.18/24 GW:192.168.250.68
lo:VIP:10.0.0.111/32
httpd web服务 页面内容 RS1-IP18 IP:192.168.250.18
主机名:RS2-IP28
CentOS 8.4
eth0:RIP:192.168.250.28/24 GW:192.168.250.68
lo:VIP:10.0.0.111/32
httpd web服务 页面内容 RS2-IP28 IP:192.168.250.28
2 1台LVS服务器 :
主机名: LVS-IP08
CentOS 8.4
lo:VIP:10.0.0.111/32
eth0:DIP:192.168.250.8/24 GW:192.168.250.68
ipvsadm
3 1台充当路由器的主机
主机名: Router-IP68
eth0 IP:192.168.250.68/24 eth0:1 IP:10.0.0.68/24
eth1 IP:172.16.0.68/24
4 1台client主机 :
主机名: Client-IP48
CentOS 8.4
eth0 IP:172.16.0.48/24 GW:172.16.0.68
2. 基础环境及网络配置
任务及过程:按照RS两台主机、client终端、充当路由器角色的主机、LVS主机的顺序完成环境及网络配置。
2.1 两台RS主机
2.1.1 第一台RS1 IP192.168.250.18 配置
#### 第一台RS1 IP192.168.250.18 配置# 验证防火墙、Selinux关闭;修改主机名、同步时间等操作系统优化
[root@CentOS84 ]#hostnamectl set-hostname RS1-IP18
[root@CentOS84 ]#exit
[root@RS1-IP18 ]#systemctl enable --now chronyd.service
# 安装Apache httpd,并定义和修改主页,让后面测试更直观;可以用下面的命令将主机名和IP地址同时写在 index.html 文件中,分成了两行
[root@RS1-IP18 ]#yum -y install httpd;hostname > /var/www/html/index.html;hostname -I >> /var/www/html/index.html;systemctl enable --now httpd
# VCSA下创建得虚拟机会自动生成一个虚拟桥接网卡 virbr0,用下面命令临时删除掉
[root@RS1-IP18 ]#nmcli con del virbr0
# 修改网卡得配置,符合网络规划
[root@RS1-IP18 ]#vim /etc/sysconfig/network-scripts/ifcfg-Profile_1
TYPE=Ethernet
DEVICE=eth0
NAME="eth0"
IPADDR=192.168.250.18
PREFIX=24
GATEWAY=192.168.250.68
DEFROUTE=yes
ONBOOT=yes
# 让网卡配置生效
[root@RS1-IP18 ]#nmcli con reload
[root@RS1-IP18 ]#nmcli con up eth0
[root@RS1-IP18 ]#ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e8:6b brd ff:ff:ff:ff:ff:ff
inet 192.168.250.18/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@RS1-IP18 ]#route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.250.68 0.0.0.0 UG 100 0 0 eth0
192.168.250.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@RS1-IP18 ]#
[root@RS1-IP18 ]#curl 192.168.250.18
RS1-IP18
192.168.250.18
2.1.1 第二台RS2 IP192.168.250.28 配置
#### 第二台RS2 IP192.168.250.28 配置# 验证防火墙、Selinux关闭;修改主机名、同步时间等操作系统优化
[root@CentOS84 ]#hostnamectl set-hostname RS2-IP28
[root@CentOS84 ]#exit
[root@RS2-IP28 ]#systemctl enable --now chronyd.service
# 安装Apache httpd,并定义和修改主页
[root@RS2-IP28 ]#yum -y install httpd;hostname > /var/www/html/index.html;hostname -I >> /var/www/html/index.html;systemctl enable --now httpd
# VCSA下创建得虚拟机会自动生成一个虚拟桥接网卡 virbr0,用下面命令临时删除掉
[root@RS1-IP28 ]#nmcli con del virbr0
# 按照规划修改网卡配置
[root@RS2-IP28 ]#vim /etc/sysconfig/network-scripts/ifcfg-Profile_1
[root@RS2-IP28 ]#cat /etc/sysconfig/network-scripts/ifcfg-Profile_1
TYPE=Ethernet
DEVICE=eth0
NAME="eth0"
IPADDR=192.168.250.28
PREFIX=24
GATEWAY=192.168.250.68
DEFROUTE=yes
ONBOOT=yes
# 使得网卡配置生效
[root@RS2-IP28 ]#nmcli con reload
[root@RS2-IP28 ]#nmcli con up eth0
# 验证网络信息
[root@RS2-IP28 ]#route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.250.68 0.0.0.0 UG 100 0 0 eth0
192.168.250.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@RS2-IP28 ]#
[root@RS2-IP28 ]#ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e2:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.250.28/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@RS2-IP28 ]#curl 192.168.250.28
RS2-IP28 IP:192.168.250.28
[root@RS2-IP28 ]#
[root@RS2-IP28 ]#curl 192.168.250.28
RS2-IP28
192.168.250.28
2.2 测试用client 主机
# 验证防火墙、Selinux关闭;修改主机名、同步时间等操作系统优化[root@CentOS84 ]#hostnamectl set-hostname Client-IP48
[root@CentOS84 ]#exit
[root@Client-IP48 ]#systemctl enable --now chronyd.service
# 修改网卡配置
[root@Client-IP48 ]#vim /etc/sysconfig/network-scripts/ifcfg-Profile_1
TYPE=Ethernet
DEVICE=eth0
NAME="eth0"
IPADDR=172.16.0.48
PREFIX=24
GATEWAY=172.16.0.68
DEFROUTE=yes
ONBOOT=yes
# 使得网卡配置生效
[root@Client-IP48 ]#nmcli connection reload
[root@Client-IP48 ]#nmcli connection up eth0
# 查看网卡地址和路由信息
[root@Client-IP48 ]#ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:48:a4 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.48/24 brd 172.16.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@Client-IP48 ]#ip route
default via 172.16.0.68 dev eth0 proto static metric 100
172.16.0.0/24 dev eth0 proto kernel scope link src 172.16.0.48 metric 100
#### 路由角色的主机配置好后测试跨路由通信,确保在部署IPVS配置前网络是互通的
# 测试到路由器三个IP地址的通信,确保能PING通
[root@Client-IP48 ]#ping 172.16.0.68
PING 172.16.0.68 (172.16.0.68) 56(84) bytes of data.
64 bytes from 172.16.0.68: icmp_seq=1 ttl=64 time=0.425 ms
64 bytes from 172.16.0.68: icmp_seq=2 ttl=64 time=0.442 ms
64 bytes from 172.16.0.68: icmp_seq=3 ttl=64 time=0.447 ms
^C
--- 172.16.0.68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2079ms
rtt min/avg/max/mdev = 0.425/0.438/0.447/0.009 ms
[root@Client-IP48 ]#
[root@Client-IP48 ]#ping 10.0.0.68
PING 10.0.0.68 (10.0.0.68) 56(84) bytes of data.
64 bytes from 10.0.0.68: icmp_seq=1 ttl=64 time=0.498 ms
64 bytes from 10.0.0.68: icmp_seq=2 ttl=64 time=0.392 ms
64 bytes from 10.0.0.68: icmp_seq=3 ttl=64 time=0.402 ms
^C
--- 10.0.0.68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2026ms
rtt min/avg/max/mdev = 0.392/0.430/0.498/0.053 ms
[root@Client-IP48 ]#
[root@Client-IP48 ]#ping 192.168.250.68
PING 192.168.250.68 (192.168.250.68) 56(84) bytes of data.
64 bytes from 192.168.250.68: icmp_seq=1 ttl=64 time=0.850 ms
64 bytes from 192.168.250.68: icmp_seq=2 ttl=64 time=0.467 ms
64 bytes from 192.168.250.68: icmp_seq=3 ttl=64 time=0.463 ms
^C
--- 192.168.250.68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2061ms
rtt min/avg/max/mdev = 0.463/0.593/0.850/0.182 ms
[root@Client-IP48 ]#
# ping 两台RS地址
[root@Client-IP48 ]#ping 192.168.250.18
PING 192.168.250.18 (192.168.250.18) 56(84) bytes of data.
64 bytes from 192.168.250.18: icmp_seq=1 ttl=63 time=0.662 ms
64 bytes from 192.168.250.18: icmp_seq=2 ttl=63 time=0.590 ms
64 bytes from 192.168.250.18: icmp_seq=3 ttl=63 time=0.452 ms
^C
--- 192.168.250.18 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2034ms
rtt min/avg/max/mdev = 0.452/0.568/0.662/0.087 ms
[root@Client-IP48 ]#ping 192.168.250.28
PING 192.168.250.28 (192.168.250.28) 56(84) bytes of data.
64 bytes from 192.168.250.28: icmp_seq=1 ttl=63 time=0.576 ms
64 bytes from 192.168.250.28: icmp_seq=2 ttl=63 time=0.541 ms
64 bytes from 192.168.250.28: icmp_seq=3 ttl=63 time=0.785 ms
^C
--- 192.168.250.28 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2106ms
rtt min/avg/max/mdev = 0.541/0.634/0.785/0.107 ms
2.3 路由角色的主机
# 验证防火墙、Selinux关闭;修改主机名、同步时间等操作系统优化[root@CentOS84 ]#hostnamectl set-hostname Router-IP68
[root@CentOS84 ]#exit
logout
[root@Router-IP68 ]#systemctl enable --now chronyd.service
# 优化完成CentOS配置后确认 ip_forward 已经开启,路由功能才能正常
[root@LVS-IP08 ]#cat /etc/sysctl.conf | grep ip_forward
net.ipv4.ip_forward = 1
# 查看主机的网卡信息
[root@Router-IP68 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:ba:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.250.68/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
# 默认只有一块网卡,需通过VCSA的虚拟管理中心给此台主机增加一块网卡,加好后会看到一块未配置地址的eth1网卡
[root@Router-IP68 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:ba:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.250.68/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:06:9f brd ff:ff:ff:ff:ff:ff
[root@Router-IP68 ]#
[root@Router-IP68 ]#nmcli connection
NAME UUID TYPE DEVICE
eth0 73df0eff-a623-acec-5c7e-627bb30f85d2 ethernet eth0
# 配置新增加的 eth1 网卡,并按照规划修改好 eth0 网卡的配置
[root@Router-IP68 ]#vim /etc/sysconfig/network-scripts/ifcfg-Profile_1
[root@Router-IP68 ]#cat /etc/sysconfig/network-scripts/ifcfg-Profile_1
TYPE=Ethernet
DEVICE=eth0
NAME="eth0"
IPADDR=192.168.250.68
PREFIX=24
DEFROUTE=yes
ONBOOT=yes
[root@Router-IP68 ]#
[root@Router-IP68 ]#cp /etc/sysconfig/network-scripts/ifcfg-Profile_1 /etc/sysconfig/network-scripts/ifcfg-Profile_2
[root@Router-IP68 ]#vim /etc/sysconfig/network-scripts/ifcfg-Profile_2
[root@Router-IP68 ]#cat /etc/sysconfig/network-scripts/ifcfg-Profile_2
TYPE=Ethernet
DEVICE=eth1
NAME="eth1"
IPADDR=172.16.0.68
PREFIX=24
DEFROUTE=yes
ONBOOT=yes
[root@Router-IP68 ]#
# 使得网卡配置生效
[root@Router-IP68 ]#nmcli con reload eth1
[root@Router-IP68 ]#nmcli con up eth1
[root@Router-IP68 ]#nmcli con reload eth0
[root@Router-IP68 ]#nmcli con up eth0
# 验证网卡配置
[root@Router-IP68 ]#nmcli con
NAME UUID TYPE DEVICE
eth0 73df0eff-a623-acec-5c7e-627bb30f85d2 ethernet eth0
eth1 1f162eb7-8128-c2ab-afbb-c099cbc4b75f ethernet eth1
[root@Router-IP68 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:ba:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.250.68/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:06:9f brd ff:ff:ff:ff:ff:ff
inet 172.16.0.68/24 brd 172.16.0.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
# 在client 客户端IP172.16.0.48 上测试与RS1和RS2的通信,路由配置好应该都能通信了,确保网络通了再进行下部IPVS的相关配置。 ---- 需要切换到 [root@Client-IP48 ]# 上去测试
####### 完成LVS-DR 多网段的eth0接口上双IP地址配置,这个也是区别单网段LVS-DR的配置区别之一
[root@Router-IP68 ]#nmcli con
NAME UUID TYPE DEVICE
eth0 73df0eff-a623-acec-5c7e-627bb30f85d2 ethernet eth0
eth1 1f162eb7-8128-c2ab-afbb-c099cbc4b75f ethernet eth1
virbr0 1c44ab23-b537-4dc1-8bed-52a0178203dd bridge virbr0
# 删除虚拟机创建过程中的桥接网卡 virbr0
[root@Router-IP68 ]#nmcli connection delete virbr0
Connection 'virbr0' (1c44ab23-b537-4dc1-8bed-52a0178203dd) successfully deleted.
[root@Router-IP68 ]#nmcli con
NAME UUID TYPE DEVICE
eth0 73df0eff-a623-acec-5c7e-627bb30f85d2 ethernet eth0
eth1 1f162eb7-8128-c2ab-afbb-c099cbc4b75f ethernet eth1
# eth0 网卡接口上添加IP地址
[root@Router-IP68 ]#ip address add 10.0.0.68/24 dev eth0
[root@Router-IP68 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:ba:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.250.68/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 10.0.0.68/24 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:06:9f brd ff:ff:ff:ff:ff:ff
inet 172.16.0.68/24 brd 172.16.0.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
[root@Router-IP68 ]#
# 同时在本机上也做如下测试,确保正常通信。
[root@Router-IP68 ]#ping 172.16.0.48
PING 172.16.0.48 (172.16.0.48) 56(84) bytes of data.
64 bytes from 172.16.0.48: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 172.16.0.48: icmp_seq=2 ttl=64 time=0.371 ms
64 bytes from 172.16.0.48: icmp_seq=3 ttl=64 time=0.368 ms
^C
--- 172.16.0.48 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2030ms
rtt min/avg/max/mdev = 0.368/0.590/1.032/0.312 ms
[root@Router-IP68 ]#ping 192.168.250.8
PING 192.168.250.8 (192.168.250.8) 56(84) bytes of data.
64 bytes from 192.168.250.8: icmp_seq=1 ttl=64 time=0.946 ms
64 bytes from 192.168.250.8: icmp_seq=2 ttl=64 time=0.294 ms
[root@Router-IP68 ]#ping 192.168.250.18
PING 192.168.250.18 (192.168.250.18) 56(84) bytes of data.
64 bytes from 192.168.250.18: icmp_seq=1 ttl=64 time=0.786 ms
64 bytes from 192.168.250.18: icmp_seq=2 ttl=64 time=0.291 ms
64 bytes from 192.168.250.18: icmp_seq=3 ttl=64 time=0.247 ms
64 bytes from 192.168.250.18: icmp_seq=4 ttl=64 time=0.262 ms
[root@Router-IP68 ]#ping 192.168.250.28
PING 192.168.250.28 (192.168.250.28) 56(84) bytes of data.
64 bytes from 192.168.250.28: icmp_seq=1 ttl=64 time=1.04 ms
64 bytes from 192.168.250.28: icmp_seq=2 ttl=64 time=0.288 ms
64 bytes from 192.168.250.28: icmp_seq=3 ttl=64 time=0.353 ms
64 bytes from 192.168.250.28: icmp_seq=4 ttl=64 time=0.335 ms
[root@Router-IP68 ]#ping 10.0.0.68
PING 10.0.0.68 (10.0.0.68) 56(84) bytes of data.
64 bytes from 10.0.0.68: icmp_seq=1 ttl=64 time=0.106 ms
64 bytes from 10.0.0.68: icmp_seq=2 ttl=64 time=0.087 ms
64 bytes from 10.0.0.68: icmp_seq=3 ttl=64 time=0.089 ms
[root@Router-IP68 ]#
# 至此路由角色的主机全部配置完成
2.4 LVS 主机
# 验证防火墙、Selinux关闭;修改主机名、同步时间等操作系统优化[root@CentOS84 ]#hostnamectl set-hostname LVS-IP08
[root@CentOS84 ]#exit
logout
[root@LVS-IP08 ]#systemctl enable --now chronyd.service
# 删除虚拟桥接接口
[root@LVS-IP08 ]#nmcli con delete virbr0
Connection 'virbr0' (3ac22389-a327-46d2-9e6e-a71f8fcb0d36) successfully deleted.
# 先安装好LVS的管理工具包 ipvsadm 后面需要修改网卡信息,不能连接外网了,所以要先下载
[root@LVS-IP08 ]#yum -y install ipvsadm
[root@LVS-IP08 ]#
[root@LVS-IP08 ]#vim /etc/sysconfig/network-scripts/ifcfg-Profile_1
TYPE=Ethernet
DEVICE=eth0
NAME="eth0"
IPADDR=192.168.250.8
PREFIX=24
GATEWAY=192.168.250.68
DEFROUTE=yes
ONBOOT=yes
# 使得网卡配置生效
[root@LVS-IP08 ]#nmcli con reload
[root@LVS-IP08 ]#nmcli con up eth0
# 查看路由信息及网卡信息
[root@LVS-IP08 ]#ip route
default via 192.168.250.68 dev eth0 proto static metric 100
192.168.250.0/24 dev eth0 proto kernel scope link src 192.168.250.8 metric 100
[root@LVS-IP08 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:9e:ee brd ff:ff:ff:ff:ff:ff
inet 192.168.250.8/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
3. IPVS相关的配置
任务及过程:需要完成后端RS主机和LVS服务器与LVS相关的调试。先完成后端RS的全局和环回接口上限制arp响应和通告的级别,并在lo接口上设好VIP地址;再在LVS上去完成在lo接口上设好VIP地址,并配置好ipvs的集群和规则。
3.1 后端RS的IPVS配置
3.1.1 RS1 与IPVS相关的配置
#### RS1 与IPVS相关的配置# 全局和环回接口上限制arp响应和通告的级别
[root@RS1-IP18 ]#echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@RS1-IP18 ]#echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@RS1-IP18 ]#echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@RS1-IP18 ]#echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@RS1-IP18 ]#
# 查看网卡信息
[root@RS1-IP18 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e8:6b brd ff:ff:ff:ff:ff:ff
inet 192.168.250.18/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
# lo环回接口上添加增加VIP
[root@RS1-IP18 ]#ifconfig lo:1 10.0.0.111/32
# 查看验证添加后的网卡信息
[root@RS1-IP18 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.111/0 scope global lo:1
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e8:6b brd ff:ff:ff:ff:ff:ff
inet 192.168.250.18/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@RS1-IP18 ]#
3.1.2 RS2 与IPVS相关的配置
#### RS2 与IPVS相关的配置# 全局和环回接口上限制arp响应和通告的级别
[root@RS2-IP28 ]#
[root@RS2-IP28 ]#echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@RS2-IP28 ]#echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@RS2-IP28 ]#echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@RS2-IP28 ]#echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
# 查看网卡信息
[root@RS2-IP28 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e2:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.250.28/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
# lo环回接口上添加增加VIP 实际生产中需要永久保存配置,需要写入到配置文件中去
[root@RS2-IP28 ]#ifconfig lo:1 10.0.0.111/32
# 查看验证添加后的网卡信息
[root@RS2-IP28 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.111/0 scope global lo:1
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:e2:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.250.28/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@RS2-IP28 ]#
3.2 LVS主机的IPVS相关配置
# lo环回接口上添加增加VIP 实际生产中需要永久保存配置,需要写入到配置文件中去[root@LVS-IP08 ]#ifconfig lo:1 10.0.0.111/32
# 查看网卡信息
[root@LVS-IP08 ]#route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.250.68 0.0.0.0 UG 100 0 0 eth0
192.168.250.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@LVS-IP08 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.111/0 scope global lo:1
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a3:9e:ee brd ff:ff:ff:ff:ff:ff
inet 192.168.250.8/24 brd 192.168.250.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
#### 配置LVS集群和LVS 规则
#先查看默认信息
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
# 配置集群
[root@LVS-IP08 ]#ipvsadm -A -t 10.0.0.111:80 -s rr
# 验证配置
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.111:80 rr
[root@LVS-IP08 ]#
# 添加LVS规则,下面两条都是是有实体RS对应的规则
[root@LVS-IP08 ]#ipvsadm -a -t 10.0.0.111:80 -r 192.168.250.18 -g
[root@LVS-IP08 ]#ipvsadm -a -t 10.0.0.111:80 -r 192.168.250.28 -g
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.111:80 rr
-> 192.168.250.18:80 Route 1 0 0
-> 192.168.250.28:80 Route 1 0 0
# 上面的配置完成后在client端看到的实际效果如下,符合设定的逻辑:轮询,无后端RS报错,无后端健康检查
[root@Client-IP48 ]#while :;do curl 192.168.250.111;sleep 1;done
curl: (7) Failed to connect to 192.168.250.111 port 80: No route to host
RS2-IP28 IP:192.168.250.28
RS1-IP18 IP:192.168.250.18
curl: (7) Failed to connect to 192.168.250.111 port 80: No route to host
RS2-IP28 IP:192.168.250.28
RS1-IP18 IP:192.168.250.18
# 保存配置,并以服务方式开启启动LVS
[root@LVS-IP08 ]#ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@LVS-IP08 ]#cat /etc/sysconfig/ipvsadm
-A -t 10.0.0.111:80 -s rr
-a -t 10.0.0.111:80 -r 192.168.250.18:80 -g -w 1
-a -t 10.0.0.111:80 -r 192.168.250.28:80 -g -w 1
[root@LVS-IP08 ]#
[root@LVS-IP08 ]#systemctl enable --now ipvsadm.service
# 切换到Client-IP48端查看并验证输出信息
[root@Client-IP48 ]#while :;do curl 10.0.0.111;sleep 1;done
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
4. 测试访问
基本过程:在Client-IP48启用自动访问命令;在两台RS上观测日志
### 在client终端开启下面的命令,在RS上监看日志输出[root@Client-IP48 ]#while :;do curl 10.0.0.111;sleep 1;done
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
^C
[root@Client-IP48 ]#
# RS1上的日志输出信息
[root@RS1-IP18 ]#tail -f /var/log/httpd/access_log -n0
172.16.0.48 - - [23/Mar/2022:03:33:11 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:13 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:15 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:17 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:19 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:21 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:23 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:25 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:27 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:29 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:31 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:33 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
# RS2上的日志输出信息
[root@RS2-IP28 ]#tail -f /var/log/httpd/access_log -n0
172.16.0.48 - - [23/Mar/2022:03:33:12 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:14 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:16 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:18 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:20 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:22 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:24 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:26 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:28 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:30 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
172.16.0.48 - - [23/Mar/2022:03:33:32 +0800] "GET / HTTP/1.1" 200 25 "-" "curl/7.61.1"
5. 拓展A:LVS实现https(443)转发
思路及过程:前面实现了80端口基于LVS-DR多网段的代理转发,在此基础上再拓展443端口转发
#################################################################################### 第一步:先实现两台RS上的https服务
## 第一台RS1 IP:192.168.250.18 上实现基于apache httpd的https服务
# 因为前面实验时候的网关是指向路由器的,需要修改成能上外网的网关,并正确配置好DNS,确保PING外网通
[root@RS1-IP18 ]#ping www.163.com
PING z163picipv6.v.bsgslb.cn (180.127.236.57) 56(84) bytes of data.
64 bytes from 180.127.236.57 (180.127.236.57): icmp_seq=1 ttl=56 time=8.06 ms
64 bytes from 180.127.236.57 (180.127.236.57): icmp_seq=2 ttl=56 time=8.16 ms
[root@RS1-IP18 ]#yum -y install mod_ssl
[root@RS1-IP18 ]#systemctl restart httpd
[root@RS1-IP18 ]#curl -k https://192.168.250.18
RS1-IP18
192.168.250.18
# 完成上面步骤,说明RS1上的https配置完成,再把网关修改回先前LVS-DR实验规划的IP192.168.250.68
## 第一台RS2 IP:192.168.250.28 上实现基于apache httpd的https服务
# 因为前面实验时候的网关是指向路由器的,需要修改成能上外网的网关,并正确配置好DNS,确保PING外网通
[root@RS2-IP28 ]#ping www.163.com
PING z163picipv6.v.bsgslb.cn (180.127.236.58) 56(84) bytes of data.
64 bytes from 180.127.236.58 (180.127.236.58): icmp_seq=1 ttl=56 time=11.8 ms
64 bytes from 180.127.236.58 (180.127.236.58): icmp_seq=2 ttl=56 time=11.8 ms
[root@RS2-IP28 ]#yum -y install mod_ssl
[root@RS2-IP28 ]#systemctl restart httpd
[root@RS2-IP28 ]#curl -k https://192.168.250.28
RS2-IP28
192.168.250.28
[root@RS2-IP28 ]#ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 *:443 *:*
LISTEN 0 511 *:80 *:*
# 完成上面步骤,说明RS1上的https配置完成,再把网关修改回先前LVS-DR实验规划的IP192.168.250.68
################################################################################
#### 在LVS IP192.168.0.8 服务器上添加443的集群和转发规则
[root@LVS-IP08 ]#ipvsadm -A -t 10.0.0.111:443 -s rr
[root@LVS-IP08 ]#ipvsadm -a -t 10.0.0.111:443 -r 192.168.250.18 -g
[root@LVS-IP08 ]#ipvsadm -a -t 10.0.0.111:443 -r 192.168.250.28 -g
[root@LVS-IP08 ]#
## 查看LVS规则和集群配置
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.111:80 rr
-> 192.168.250.18:80 Route 1 0 0
-> 192.168.250.28:80 Route 1 0 0
TCP 10.0.0.111:443 rr
-> 192.168.250.18:443 Route 1 0 0
-> 192.168.250.28:443 Route 1 0 0
[root@LVS-IP08 ]#
################################################################################
## 在Client-IP48 上测试443的访问
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS1-IP18
192.168.250.18
RS2-IP28
192.168.250.28
[root@Client-IP48 ]#
6. 拓展B:LVS利用防火墙标记实现http(80)和https(443)集中统一转发
6.1 相关知识解释
FWM:FireWall Mark
MARK target 可用于给特定的报文打标记
--set-mark value 其中:value 可为0xffff格式,表示十六进制数字
借助于防火墙标记来分类报文,而后基于标记定义集群服务;可将多个不同的应用使用同一个集群服务进行调度
实现方法:
首先在Director主机打标记:
再在Director主机基于标记定义集群服务。
6.2 实现过程
## LVS 上利用防火墙标签技术实现https和http在LVS统一转发################################################################################
################################################################################
# 清空原先的ipvsadm 配置
[root@LVS-IP08 ]#ipvsadm -C
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
# iptables打标签
[root@LVS-IP08 ]#iptables -t mangle -A PREROUTING -d 10.0.0.111 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 111
# 验证标签
[root@LVS-IP08[root@LVS-IP08 ]#iptables -vnL -t mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp -- * * 0.0.0.0/0 10.0.0.111 multiport dports 80,443 MARK set 0x6f
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
15108 1651K LIBVIRT_PRT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_PRT (1 references)
pkts bytes target prot opt in out source destination
0 0 CHECKSUM udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill
################################################################################
# 定义,查看LVS集群规则
[root@LVS-IP08 ]#ipvsadm -A -f 111 -s rr
[root@LVS-IP08 ]#ipvsadm -a -f 111 -r 192.168.250.18 -g
[root@LVS-IP08 ]#ipvsadm -a -f 111 -r 192.168.250.28 -g
[root@LVS-IP08 ]#
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 111 rr
-> 192.168.250.18:0 Route 1 0 0
-> 192.168.250.28:0 Route 1 0 0
[root@LVS-IP08 ]#
################################################################################
## 再启用持久连接 -p 默认360秒
# 修改LVS集群规则,并查看
[root@LVS-IP08 ]#ipvsadm -E -f 111 -s rr -p
[root@LVS-IP08 ]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 111 rr persistent 360
-> 192.168.250.18:0 Route 1 0 0
-> 192.168.250.28:0 Route 1 0 0
[root@LVS-IP08 ]#
################################################################################
# 永久保存规则
[root@LVS-IP08 ]#ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@LVS-IP08 ]#systemctl enable --now ipvsadm.service
[root@LVS-IP08 ]#
################################################################################
################################################################################
## 在Client-IP48 上测试443的访问
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
[root@Client-IP48 ]#curl -k https://10.0.0.111;curl http://10.0.0.111
RS2-IP28
192.168.250.28
RS1-IP18
192.168.250.18
[root@Client-IP48 ]#