Redis集群(twemproxy+haproxy)架构

系统版本:

[root@node1 ~]# cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)

Redis下载：

https://download.redis.io/releases/ redis-4.0.14.tar.gz

twemproxy官网：

https://github.com/twitter/twemproxy

twemproxy软件包下(可能需要外网):

wget https://github.com/twitter/twemproxy/releases/download/0.5.0/twemproxy-0.5.0.tar.gz

解压安装twemproxy:

tar zxf twemproxy-0.5.0.tar.gz

编译报错，没有安装gcc:

./configure 报错: configure: error: in `/root/twemproxy/twemproxy-0.5.0': configure: error: no acceptable C compiler found in $PATH

报错处理:

yum install gcc make -y

再次编译安装:

mkdir /usr/local/twemproxy ./configure --prefix=/usr/local/twemproxy/

编译完成，直接安装:

# make -j 4 && make install

Tar包下复制nutcracker.yml文件到安装目录下:

[root@master conf]# pwd /root/twemproxy/twemproxy-0.5.0/conf cp nutcracker.yml /usr/local/twemproxy/

安装redis:

安装步骤忽略,太简单。

redis编译报错:

In file included from adlist.c:34:0: zmalloc.h:50:31: fatal error: jemalloc/jemalloc.h: No such file or directory #include <jemalloc/jemalloc.h>

错误的本质是我们在开始执行make 时遇到了错误gcc未安装,然后我们安装好了gcc 后，我们再执行make ,这时就出现了jemalloc/jemalloc.h: No such file or directory 编译失败，有残留的文件，我们需要清理下，然后重新编译就可以了.

解决办法两种:

第一种:

make MALLOC=libc && make install

第二种:

make distclean make && make install

修改nutcracker.yml配置文件:

[root@master twemproxy]# cat nutcracker.yml redis_test: listen: 0.0.0.0:22121 hash: fnv1a_64 distribution: ketama auto_eject_hosts: true redis: true #redis_auth: mldnjava server_retry_timeout: 2000 server_failure_limit: 1 servers: - 10.40.42.105:6379:1 - 10.40.42.127:6379:1

启动twemproxy服务前，校验文件配置:

# /usr/local/twemproxy/sbin/nutcracker -t /usr/local/twemproxy/nutcracker.yml

查看twemproxy启动命令并启动服务:

[root@master sbin]# /usr/local/twemproxy/sbin/nutcracker -c /usr/local/twemproxy/nutcracker.yml -o /usr/local/twemproxy/twemproxy.log -d

查看nutcracker服务进程:

[root@master sbin]# ps -ef | grep nutc root 114525 1 0 00:03 ? 00:00:00 /usr/local/twemproxy/sbin/nutcracker -c /usr/local/twemproxy/nutcracker.yml -o /usr/local/twemproxy/twemproxy.log -d root 114704 91662 0 00:04 pts/0 00:00:00 grep --color=auto nutc

查看nutcracker服务端口:

[root@master twemproxy]# netstat -antlp | grep 22121 tcp 0 0 0.0.0.0:22121 0.0.0.0:* LISTEN 114525/nutcracker

测试

连接Twemproxy新建一个key,node1 redis客户端直接查询：

Twemproxy新建一个key:

10.40.42.103:22121> set hello world OK

 

Haproxy+ twemproxy架构:

部署2台twemproxy，代理相同的redis服务：

Node2上的twemproxy服务:

[root@node2 twemproxy]# cat nutcracker.yml redis_test: listen: 0.0.0.0:22121 hash: fnv1a_64 distribution: ketama auto_eject_hosts: true redis: true #redis_auth: redhat server_retry_timeout: 2000 server_failure_limit: 1 servers: - 10.40.42.105:6379:1 - 10.40.42.127:6379:1

 

Master上的twemproxy服务:

[root@master twemproxy]# cat nutcracker.yml redis_test: listen: 0.0.0.0:22121 hash: fnv1a_64 distribution: ketama auto_eject_hosts: true redis: true #redis_auth: redhat server_retry_timeout: 2000 server_failure_limit: 1 servers: - 10.40.42.105:6379:1 - 10.40.42.127:6379:1

安装haproxy服务:

[root@node1 ~]# yum install haproxy -y

修改haproxy.cfg配置文件:

[root@node1 ~]# cat /etc/haproxy/haproxy.cfg | grep -v "^#" | grep -v "^[ ]\+#" global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen twemproxy bind 0.0.0.0:5666 option tcplog mode tcp balance roundrobin server app1 10.40.42.103:22121 check inter 10s server app2 10.40.42.127:22121 check inter 10s listen admin_stats bind *:11001 mode http stats refresh 30s stats uri /admin stats realm welcome login\ Haproxy stats auth admin:admin123 stats admin if TRUE # 通过设置此选项，可以在监控页面上手工启用或禁用后端真实服务器

启动haproxy服务，并查看进程:

[root@node1 ~]# systemctl start haproxy [root@node1 ~]# ps -ef | grep hapro root 79089 1 0 05:05 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid haproxy 79090 79089 0 05:05 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds haproxy 79091 79090 0 05:05 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds root 79102 49336 0 05:05 pts/1 00:00:00 grep --color=auto hapro

浏览器访问:

http://10.40.42.105:11001/admin

测试:

通过haproxy代理连接新建一个key:

[root@node1 bin]# ./redis-cli -h 10.40.42.105 -p 5666 10.40.42.105:5666> set mew happy OK

Node2本地redis客户端连接获取key:

[root@node2 bin]# ./redis-cli 127.0.0.1:6379> get mew "happy"