Linux小型环境下通过网络实现rsyslog日志收集、MySQL保存及LogAnalyzer图形展示

Linux小型环境下通过网络实现rsyslog日志收集、MySQL保存及loganalyzer图形展示

任务内容：收集Linux体系下主机rsyslog类的日志（SyslogClient-IP07 192.168.250.7 和SyslogClient-IP08 192.168.250.8），先以目录文件方式（Rsyslog-server-IP18 192.168.250.18）集中记录，再转录到MySQL数据库（Syslog-MySQL-IP28 192.168.250.28）内集中记录，再用LogAnalyzer（LogAnalyzer-IP38 192.168.250.38）对记录在MySQL内的日志进行图形展示。

1. 架构及主机

五台服务器
1 被收集日志主机A ：
主机名：SyslogClient-IP07
CentOS 7.9
IP: 192.168.250.7
rsyslog 8.24.0

2 被收集日志主机B ：
主机名：SyslogClient-IP08
CentOS 8.4
IP: 192.168.250.8
rsyslog 8.1911.0

3 日志服务器 (目录文件) ：
主机名：Rsyslog-server-IP18
CentOS 8.4
IP: 192.168.250.18/24

4. 日志服务器(MySQ库) ：
主机名：Syslog-MySQL-IP28
CentOS 8.4
IP: 192.168.250.28/24

5.日志图形展示服务器 ：
主机名：LogAnalyzer-IP38
CentOS 8.4
IP: 192.168.250.38/24

# 说明：按照上面的架构图，准备好五台主机，将以此为基础环境完成日志的收集、集中存放、日志展示的实验过程

2. 开启网络日志服务记录

任务内容：启用被收集的主机的网络日志服务记录功能，可以将多个远程主机的日志，发送到集中的日志服务器，集中日志服务器可以是以目录内日志文件格式或者数据库方式，这样方便日志的统一管理和处理。

2.1. 认识rsyslog并开启主机网络日志记录功能

2.1.1. CentOS8主机 192.168.250.8 开启日志记录功能

## 被收集日志主机B SyslogClient-IP08 CentOS8.4 IP: 192.168.250.8 启用网络日志记录功能

# 修改主机名及验证服务器时间同步
[root@CentOS84 ]#
[root@CentOS84 ]#hostnamectl set-hostname rsyslog-IP08
[root@CentOS84 ]#exit
[root@SyslogClient-IP08 ]#systemctl enable --now chronyd.service
[root@SyslogClient-IP08 ]#
[root@SyslogClient-IP08 ]#date
[root@SyslogClient-IP08 ]#

[root@SyslogClient-IP08 ]#rpm -qi rsyslog
Name : rsyslog
Version : 8.1911.0
...........

[root@SyslogClient-IP08 ]#rpm -ql rsyslog
# 主要的配置文件
/etc/logrotate.d/syslog
/etc/pki/rsyslog
/etc/rsyslog.conf
/etc/rsyslog.d
/etc/sysconfig/rsyslog
...........
# 很多SO模块
/usr/lib64/rsyslog/omhttp.so
/usr/lib64/rsyslog/omjournal.so
/usr/lib64/rsyslog/ommail.so
/usr/lib64/rsyslog/omprog.so
/usr/lib64/rsyslog/omstdout.so
/usr/lib64/rsyslog/omtesting.so
/usr/lib64/rsyslog/omuxsock.so
/usr/lib64/rsyslog/pmaixforwardedfrom.so
/usr/lib64/rsyslog/pmcisconames.so
/usr/lib64/rsyslog/pmlastmsg.so
/usr/lib64/rsyslog/pmsnare.so
/usr/sbin/rsyslogd
/var/lib/rsyslog

[root@SyslogClient-IP08 ]#vim /etc/rsyslog.conf
[root@SyslogClient-IP08 ]#cat /etc/rsyslog.conf
.............
#### RULES ####
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# 保留上面行，日志可保存本地；增加下面行，日志通过UDP514端口传送到 192.168.250.18 服务器记录
*.info;mail.none;authpriv.none;cron.none @192.168.250.18:514
.................
[root@SyslogClient-IP08 ]#systemctl restart rsyslog

## 提示说明 @ 表示UDP514协议传输日志 @@ 表示TCP514协议传输日志

2.1.2. CentOS7主机 192.168.250.7 开启日志记录功能

[root@CentOS79 ]#hostnamectl set-hostname SyslogClient-IP07
[root@CentOS79 ]#exit
[root@SyslogClient-IP07 ]#vim /etc/rsyslog.conf
[root@SyslogClient-IP07 ]#cat /etc/rsyslog.conf
.............
#### RULES ####
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# 保留上面行，日志可保存本地；增加下面行，日志通过TCP514端口传送到 192.168.250.18 服务器记录
*.info;mail.none;authpriv.none;cron.none @@192.168.250.18:514
.................
[root@SyslogClient-IP07 ]#systemctl restart rsyslog

2.2. 启用日志服务器网络记录功能

基本任务：设定这个以目录下日志文件记录来自网络的其他主机日志的服务器意图是能更好理解网络中日志传输流程和原理。本次计划将IP192.168.250.7和IP192.168.250.8两台主机日志，集中记录到IP192.168.250.18 的 /var/log/messages下，虽然集中记录在同一个文件内，日志文件中对不同主机的日志都有明确标识，能完美分辨出来自哪台主机的日志。

## 日志服务器 (文件记录方式) 主机名：Rsyslog-server-IP18 IP: 192.168.250.18的配置

[root@CentOS84 ]#hostnamectl set-hostname Rsyslog-server-IP18
[root@CentOS84 ]#exit
[root@Rsyslog-server-IP18 ]#systemctl enable --now chronyd.service
[root@Rsyslog-server-IP18 ]#

[root@Rsyslog-server-IP18 ]#cat /etc/rsyslog.conf
................
#### MODULES ####
................
# Provides UDP syslog reception 取消下面行行首的#注释符，让其生效，接受来自网络的UPD514日志
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# Provides TCP syslog reception 取消下面行行首的#注释符，让其生效，接受来自网络的TCP514日志
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
.................

[root@Rsyslog-server-IP18 ]#
[root@Rsyslog-server-IP18 ]#systemctl restart rsyslog

2.3. 验证日志传输和记录

# 在192.168.250.7上用logger发送一个测试日志信息
[root@SyslogClient-IP07 _{]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07}]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 _{]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07}]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 _{]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07}]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 _{]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07}]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 ~]#

# 在192.168.250.8上用logger发送一个测试日志信息
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#

# 在192.168.250.18上用查看日志信息记录
[root@Rsyslog-server-IP18 ]#tail -f /var/log/messages
...................
Mar 10 19:03:05 SyslogClient-IP08 root[21952]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:09 SyslogClient-IP08 root[21953]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:09 SyslogClient-IP08 root[21954]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:10 SyslogClient-IP08 root[21955]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:10 SyslogClient-IP08 root[21956]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:11 SyslogClient-IP08 root[21957]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:11 SyslogClient-IP08 root[21958]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:11 SyslogClient-IP08 root[21959]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:18 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:18 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:19 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP

3. 利用 MySQL 存储日志信息

基本任务：利用Rsyslog-server-IP18服务器的rsyslog日志服务， 将日志服务器 (文件记录方式) syslog-server-IP18收集记录的日志，转储到日志服务器(MySQ库)Syslog-MySQL-IP28上利用 MySQL 存储日志信息。

3.1. 在Rsyslog-server-IP18服务器上安装连接mysql模块相关的程序包

# 安装数据库连接包
[root@Rsyslog-server-IP18 ]#yum -y install rsyslog-mysql
# 验证并查看包内容
[root@Rsyslog-server-IP18 ]#rpm -ql rsyslog-mysql
/usr/lib/.build-id
/usr/lib/.build-id/e6
/usr/lib/.build-id/e6/aa0e40c19a2e0524d72780eee3b1698684cbe7
/usr/lib64/rsyslog/ommysql.so
/usr/share/doc/rsyslog/mysql-createDB.sql #这个是写好的后端数据库文件

# 查看SQL脚本文件内容，是后端服务器生成数据库和表用
[root@Rsyslog-server-IP18 ]#cat /usr/share/doc/rsyslog/mysql-createDB.sql
CREATE DATABASE Syslog;
USE Syslog;
CREATE TABLE SystemEvents
(
ID int unsigned not null auto_increment primary key,
CustomerID bigint,
ReceivedAt datetime NULL,
DeviceReportedTime datetime NULL,
Facility smallint NULL,
Priority smallint NULL,
FromHost varchar(60) NULL,
Message text,
NTSeverity int NULL,
Importance int NULL,
EventSource varchar(60),
EventUser varchar(60) NULL,
EventCategory int NULL,
EventID int NULL,
EventBinaryData text NULL,
MaxAvailable int NULL,
CurrUsage int NULL,
MinUsage int NULL,
MaxUsage int NULL,
InfoUnitID int NULL ,
SysLogTag varchar(60),
EventLogType varchar(60),
GenericFileName VarChar(60),
SystemID int NULL
);

CREATE TABLE SystemEventsProperties
(
ID int unsigned not null auto_increment primary key,
SystemEventID int NULL ,
ParamName varchar(255) NULL ,
ParamValue text NULL
);

#将sql脚本复制到数据库服务器192.168.250.28上
[root@Rsyslog-server-IP18 ]#scp /usr/share/doc/rsyslog/mysql-createDB.sql 192.168.250.28:/data/

3.2. 准备日志存放的MySQL数据库服务器

# 修改主机名和同步时间
[root@CentOS84 ]#hostnamectl set-hostname Syslog-MySQL-IP28
[root@CentOS84 ]#exit
[root@Syslog-MySQL-IP28 ]#systemctl enable --now chronyd.service
[root@Syslog-MySQL-IP28 ]#date
# 安装数据库并启动
[root@Syslog-MySQL-IP28 ]#yum -y install mysql-server
[root@Syslog-MySQL-IP28 ]#systemctl start mysqld
# 利用前面复制过来的sql脚本生成日志数据库和表
[root@Syslog-MySQL-IP28 ]#mysql < mysql-createDB.sql
[root@Syslog-MySQL-IP28 ]#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.26 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| Syslog |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.01 sec)

mysql> use Syslog
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+------------------------+
| Tables_in_Syslog |
+------------------------+
| SystemEvents |
| SystemEventsProperties |
+------------------------+
2 rows in set (0.00 sec)

# 在MySQL数据库服务器上创建好相关数据库和表，授权rsyslog IP18 能连接至当前服务器
mysql> create user rsyslog@'192.168.250.%' identified by 'shone123456';
Query OK, 0 rows affected (0.01 sec)

mysql> grant all on Syslog.* to rsyslog@'192.168.250.%';
Query OK, 0 rows affected (0.01 sec)

# 用下面命令记录下数据库内的记录
mysql> select * from SystemEvents\G

3.3. 配置日志服务器将日志发送到后端数据库服务器

基本内容：配置日志服务器IP192.168.250.18 将日志发送到后端数据库服务器IP192.168.250.28

# 配置rsyslog将日志保存到mysql中
# 对/etc/rsyslog.conf文件进行再次修改，增加 module(load="ommysql") 和 *.info;mail.none;authpriv.none;cron.none :ommysql:192.168.250.28,Syslog,rsyslog,shone123456 这两行。
[root@Rsyslog-server-IP18 ]#vim /etc/rsyslog.conf
[root@Rsyslog-server-IP18 ]#cat /etc/rsyslog.conf
................
#### MODULES ####
................
# 下面行为本次增加的内容，启动sql连接
module(load="ommysql")
# Provides UDP syslog reception
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# Provides TCP syslog reception
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

#### RULES ####

*.info;mail.none;authpriv.none;cron.none /var/log/messages
# 下面行为本次增加的内容，将日志转存到后端的MySQL数据库，定义和后端数据库连接的用户名和密&码
*.info;mail.none;authpriv.none;cron.none :ommysql:192.168.250.28,Syslog,rsyslog,shone123456

[root@Rsyslog-server-IP18 ]#systemctl restart rsyslog.service

3.4. 验证日志传输和记录

# 在两台日志client主机上发送两台测试日志
[root@syslogclient-ip07 ~]# logger "hello,I am log test! IP IS 192.168.250.7 TCP MYSQL LOG"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP MYSQL LOG"

# 在数据库内查询，并确认日志是否被正确记录
[root@Syslog-MySQL-IP28 ]#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 54
Server version: 8.0.26 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| Syslog |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)

mysql> use Syslog
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from SystemEvents\G
....................
*************************<strong> 37725. row </strong>***********************<strong>
ID: 37725
CustomerID: NULL
ReceivedAt: 2022-03-10 19:52:49
DeviceReportedTime: 2022-03-10 19:52:49
Facility: 1
Priority: 5
FromHost: SyslogClient-IP08
Message: hello,I am log test! IP IS 192.168.250.8 UDP MYSQL LOG
NTSeverity: NULL
Importance: NULL
EventSource: NULL
EventUser: NULL
EventCategory: NULL
EventID: NULL
EventBinaryData: NULL
MaxAvailable: NULL
CurrUsage: NULL
MinUsage: NULL
MaxUsage: NULL
InfoUnitID: 1
SysLogTag: root[22397]:
EventLogType: NULL
GenericFileName: NULL
SystemID: NULL
processid:

</strong>***********************<strong> 37729. row </strong>*************************
ID: 37729
CustomerID: NULL
ReceivedAt: 2022-03-10 19:52:53
DeviceReportedTime: 2022-03-10 19:52:53
Facility: 1
Priority: 5
FromHost: syslogclient-ip07
Message: hello,I am log test! IP IS 192.168.250.7 TCP MYSQL LOG
NTSeverity: NULL
Importance: NULL
EventSource: NULL
EventUser: NULL
EventCategory: NULL
EventID: NULL
EventBinaryData: NULL
MaxAvailable: NULL
CurrUsage: NULL
MinUsage: NULL
MaxUsage: NULL
InfoUnitID: 1
SysLogTag: root:
EventLogType: NULL
GenericFileName: NULL
SystemID: NULL
processid:
..............

4. 通过 loganalyzer 展示数据库中记录的日志

基本任务：前面完成了日志两台clients主机（192.168.250.7和192.168.250.8）、一台日志服务器（文件记录）192.168.250.18 、一台日志服务器（MySQL记录）192.168.250.28 的配置，在这个小章节内将完成在一台192.168.250.38服务器上安装httpd+php +loganalyzer以图形展示日志信息。

4.1. 安装 php和相关软件包

# 同步时间，修改主机名
[root@CentOS84 ]#hostnamectl set-hostname LogAnalyzer-IP38
[root@CentOS84 ]#exit
[root@LogAnalyzer-IP38 ]#systemctl enable --now chronyd.service

# 安装httpd，php，其他依赖包，其中php-gd是PHP中生成图形的支撑包，不安装无法汇出图
[root@LogAnalyzer-IP38 ]#yum -y install httpd php-fpm php-mysqlnd php-gd
# 启动服务
[root@LogAnalyzer-IP38 ]#systemctl enable --now httpd php-fpm

# 默认安装后的PHP是以UDS的方式运行
[root@LogAnalyzer-IP38 ]#ll /run/php-fpm/
total 4
-rw-r--r-- 1 root root 6 Mar 9 23:11 php-fpm.pid
srw-rw----+ 1 root root 0 Mar 9 23:11 www.sock
[root@LogAnalyzer-IP38 ]#ll /run/php-fpm/www.sock
srw-rw----+ 1 root root 0 Mar 9 23:11 /run/php-fpm/www.sock
[root@LogAnalyzer-IP38 ]#grep www.sock /etc/httpd/conf.d/php.conf
SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"

[root@LogAnalyzer-IP38 ]#rpm -qf /etc/httpd/conf.d/php.conf
php-fpm-7.2.24-1.module_el8.2.0+313+b04d0a66.x86_64

# 准备一个PHP测试页面
[root@LogAnalyzer-IP38 ]#vim /var/www/html/phpinfo.php
[root@LogAnalyzer-IP38 ]#cat /var/www/html/phpinfo.php
<?php phpinfo() ?>

​在浏览器内打开PHP测试页面，验证PHP环境正确配置了 http://192.168.250.38/phpinfo.php​

4.2. 下载LogAnalyzer

登录官网 https://loganalyzer.adiscon.com/ ，下载 loganalyzer-4.1.12.tar.gz安装包

4.3. 安装 LogAnalyzer

# 解压安装包
[root@LogAnalyzer-IP38 ]#ll
-rw-r--r-- 1 root root 5028816 Mar 9 23:12 loganalyzer-4.1.12.tar.gz
[root@LogAnalyzer-IP38 ]#tar xvf loganalyzer-4.1.12.tar.gz
[root@LogAnalyzer-IP38 ]#ll
drwxrwxr-x 5 root root 90 Apr 29 2021 loganalyzer-4.1.12
-rw-r--r-- 1 root root 5028816 Mar 9 23:12 loganalyzer-4.1.12.tar.gz
[root@LogAnalyzer-IP38 ]#ll loganalyzer-4.1.12
total 104
-rw-rw-r-- 1 root root 50019 Apr 29 2021 ChangeLog
drwxrwxr-x 2 root root 43 Apr 29 2021 contrib
-rw-rw-r-- 1 root root 35497 Apr 29 2021 COPYING
drwxrwxr-x 3 root root 258 Apr 29 2021 doc
-rw-rw-r-- 1 root root 8449 Apr 29 2021 INSTALL
drwxrwxr-x 13 root root 4096 Apr 29 2021 src
# 将loganalyzer源文件迁移到网站默认目录/var/www/html/下
[root@LogAnalyzer-IP38 ]#mv loganalyzer-4.1.12/src/ /var/www/html/log

# 创建好后面PHP初始化需要重写的配置文件，并授权。如果不创建这个文件，初始化会提示错误。
[root@LogAnalyzer-IP38 ]#touch /var/www/html/log/config.php
[root@LogAnalyzer-IP38 ]#chmod 666 /var/www/html/log/config.php

4.4. 基于 web 页面初始化

4.4.1 浏览器内访问 http://192.168.250.38/log 实现初始化

选择：MySQL Native, Syslog Fields, Monitorware

4.4.2 配置完成后可以以图形方式显示存放在MySQL中的日志

4.5. 安全加固

缩小配置文件权限，提高安全性，也避免这个文件被重新。

[root@LogAnalyzer-IP38 ]#chmod 644 /var/www/html/log/config.php

4.6. 验证图形下日志信息

# 在两台日志clients主机上发送几条日志测试信息
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP MYSQL LOG"

[root@syslogclient-ip07 ~]#logger "hello,I am log test! IP IS 192.168.250.7 TCP MYSQL LOG"

在图形界面下可以看到相关的日志记录，完美展示出来了

至此，在一个小型LINUX环境内搭建日志收集、集中保存和展示的任务全部完成。在大型的计算环境下，一般用ELK（ELK：由Elasticsearch, Logstash, Kibana三个软件组成）对所有主机的日志进行记录、管理和分析，这些内容将在后面的文章中再介绍。