kube-eventer事件监控

文章链接

下载deployment

我这里保存成kube-event.yaml

# cat kube-event.yaml --- apiVersion: apps/v1 kind: Deployment metadata: labels: name: kube-eventer name: kube-eventer namespace: kube-system spec: replicas: 1 selector: matchLabels: app: kube-eventer template: metadata: labels: app: kube-eventer annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: dnsPolicy: ClusterFirstWithHostNet serviceAccount: kube-eventer containers: - image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.2.0-484d9cd-aliyun name: kube-eventer command: - "/kube-eventer" - "--source=kubernetes:https://kubernetes.default" ## .e.g,dingtalk sink demo #- --sink=dingtalk:[your_webhook_url]&label=[your_cluster_id]&level=[Normal or Warning(default)] - --sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=355cf0156xxxxxxxxxxxxxxxxxx&level=Warning env: # If TZ is assigned, set the TZ value as the time zone - name: TZ value: "Asia/Shanghai" volumeMounts: - name: localtime mountPath: /etc/localtime readOnly: true - name: zoneinfo mountPath: /usr/share/zoneinfo readOnly: true resources: requests: cpu: 100m memory: 100Mi limits: cpu: 500m memory: 250Mi volumes: - name: localtime hostPath: path: /etc/localtime - name: zoneinfo hostPath: path: /usr/share/zoneinfo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-eventer rules: - apiGroups: - "" resources: - configmaps - events verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-eventer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-eventer subjects: - kind: ServiceAccount name: kube-eventer namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: kube-eventer namespace: kube-system

钉钉群里创建自定义webhook

设置--智能群助手--添加机器人--选择WeebHook。定义机器人名称和安全设置

安全设置这里我定义了关键字，Waring。创建后复制webhook地址。然后更改上面deployment中的sink处。

我把上面的label删掉了，只留下了level=Waring，刚好对应了我关键字的Waring。只有带有关键字的才会触发告警。

测试告警

然后创建一个测试的Tomcat的deployment，故意把image镜像的tag写错，让他无法拉取镜像

[root@master allenjol]# kubectl apply -f deploy-tomcat-test.yaml deployment.apps/tomcat-deployment-allenjol created service/tomcat-service-allenjol created [root@master allenjol]# kubectl get po NAME READY STATUS RESTARTS AGE tomcat-deployment-allenjol-b6687f99-l5vj9 0/1 ImagePullBackOff 0 45s

部署kube-event.yaml并查看日志。可以看到隔30s去检测一次

]# kubectl apply -f kube-event.yaml ]# kubectl get po -n kube-system | grep kube-event [root@master allenjol]# kubectl logs -f kube-eventer-648f64c985-zfkkg -n kube-system I0708 09:26:36.409034 1 eventer.go:67] /kube-eventer --source=kubernetes:https://kubernetes.default --sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=355cf01569aef206dc6c05681aaf3ed0ea19ed3597db4c26c565dbeb69ce1303&level=Warning I0708 09:26:36.409191 1 eventer.go:68] kube-eventer version: v1.2.0 commit: 484d9cd I0708 09:26:36.411557 1 eventer.go:94] Starting with DingTalkSink sink I0708 09:26:36.411596 1 eventer.go:108] Starting eventer I0708 09:26:36.411678 1 eventer.go:116] Starting eventer http service I0708 09:27:00.000163 1 manager.go:102] Exporting 5 events I0708 09:27:30.000130 1 manager.go:102] Exporting 9 events I0708 09:28:00.000147 1 manager.go:102] Exporting 1 events I0708 09:28:30.000150 1 manager.go:102] Exporting 4 events I0708 09:29:00.000138 1 manager.go:102] Exporting 1 events ...

可以看到这里已经看到了钉钉的webhook地址了，并且还收集到了events。查看钉钉群，就会看到已经出现了告警了。

其实这个告警当前还存在点问题。个人认为不应该这么频繁发送，应该像prometheus一样可以配置抑制和静默。然后监控时间可以更改。当然熟悉go语言可以自己改源码然后构建成镜像。文章链接