Kubernetes集群实践（10）滚动更新和回滚

Kubernetes集群的证书有效期是1年，如果超过有效期，kublet服务会无效，此时继续使用kubelet命令，将会得到类似这样的提示“Unable to connect to the server: x509: certificate has expired or is not yet valid.”因此，我们需要对其进行维护。

登陆Master节点查看证书有效期

kubeadm alpha certs check-expiration

备份现有证书

mkdir -p $HOME/k8s-old-certs/pki cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki ls -l $HOME/k8s-old-certs/pki/

结果：

total 56 -rw-r--r-- 1 root root 1261 Sep 4 2019 apiserver.crt -rw-r--r-- 1 root root 1090 Sep 4 2019 apiserver-etcd-client.crt -rw------- 1 root root 1679 Sep 4 2019 apiserver-etcd-client.key -rw------- 1 root root 1679 Sep 4 2019 apiserver.key -rw-r--r-- 1 root root 1099 Sep 4 2019 apiserver-kubelet-client.crt -rw------- 1 root root 1679 Sep 4 2019 apiserver-kubelet-client.key -rw-r--r-- 1 root root 1025 Sep 4 2019 ca.crt -rw------- 1 root root 1675 Sep 4 2019 ca.key -rw-r--r-- 1 root root 1038 Sep 4 2019 front-proxy-ca.crt -rw------- 1 root root 1675 Sep 4 2019 front-proxy-ca.key -rw-r--r-- 1 root root 1058 Sep 4 2019 front-proxy-client.crt -rw------- 1 root root 1679 Sep 4 2019 front-proxy-client.key -rw------- 1 root root 1675 Sep 4 2019 sa.key -rw------- 1 root root 451 Sep 4 2019 sa.pub

备份配置文件

cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs ls -ltr $HOME/k8s-old-certs

结果：

total 36 -rw------- 1 root root 5451 Sep 4 2019 admin.conf -rw------- 1 root root 5595 Sep 4 2019 kubelet.conf -rw------- 1 root root 5483 Sep 4 2019 controller-manager.conf -rw------- 1 root root 5435 Sep 4 2019 scheduler.conf drwxr-xr-x 2 root root 4096 Dec 19 21:21 pki

备份Home目录的配置文件

mkdir -p $HOME/k8s-old-certs/.kube cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/. ls -l $HOME/k8s-old-certs/.kube/.

结果：

-rw------- 1 root root 5451 Sep 4 2019 config

更新Kubernetes证书

kubeadm alpha certs renew all

结果

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healtcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed

在次检查证书有效期是否是364天后无效

kubeadm alpha certs check-expiration

确保kubelet服务器正常，同时work和master间通信正常。

等待几分钟，使用如下命令确保work可用

kubectl get nodes

如果输出以下信息

The connection to the server 9.37.21.119:6443 was refused - did you specify the right host or port?

则需要执行下面步骤继续进行修复

检查比/etc/kubernetes/kubelet.conf文件

diff $HOME/k8s-old-certs/kubelet.conf /etc/kubernetes/kubelet.conf

如果没有任何输出，则更新证书操作没有影响此文件，继续下面的步骤手动修复。

更新/etc/kubernetes/kubelet.conf文件

cd /etc/kubernetes sudo kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf diff $HOME/k8s-old-certs/kubelet.conf /etc/kubernetes/kubelet.conf

更新后可以查看与备份文件的区别.

检查对比~/.kube/config文件

diff ~/.kube/config $HOME/fcik8s-old-certs/.kube/config

如果没有任何输出，则该文件包含过期的key和证书，继续下面的步骤手动修复。

使用当前更新后的/etc/kubernetes/kubelet.conf中的'client-certificate-data '和'client-key-data'的值更新~/.kube/config文件中对应的值。

重启kubelet服务

systemctl daemon-reload&&systemctl restart kubelet

再次查看节点和Pod是否正常

kubectl get nodes ... kubectl get pods

参考资料：https://www.ibm.com/docs/en/fci/1.1.0?topic=kubernetes-renewing-cluster-certificates