一、机器环境
3台4C8G50GB磁盘虚拟机
角色
IP
主机名
master
10.101.14.148
k8s-master-10-101-14-148
node
10.101.14.19
node1-10-101-14-19
node
10.101.14.192
node2-10-101-14-192
二、虚拟机系统版本和k8s版本
操作系统/内核版本
AnliOS7.9/4.19.91-25.3
kubernetes
v1.24.3
containerd
v1.6.7
本次环境使用的AnliOS7.9为国内龙蜥社区开源版本，在CentOS停止更新之后，可以有效的替换，该系统完全兼容CentOS感兴趣可以了解下龙蜥系统
三、集群环境部署
3.1、集群环境准备(所有节点)
3.1.1、修改主机名
#修改主机名：10.101.14.148hostnamectl set-hostname k8s-master-10-101-14-148#修改主机名：10.101.14.19hostnamectl set-hostname node1-10-101-14-19#修改主机名：10.101.14.192hostnamectl set-hostname node2-10-101-14-192
3.1.2 、关闭防火墙和selinux
每个节点关闭
systemctl stop firewalld && systemctl disable firewallsetenforce 0sed -i 's/SELINUX=enforcing/SELINUX=disabled/g'
3.1.3、配置hosts
每个节点配置hosts信息
cat > /etc/hosts << EOF127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.101.14.13 k8s-master-10-101-14-1310.101.14.19 node1-10-101-14-1910.101.14.192 node2-10-101-14-192EOF
3.1.4 加载内核模块并开启
每个节点开启内核模块
modprobe br_netfiltercat > /etc/sysctl.d/k8s.conf << EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1EOFsysctl -p
安装ipvs模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modulesyum install ipset -yyum install ipvsadm -y
3.1.5 同步服务器时间
yum install chrony -y systemctl enable chronyd systemctl start chronyd chronyc sources
3.2 安装containerd(所有节点)
3.2.1 配置yum源和依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install containerd.io -y
3.2.2 安装、配置和启动containerd
配置和启动containerd
mkdir -p /etc/containerd containerd config default > /etc/containerd/config.toml#替换配置文件(国内因为防火墙的原因)sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" /etc/containerd/config.tomlsed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g" /etc/containerd/config.toml sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml systemctl daemon-reload systemctl enable containerd systemctl restart containerd
3.3 配置、安装kubeadm(所有节点)
当Containerd安装启动完成后，现在可以来安装 Kubeadm 了，我们这里是通过指定使用阿里云的yum源进行安装
3.3.1 配置阿里云的k8s yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
3.3.2 安装kubeadm、kubelet、kubectl，并配置crictl
# 安装kubeadm kubelet kubectl包yum install kubelet-1.24.3 kubeadm-1.24.3 kubectl-1.24.3 -y#生成 crictl 的配置文件，用于连接到containerdcat > /etc/crictl.yaml <<EOFruntime-endpoint: unix:///run/containerd/containerd.sockimage-endpoint: unix:///run/containerd/containerd.socktimeout: 10debug: falseEOF#启动kubelet服务systemctl daemon-reloadsystemctl enable kubelet && systemctl start kubelet
四、初始化集群
4.1.1 master节点执行
kubeadm config print init-defaults > kubeadm.yaml
根据自己的需求修改配置，包括 imageRepository 的值，kube-proxy模式等，我们使用的containerd作为运行时，在初始化节点的时候需要指定cgroupDriver为systemd
apiVersion: kubeadm.k8s.io/v1beta3bootstrapTokens:- groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authenticationkind: InitConfigurationlocalAPIEndpoint: advertiseAddress: 10.101.14.148 # 修改指定Master的地址 bindPort: 6443nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent name: k8s-master-10-101-14-148 # 修改为master的主机名 taints: # 给master节点打上污点，不允许调度应用容器到上面 - effect: NoSchedule key: node-role.kubernetes.io/master ------apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationmode: ipvs # 修改kube-proxy模式为ipvs ---apiServer: timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta3certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns: {}etcd: local: dataDir: /var/lib/etcdimageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # 修改获取镜像的地址kind: ClusterConfigurationkubernetesVersion: 1.24.3 # 指定k8s的版本networking: dnsDomain: cluster.local podSubnet: 172.16.0.0/16 # 指定pod的CIDR serviceSubnet: 10.96.0.0/12 # 指定service的CIDRscheduler: {}---apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd # 指定cgroupDrive为systemd
使用上述生成的配置文件进行初始化：
kubeadm init --config=kubeadm.yaml[init] Using Kubernetes version: v1.24.3 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.5] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.5 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.5 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. [apiclient] All control plane components are healthy after 56.001862 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)" [mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.101.14.13:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:55c5c0593f8e07a51aaa6dac6c0187a289dee30998dc9cf58516a2195cde8fb1
根据安装提示开始拷贝kubeconfig文件
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
4.1.2 node节点执行
kubeadm join 10.101.14.13:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:55c5c0593f8e07a51aaa6dac6c0187a289dee30998dc9cf58516a2195cde8fb1 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
4.1.3 master节点calico网络插件安装
Calico网络安装方法
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.3/manifests/tigera-operator.yaml#获取网络插件修改pod地址wget https://raw.githubusercontent.com/projectcalico/calico/v3.24.3/manifests/custom-resources.yaml
编辑custom-resources.yaml文件，修改cidr地址和kubeadm.yaml中pod cidr保持一致
# This section includes base Calico installation configuration.# For more information, see: https://projectcalico.docs.tigera.io/v3.23/reference/installation/api#operator.tigera.io/v1.InstallationapiVersion: operator.tigera.io/v1kind: Installationmetadata: name: defaultspec: # Configures Calico networking. calicoNetwork: # Note: The ipPools section cannot be modified post-install. ipPools: - blockSize: 26 cidr: 172.16.0.0/16 #修改该地址 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all()---# This section configures the Calico API server.# For more information, see: https://projectcalico.docs.tigera.io/v3.23/reference/installation/api#operator.tigera.io/v1.APIServerapiVersion: operator.tigera.io/v1kind: APIServer metadata: name: default spec: {}
执行custom-resources.yaml文件
kubectl create -f custom-resources.yaml
至此基本上安装成功没有问题了，稍等几分钟，可以通过命令查看状态,
#查看集群状态kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME STATUS MESSAGE ERRORscheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health":"true","reason":""}
可以看到集群中调度服务，控制器服务，etcd服务均是正常的，说明没有问题，再看看calico服，也是running状态，至此k8s服务搭建起来了
kubectl get pod -n kube-systemNAME READY STATUS RESTARTS AGEcoredns-7f74c56694-2hsxl 1/1 Running 0 77dcoredns-7f74c56694-rssvf 1/1 Running 0 77detcd-k8s-master-10-101-14-148 1/1 Running 0 77dkube-apiserver-k8s-master-10-101-14-148 1/1 Running 0 77dkube-controller-manager-k8s-master-10-101-14-148 1/1 Running 0 77dkube-proxy-d2896 1/1 Running 0 77dkube-proxy-tlqrt 1/1 Running 0 77dkube-proxy-wvsmg 1/1 Running 0 77dkube-scheduler-k8s-master-10-101-14-148 1/1 Running 0 77d