特色栏目： python 批处理 net编程 Javascript Php Asp Css Html5 Android seo centos

k8s dashboard 配置指导

来源：互联网收集：自由互联发布时间：2022-06-20

k8s dashboard 配置指导背景开发人员登陆服务器yaml去部署项目，一来开发人员登陆服务器显得权限太大不好管理，二来希望有图形化的后台去可视化部署维护项目，所以就用到了k8s的d

k8s dashboard 配置指导

背景

开发人员登陆服务器yaml去部署项目，一来开发人员登陆服务器显得权限太大不好管理，二来希望有图形化的后台去可视化部署维护项目，所以就用到了k8s的dashboad，这个程序会调用k8s集群的api，展示我们集群的所有资源，还可以基于serviceaccount rbac定制化对用户进行授权鉴权，其实对于多数的场景k8s的dashboard足够用了，你可以基于python+dgango+laiui框架自定义调用k8s的客户端api库接口实例组实现运开平台的开发，也可以基于java或者go开发都是可以的，但是自定义开发的后台一般都是和其他流程功能结合的，比如流水线，镜像库等等，很少单独存在。

安装部署

采用svc nodeport方式部署dashboard，其次，token的时间设置为永不过期

# vi kubernertes-dashboard.yaml //如果设置登陆dashboard token过期为永不过期，需要在dashboard镜像 command里面加个参数- --token-ttl=0，不加默认token过期时间是15分钟，我们并不希望来回的切换token # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # 在kubernetes-dashboard命名空间下创建 apiVersion: v1 kind: Namespace metadata: name: kubernetes-dashboard --- # 创建一个sa服务账号 apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard --- # svc nodeport方式部署，宿主机暴露30001端口，实际生产环境需要避免端口冲突，也不可以不指定svc自动分配 kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard --- # dashboard程序的secret和cm，这里都没有配置为空 apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboard type: Opaque data: csrf: "" --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboard type: Opaque --- kind: ConfigMap apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard --- #创建了Role和ClsterRole，进行一个权限的集合定义，dashboard的role、clusterrole 授予操作k8s集群资源的权限，很显然这里定义的role、ClsterRole权限远远不够，ingress、deploy都没有对吧 kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"] --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"] --- #RoleBinding是sa（serviceaccount服务账号）绑定role、clusterole权限的，这里是名为kubernetes-dashboard这个sa绑定的上面定义的role和clusterrole apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- #这里进行deployment部署dashboard kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.0.3 //镜像用的是v2.0.3的版本 imagePullPolicy: Always ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates - --namespace=kubernetes-dashboard - --token-ttl=0 //arg里设置token永不过期 # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule --- #下面是部署dashboard里自带监控指标看板的的容器pod kind: Service apiVersion: v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper annotations: seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' spec: containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.4 ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {}

查看dashboard运行状态

# kubectl get pod -o wide -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES dashboard-metrics-scraper-7b59f7d4df-ksn7g 1/1 Running 0 65s 10.244.36.74 k8s-node1 <none> <none> kubernetes-dashboard-9bfbfc865-jbgcv 1/1 Running 0 65s 10.244.36.73 k8s-node1 <none> <none> # kubectl get svc -n kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.99.210.9 <none> 8000/TCP 73s kubernetes-dashboard NodePort 10.108.97.224 <none> 443:30001/TCP 73s

访问地址：https://NodeIP:30001

配置dashboard admin sa role rolebinding

很显然kubernetes-dashboard这个sa绑定的role、clusterrole权限并不大

# kubectl get sa -A | grep kubernetes-dashboard kubernetes-dashboard default 1 7m14s kubernetes-dashboard kubernetes-dashboard 1 7m14s # kubectl get role -A | grep kubernetes-dashboard kubernetes-dashboard kubernetes-dashboard 2021-08-22T10:18:38Z # kubectl get rolebinding -A | grep kubernetes-dashboard kubernetes-dashboard kubernetes-dashboard Role/kubernetes-dashboard 22m # kubectl edit role kubernetes-dashboard -n kubernetes-dashboard //可以看出改kubernetes-dashboard sa绑定的role只是kubernetes-dashboard命名空间下一些secret cm的权限 apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: - apiGroups: - "" resourceNames: - kubernetes-dashboard-key-holder - kubernetes-dashboard-certs - kubernetes-dashboard-csrf resources: - secrets verbs: - get - update - delete - apiGroups: - "" resourceNames: - kubernetes-dashboard-settings resources: - configmaps # kubectl get clusterrole -A | grep kubernetes-dashboard kubernetes-dashboard 2021-08-22T09:44:16Z # kubectl get clusterrolebinding -A | grep kubernetes-dashboard kubernetes-dashboard ClusterRole/kubernetes-dashboard 41m # kubectl edit clusterrole kubernetes-dashboard //可以看出改kubernetes-dashboard sa绑定的cluaterrole 调用的是metrics.k8s.io这个监控相关的api组 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: - apiGroups: - metrics.k8s.io resources: - pods - nodes verbs: - get - list - watch

程序或者用户去访问k8s集群，都是基于kubeconifg配置文件或者serviceaccount去访问apiservers的，kubeconfig就是基于ca证书秘钥的https方式访问apiservers，而serviceaccount是基于token（token信息保存在secret中）方式去访问apiservers的，很显然dashboard创建的sa、role、clusterrole不能满足鉴权的需求

# kubectl get secret -A | grep kubernetes-dashboard kubernetes-dashboard default-token-mqm56 kubernetes.io/service-account-token 3 15m kubernetes-dashboard kubernetes-dashboard-certs Opaque 0 15m kubernetes-dashboard kubernetes-dashboard-csrf Opaque 1 15m kubernetes-dashboard kubernetes-dashboard-key-holder Opaque 2 15m kubernetes-dashboard kubernetes-dashboard-token-q88sf kubernetes.io/service-account-token 3 15m # kubectl describe secret kubernetes-dashboard-token-q88sf -n kubernetes-dashboard Name: kubernetes-dashboard-token-q88sf Namespace: kubernetes-dashboard Labels: <none> Annotations: kubernetes.io/service-account.name: kubernetes-dashboard kubernetes.io/service-account.uid: c3fab341-6322-4210-ab5b-33bc1f88222e Type: kubernetes.io/service-account-token Data ==== namespace: 20 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjJXaVMtcVlJTXVLQlJ0OTJoWEhkUGRjQ1pCOVZ1LU5VTkgzYkQzam1RQVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1xODhzZiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMzZmFiMzQxLTYzMjItNDIxMC1hYjViLTMzYmMxZjg4MjIyZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.aeUcDnjXsIbr9i3UvtW1QRIzzJpVdqDOrRfkZVVzY7bPZ1-J7oEUrijH32lrSm1ro8oibyNqO9QN3RyNRZR9PuTsHmz0C6D4_0Ha5S7BfHqTWURaZW0_xJiR6u1sDgYc84UtG_FdBFHZNLKAr0ULl0uHVwvY4dNfTgkYXUwVi6kjsdPKTZcz4dhIOP3XC5q2cAwCcYIaXPAvUE8YeSoYGpX1RQwpiz5qtUa7bUXyReK5_Eeb05_ufIvRgLRaSbH2YI1FzI9n0RNTka7vTG640oCW4TgtMXXInRh1HturPMPAHsamTMisq0ttGozoTZP2ELO-ZS4qR_F0LQKhBQoAKA

查看cluster-admin 绑定的clusterrole

# kubectl get clusterrole -A | grep cluster-admin cluster-admin # kubectl edit clusterrole cluster-admin apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: rules: - apiGroups: - '*' resources: - '*' verbs: - '*' - nonResourceURLs: - '*' verbs: - '*' # kubectl edit clusterrolebinding cluster-admin apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: kubernetes.io/bootstrapping: rbac-defaults name: cluster-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - apiGroup: rbac.authorization.k8s.io kind: Group //cluster-admin不是sa账号是一个组超级管理员账号 name: system:masters

我们需要创建一个sa账号dashboard-admin 绑定clusterrolebinding集群内部cluster-admin超级管理员的绑定的clusterrole，实现对所有k8s集群资源的授权操作

# 创建dashboard-admin这个sa $ kubectl create serviceaccount dashboard-admin -n kube-system # 为dashboard-admin这个sa clusterrolebinding 绑定集群超级管理员绑定用的clusterrole ，使得dashboard-admin这个sa获得集群的所有权 $ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin # 获取dashboard-admin这个sa的token $ kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

查看dashboard-admin这个sa的token

# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') Name: dashboard-admin-token-ncrw6 Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: dashboard-admin kubernetes.io/service-account.uid: cab42a17-328f-4eb9-81af-bd810143c456 Type: kubernetes.io/service-account-token Data ==== token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjJXaVMtcVlJTXVLQlJ0OTJoWEhkUGRjQ1pCOVZ1LU5VTkgzYkQzam1RQVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tbmNydzYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiY2FiNDJhMTctMzI4Zi00ZWI5LTgxYWYtYmQ4MTAxNDNjNDU2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.W1a0bDx3-3eycYUGRSXu7dzP-ibaoYLCd9Rj2dGMkHGSEXCiAysMpWl8G5_zIyRVzxmRzv7ay78X7khij6Dg9mLJq0K1NXMp0j706qKp4DgSdGM3qhhbE97BbEUKg-NjKplUQvejvFUiDEV0pi-neBiZeBP8mIbmGxYqPGO3eZP-ZnUJ8oQ5rluZcIGtlObhTn8UwsXMdBNnbD6f5eSGdNurEZhz2WZ2URpPrXWo6ewvqwcouxN4mmYM0419oKrxo1GPxhpgC5751reNfpBRWepd9XeuS1-uBPpMIjw0LU3151XhIrN0g5rSQOCjGDkMYf23fUy9HWihgjuwToOCNQ ca.crt: 1066 bytes namespace: 11 bytes

用dashboard-admin这个sa的token的这个token连接k8s-dashboard

dashboard看板功能

总体资源看板

具体资源看板（deploy为例）

可以基于namesapce查看deploy的总利用率和部署情况，副本数等等

更新发布

还可以进行一个在线的更新发布，比如更新deploy的镜像，直接编辑yaml即可，例如把nginx-deploy的pod镜像nginx1.17版本更换成nginx1.16版本

登陆服务器查看正在更新中

更新完毕确认nginx版本为1.16

# curl -i 10.244.36.75 HTTP/1.1 200 OK Server: nginx/1.16.1 Date: Sun, 22 Aug 2021 11:23:19 GMT

查看pod日志

exec进入容器

总体节点看板

![image-20210822190923644]

节点详情看板

显示节点cpu、内存压力，节点上pod的cpu、内存压力等，节点的kubelet资源预留与驱逐信息

发布一个新服务

![image-20210822193151774]

deploy创建一个nginx1.14服务为例，可以直接在dashboard里写yaml，也可以传yaml文件到dashboard中

dashboard基于命名空间的权限分发

服务根据namesapces划分，本着权限最小化原则，我们给每个领域或者团队创建一个sa账号token，实现每个团队只能看到操作自己namespace下的k8s资源

举例：为vtest2团队创建vtest2 sa账号对vtest2 命名空间授予所有资源操作权限

# vi vtest2.yaml apiVersion: v1 kind: ServiceAccount metadata: name: vtest2 namespace: vtest2 --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: vtest2-role namespace: vtest2 rules: //apigroups组 resources资源 ver动作符都是*，代表所有，你可以可以单独定义资源操作 - apiGroups: - '*' resources: - '*' verbs: - '*' --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: vtest2-rolebinding namespace: vtest2 roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: vtest2-role subjects: - kind: ServiceAccount name: vtest2 namespace: vtest2

查看vtest2 sa的token

# kubectl describe secrets -n vtest2 $(kubectl -n vtest2 get secret | awk '/vtest2/{print $1}') Name: vtest2-token-2jg55 Namespace: vtest2 Labels: <none> Annotations: kubernetes.io/service-account.name: vtest2 kubernetes.io/service-account.uid: 67f46bf1-de03-4496-93a2-e34df1115426 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1066 bytes namespace: 6 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjJXaVMtcVlJTXVLQlJ0OTJoWEhkUGRjQ1pCOVZ1LU5VTkgzYkQzam1RQVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ2dGVzdDIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoidnRlc3QyLXRva2VuLTJqZzU1Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZ0ZXN0MiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY3ZjQ2YmYxLWRlMDMtNDQ5Ni05M2EyLWUzNGRmMTExNTQyNiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp2dGVzdDI6dnRlc3QyIn0.tMKPRjFiYtWyKlwpE4kf1MAntCyENdDwjbcp9q8XRX8eTP82hjO5-IC24YG5Ht-rlmZtq8wtNqbMRM5e3uhLi8sUNe5kEAx_F0-hfbiRM39E-xCPh41kL7fvYCwpOvZoty9xKSJRuLdoqcZKOzvU22gJ9o_9DGhp0pbPemYEg9kFFM7HgOV6CW2HwKdcDP5vyzEd1zUbwGHJ97ISUUKS3y4NKlysGwrRB18r6518RsNw3cXvdkOZDDWB_Ysmi67utEfhPqALP1mM02v0lrga8wqd4L1lCrVRZKzs4CjMoswRzyEn4efBVSankDhdweU9-xDDEbwcOq8ZthsZjhCsg

服务器查看vtest2 namespace资源