一、认识ingress 1.1、什么是Ingress ingress公开从集群外部到集群内服务的http和https路由，流量路由由ingress资源还是那个定义规则控制（授权入站连接到达集群服务的规则集合）。由于ser
一、认识ingress
1.1、什么是Ingress
ingress公开从集群外部到集群内服务的http和https路由，流量路由由ingress资源还是那个定义规则控制（授权入站连接到达集群服务的规则集合）。由于service的IP集群外不能访问，就使用ingress方式再代理一次，即ingress代理service，service代理pod。
可以给Ingress配置提供外部可访问的URL、负载均衡、SSL、基于名称的虚拟主机等。用户通过POST Ingress资源到API server的方式来请求ingress。 Ingress controller负责实现Ingress，通常使用负载平衡器，它还可以配置边界路由和其他前端，这有助于以HA方式处理流量。
1.2、Ingress工作原理
ingress具体的工作原理如下:
- step1：ingress contronler通过与k8s的api进行交互，动态的去感知k8s集群中ingress服务规则的变化，然后读取它，并按照定义的ingress规则，转发到k8s集群中对应的service。
- step2：而这个ingress规则写明了哪个域名对应k8s集群中的哪个service，然后再根据ingress-controller中的nginx配置模板，生成一段对应的nginx配置。
- step3：然后再把该配置动态的写到ingress-controller的pod里，该ingress-controller的pod里面运行着一个nginx服务，控制器会把生成的nginx配置写入到nginx的配置文件中，然后reload一下，使其配置生效,以此来达到域名分配置及动态更新的效果。
1.3、Ingress解决的问题
- 1）动态配置服务
如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作。
- 2）减少不必要的端口暴露
配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式。
1.4、Ingress运行方式
- 以Deployment控制器管理Ingress控制器的Pod资源，并通过NodePort或LoadBalancer类型的Service对象为其接入集群外部的请求流量，这就意味着，定义一个Ingress控制器时，必须在其前端定义一个专用的Service资源。
- 借助于DaemonSet控制器，将Ingress控制器的Pod资源各自以单一实例的方式运行于集群的所有或部分工作节点之上，并配置这类Pod对象以hostPort或hostNetwork的方式在当前节点接入外部流量。
1.5、Ingress定义资源清单字段
- apiVersion： v1 版本
- kind： Ingress 类型
- metadata 元数据
- spec 期望状态
- backend： 默认后端，能够处理与任何规则不匹配的请求
- rules：用于配置Ingress的主机规则列表
- tls：目前Ingress仅支持单个TLS端口443
- status 当前状态
二、部署Ingress
2.1、部署Ingress(deployment的方式）
2.1.1、下载文件并部署
gitlab ingress-nginx项目：https://github.com/kubernetes/ingress-nginx
ingress安装指南：https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/index.md
ingress-nignx下载路径（NodePort）：https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml
[root@k8s-master ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml
[root@k8s-master ingress]# kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
如果没有链接外网可以对镜像进行修改
configmap.yaml:提供configmap可以在线更新nginx的配置
default-backend.yaml:提供一个缺省的后台错误页面 404
namespace.yaml:创建一个独立的命名空间 ingress-nginx
rbac.yaml:创建对应的role rolebinding 用于rbac
tcp-services-configmap.yaml:修改L4负载均衡配置的configmap
udp-services-configmap.yaml:修改L4负载均衡配置的configmap
with-rbac.yaml:有应用rbac的nginx-ingress-controller组件
2.1.2、默认已安装Service，如果没有安装service的话，可以重新安装
[root@k8s-master ingress]# cat service-nodeport.yaml
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.5.1
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
---
[root@k8s-master ingress]# kubectl apply -f service-nodeport.yaml
service/ingress-nginx-controller created
2.1.3、查看各服务
[root@k8s-master ingress]# kubectl get pod,svc -ningress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-t96bk 0/1 Completed 0 10m
pod/ingress-nginx-admission-patch-kwzx5 0/1 Completed 1 10m
pod/ingress-nginx-controller-6668b8dd46-4c8rt 1/1 Running 0 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.102.162.187 <none> 80:31648/TCP,443:30422/TCP 10m
service/ingress-nginx-controller-admission ClusterIP 10.105.57.63 <none> 443/TCP 10m
至此，ingress-nginx服务已经通过deployment的方式部署至kubernetes环境中
2.1.4、创建Ingress，代理到后端nginx服务
创建后端的pod和service
###创建pod和service的yaml文件
[root@k8s-master ingress]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: nginx-app
template:
metadata:
labels:
app: nginx-app
spec:
containers:
- name: nginx
imagePullPolicy: Always
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: default
spec:
selector:
app: nginx-app
ports:
- name: nginx-port
port: 80
targetPort: 80
protocol: TCP
###应用该yaml文件
[root@k8s-master ingress]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deployment created
service/nginx-service created
###查询验证
[root@k8s-master ingress]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 160d
nginx-service ClusterIP 10.108.43.35 <none> 80/TCP 119s
[root@k8s-master ingress]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-65f969c774-ccjzg 1/1 Running 0 13d
nginx-deployment-6f7d8d4d55-nt4k7 1/1 Running 0 2m17s
nginx-deployment-6f7d8d4d55-qrfzd 1/1 Running 0 2m17s
[root@k8s-master ingress]# kubectl describe svc nginx-service
Name: nginx-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx-app
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.108.43.35
IPs: 10.108.43.35
Port: nginx-port 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.3.224:80,10.244.3.225:80
Session Affinity: None
Events: <none>
创建Ingress绑定后端nginx服务
[root@k8s-master ingress]# cat ingress-nginx.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-http
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: "nginx.mytest.org"
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
#####host和path加双引号
[root@k8s-master ingress]# kubectl apply -f ingress-nginx.yaml
ingress.networking.k8s.io/ingress-http created
###查看相关的服务
[root@k8s-master ingress]# kubectl describe ingress/ingress-http
Name: ingress-http
Namespace: default
Address: 172.16.7.210
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
nginx.mytest.org
/ nginx-service:80 (10.244.2.187:80,10.244.3.230:80)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 4m31s (x3 over 10m) nginx-ingress-controller Scheduled for sync
集群外验证，通过修改hosts
###修改下主机的hosts文件，添加如下解析
172.16.5.194 nginx.mytest.org
###通过查询ingress-nginx的svc，访问80的应用端口是30358
[root@k8s-master ingress]# kubectl get pod,svc -ningress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-sgd8f 0/1 Completed 0 170m
pod/ingress-nginx-admission-patch-z9nck 0/1 Completed 3 170m
pod/ingress-nginx-controller-5696bb69ff-jzs7n 1/1 Running 0 170m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.100.26.234 <none> 80:30358/TCP,443:31117/TCP 170m
service/ingress-nginx-controller-admission ClusterIP 10.100.183.252 <none> 443/TCP 170m
额外：限流的配置
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-http
namespace: default
annotations:
annotations:
# 加上此重写配置
nginx.ingress.kubernetes.io/rewrite-target: /$2
# 限流
nginx.ingress.kubernetes.io/limit-rps: "1"
spec:
ingressClassName: nginx
rules:
- host: "nginx.mytest.org"
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
- host: "myapp.mytest.org"
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: myapp
port:
number: 80
2.2、部署Ingress(DaemonSet的方式)
由于官方原始文件使用的是deployment，replicate 为 1，这样将会在某一台节点上启动对应的nginx-ingress-controller pod。外部流量访问至该节点，由该节点负载分担至内部的service。测试环境考虑防止单点故障，改为DaemonSet然后删掉replicate ，配合亲和性部署在制定节点上启动nginx-ingress-controller pod，确保有多个节点启动nginx-ingress-controller pod，后续将这些节点加入到外部硬件负载均衡组实现高可用性。
修改的字段
三、使用https进行访问
3.1、创建私钥
[root@k8s-master ingress]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
.................+++
................................+++
e is 65537 (0x10001)
3.2、创建证书
[root@k8s-master ingress]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=ShangHai/L=ShangHai/O=DevOps/CN=nginx.mytest.org
[root@k8s-master ingress]# ll
total 84
drwxr-xr-x 2 root root 81 Nov 16 14:12 bak
-rw-r--r-- 1 root root 15455 Nov 16 18:06 deploy.yaml
-rw-r--r-- 1 root root 15655 Nov 16 12:01 deploy.yaml-1.5.1
-rw-r--r-- 1 root root 15760 Nov 15 17:39 deploy.yaml-loadbalance
-rw-r--r-- 1 root root 380 Nov 16 15:31 ingress-1.yaml
-rw-r--r-- 1 root root 408 Nov 16 15:22 ingress-app.yml
-rw-r--r-- 1 root root 310 Nov 16 15:12 ingress-myapp.yml
-rw-r--r-- 1 root root 714 Nov 16 17:37 ingress-nginx.yaml
-rw-r--r-- 1 root root 595 Nov 16 14:13 myapp-deploymet.yml
-rw-r--r-- 1 root root 573 Nov 16 17:35 nginx-deploy.yaml
-rw-r--r-- 1 root root 471 Nov 15 15:19 service-nodeport.yaml
-rw-r--r-- 1 root root 1298 Nov 16 18:31 tls.crt
-rw-r--r-- 1 root root 1679 Nov 16 18:28 tls.key
3.3、创建secret
[root@k8s-master ingress]# kubectl create secret tls nginx-ingress-secret --cert=tls.crt --key=tls.key
secret/nginx-ingress-secret created
[root@k8s-master ingress]# kubectl get secret/nginx-ingress-secret
NAME TYPE DATA AGE
nginx-ingress-secret kubernetes.io/tls 2 25s
[root@k8s-master ingress]# kubectl describe secret/nginx-ingress-secret
Name: nginx-ingress-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1298 bytes
tls.key: 1679 bytes
3.4、创建ingress，使用https协议绑定后端svc服务
[root@k8s-master ingress]# cat ingress-tls-nginx.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-http-tls
namespace: default
annotations:
# 加上此重写配置
nginx.ingress.kubernetes.io/rewrite-target: /$2
# 限流
nginx.ingress.kubernetes.io/limit-rps: "1"
spec:
tls:
- hosts:
- nginx.mytest.org
secretName: nginx-ingress-secret
rules:
- host: "nginx.mytest.org"
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
- host: "myapp.mytest.org"
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: myapp
port:
number: 80
##
[root@k8s-master ingress]# kubectl apply -f ingress-tls-nginx.yaml
ingress.networking.k8s.io/ingress-http-tls created
[root@k8s-master ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-http nginx nginx.mytest.org,myapp.mytest.org 172.16.7.210 80 15h
ingress-http-tls <none> nginx.mytest.org,myapp.mytest.org 80, 443 5s
[root@k8s-master ingress]# kubectl describe ingress/ingress-http-tls
Name: ingress-http-tls
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
nginx-ingress-secret terminates nginx.mytest.org
Rules:
Host Path Backends
---- ---- --------
nginx.mytest.org
/ nginx-service:80 (10.244.2.187:80,10.244.3.230:80)
myapp.mytest.org
/ myapp:80 (10.244.2.184:80,10.244.2.185:80,10.244.3.227:80)
Annotations: nginx.ingress.kubernetes.io/limit-rps: 1
nginx.ingress.kubernetes.io/rewrite-target: /$2
Events: <none>
3.5、验证服务
##查看服务nodeport的https
[root@k8s-master ingress]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.100.26.234 <none> 80:30358/TCP,443:31117/TCP 20h
四、总结
目前，Intgress 只能工作在七层（ingress是一个七层代理必须绑定一个域名进行访问），而 Service 只能工作在四层。所以当你想要在 Kubernetes 里为应用进行 TLS 配置等 HTTP 相关的操作时，都必须通过 Ingress 来进行。
