容器云平台No.8~kubernetes负载均衡之ingress-nginx

Ingress 是什么？

Ingress 公开了从集群外部到集群内服务的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 资源上定义的规则控制。
可以将 Ingress 配置为服务提供外部可访问的 URL、负载均衡流量、终止 SSL/TLS，以及提供基于名称的虚拟主机等能力。 Ingress 控制器 通常负责通过负载均衡器来实现 Ingress，尽管它也可以配置边缘路由器或其他前端来帮助处理流量。

本文使用host network模式，示意图如下

下载部署文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/baremetal/deploy.yaml

修改images为国内仓库

k8s.gcr.io/ingress-nginx/controller:v0.35.0@sha256:fc4979d8b8443a831c9789b5155cded454cb7de737a8b727bc2ba0106d2eae8b

修改为，也可以自行使用魔法到k8s.gcr.io下载

scofield/ingress-nginx-controller:v0.35.0

修改网络模式为host network

template:
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

执行部署

kubectl apply -f  deploy.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get po -n ingress-nginx
[root@k8s-master001 ingress-nginx]# kubectl  get po,svc  -n ingress-nginx -o wide
NAME                                            READY   STATUS      RESTARTS   AGE     IP             NODE            NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-dfg8g        0/1     Completed   0          47m     10.244.2.155   k8s-master003   <none>           <none>
pod/ingress-nginx-admission-patch-cfl4r         0/1     Completed   1          47m     10.244.1.134   k8s-master002   <none>           <none>
pod/ingress-nginx-controller-6fdd8c7f88-5gzdv   1/1     Running     0          2m42s   10.26.25.21    k8s-master002   <none>           <none>

注意：ingress-nginx-controller的IP应该是宿主机IP，这里是10.26.25.21,至此，ingress-nginx就部署好了

使用ingress-nginx暴露http服务

部署一个最常用的http服务nginx,使用ingress-nginx暴露http服务
1、编写demo.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nginx
labels:
app: nginx
spec:
serviceName: nginx
replicas: 1
selector:
m
5b28
atchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 180
initContainers:
- name: init
image: busybox
command: ["chmod","777","-R","/var/www"]
imagePullPolicy: Always
volumeMounts:
- name: volume
mountPath: /var/www/html
containers:
- name: nginx
image: nginx
imagePullPolicy: Always
ports:
- containerPort: 80
name: port
volumeMounts:
- name: volume
mountPath: /var/www/html
volumeClaimTemplates:
- metadata:
name: volume
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rook-ceph
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
selector:
app: nginx

2、执行部署

[root@k8s-master001 ingress-nginx]# kubectl apply -f  demo.yaml
statefulset.apps/nginx configured
service/nginx created

[root@k8s-master001 ~]# kubectl get po,svc
NAME          READY   STATUS    RESTARTS   AGE
pod/nginx-0   1/1     Running   0          21m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        5d21h
service/nginx        NodePort    10.106.146.193   <none>        80:31389/TCP   21m

[root@k8s-master001 ~]# curl  -I  10.106.146.193
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 16 Sep 2020 07:03:26 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
Connection: keep-alive
ETag: "5f32b03b-264"
Accept-Ranges: bytes

3、nginx已经部署好，而且访问已经OK，接下来创建Ingress
demo-ingress.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: nginx.text.cn
http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80

[root@k8s-master001 ~]# kubectl  apply -f nginx-ingress.yaml
error: error validating "nginx-ingress.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false

修改apiVersion为networking.k8s.io/v1

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: nginx.text.cn
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: nginx
port:
number: 80

[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\"},\"name\":\"nginx\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"nginx.ieasou.cn\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"nginx\",\"port\":{\"number\":80}}},\"path\":\"/\",\"pathType\":\"Prefix\"}]}}]}}\n","kubernetes.io/ingress.class":"nginx"}},"spec":{"rules":[{"host":"nginx.ieasou.cn","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
to:
Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"
Name: "nginx", Namespace: "default"
for: "demo-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate is valid for k8s-master002, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not ingress-nginx-controller-admission.ingress-nginx.svc

还是不行。。。

解决办法1、把Webhook删了

admission webhook 传送门

[root@k8s-master001 ingress-nginx]# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted

再来

[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml
ingress.networking.k8s.io/nginx configured

解决办法2、降级为0.32.0（未测）

传送门了解更多

4、现在来查看创建好的ingress,已经创建好了

[root@k8s-master001 ingress-nginx]# kubectl  get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE
nginx   <none>   nginx.ieasou.cn   10.26.25.21   80      3d19h

5、访问验证,现在在外部就可以通过域名nginx.text.cn访问到nginx了

[root@k8s-master001 ingress-nginx]# vim /etc/hosts
10.26.25.21 nginx.text.cn

[root@k8s-master001 ingress-nginx]# curl -I   nginx.text.cn
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 16 Sep 2020 08:05:06 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
ETag: "5f32b03b-264"
Accept-Ranges: bytes

使用ingress-nginx暴露TCP服务

部署一个常用的redis服务,使用ingress-nginx暴露tcp服务
1、编写redis.yaml文件

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
labels:
app: redis
spec:
serviceName: redis
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
terminationGracePeriodSeconds: 180
initContainers:
- name: init
image: busybox
command: ["chmod","777","-R","/var/www"]
imagePullPolicy: Always
volumeMounts:
- name: volume
mountPath: /data
containers:
- name: redis
image: redis
imagePullPolicy: Always
ports:
- containerPort: 6379
name: port
volumeMounts:
- name: volume
mountPath: /data
volumeClaimTemplates:
- metadata:
name: volume
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rook-ceph
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: redis
labels:
app: redis
spec:
type: NodePort
ports:
- port: 6379
targetPort: 6379
selector:
app: redis

查看并查看结果

[root@k8s-master001 ingress-nginx]# kubectl apply -f redis.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get po,svc
NAME          READY   STATUS    RESTARTS   AGE
pod/redis-0   1/1     Running   0          104s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/redis        NodePort    10.98.28.146     <none>        80:32193/TCP   104s

测试连接redis，能够连接
[root@k8s-master001 ingress-nginx]# telnet 10.26.25.20 32193
Trying 10.26.25.20...
Connected to 10.26.25.20.
Escape character is '^]'.
info
$3615
# Server
redis_version:6.0.8

2、默认ingress-nginx的deploy.yaml部署文件并没有开启tcp服务支持，这里需要修改部署文件并重新部署。一般情形，如果需要支持tcp,udp等转发，提前规划并修改deploy.yaml文件
修改如下：--tcp-services-configmap,这里顺便把udp也开启了--udp-services-configmap

containers:
- name: controller
image: scofield/ingress-nginx-controller:v0.35.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-service
- --udp-services-configmap=$(POD_NAMESPACE)/udp-service
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key

重新部署ingress-nginx

kubectl apply -f deploy.yaml

3、创建tcp服务需要的configmap,注意namespace一定要和ingress-nginx部署的namespace一致
tcp-service.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-service
namespace: ingress-nginx
data:
6379: "default/redis:6379"

[root@k8s-master001 ingress-nginx]# kubectl  apply -f tcp-service.yaml
configmap/tcp-service created

4、创建redis-ingress.yaml文件

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: redis
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: redis.test.cn
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: redis
port:
number: 6379

5、创建ingress

kubectl apply -f redis-ingress.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get ing
NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE
redis   <none>   redis.ieasou.cn   10.26.25.20   80      13m

6、验证

[root@k8s-master001 ingress-nginx]# telnet redis.test.cn 6379
Trying 10.26.25.20...
Connected to redis.test.cn.
Escape character is '^]'.
info
$3622
# Server
redis_version:6.0.8
redis_git_sha1:00000000

更多用法，请移步

注：文中图片来源于网络，如有侵权，请联系我及时删除。