Kubernetes之服务发现ingress & ingress controller
2018-10-04 21:21
639 查看
Service虽然解决了服务发现和负载均衡的问题，但它在使用上还是有一些限制：
- 只支持4层负载均衡，没有7层功能
- 对外访问时，NodePort类型需要在外部搭建额外的负载均衡，而LoadBalancer要求kubernetes必须跑在支持的cloud provider上
一、ingress引入历程
我们已经了解到Kubernetes暴露服务的方式目前只有三种：LoadBalancer Service、ExternalName、NodePort Service；那还有没有更好的方式呢？我们先来了解一下以下几个问题：
1、Pod漂移问题
我们知道Pod是有生命周期的，Pod宕机之后，其IP必然会改变，那我们如何把这个动态的Pod IP暴露出去呢？总所周知，我们可以借助Kubernetes的Service机制，在Service中以标签的形式选定一组带有指定标签的Pod，并监控和自动负载到它们的Pod IP，那么我们向外只需要暴露Service IP就行了，这就是我们常使用的NodePort模式：即在每个节点上开启一个端口，然后转发到内部的Pod IP上。此时的访问方式为：http: //nodeip:nodeport/
2、端口管理问题
采用NodePort方式暴露服务面临的问题是：服务一旦躲起来，NodePort在每个节点上开启的端口会极其庞大，而且难以维护。想一想，鉴于nginx的能力，我们能够使用一个nginx直接对内进行转发呢？由于Pod与Pod之间是可以互相通信的，而Pod是可以共享宿主机的网络名称空间的，此时Pod上所监听的就是Node上的端口。那这又该如何实现呢？简单的实现就是使用DaemonSet在每个Node上监听80端口，然后写好规则，因为nginx外面绑定了宿主机80端口(就像NodePort)，本身又在集群内，那么向后直接转发到相应Service IP就行了
3. 域名分配及动态更新问题
用上面的方法，我们采用nginx Pod似乎已经解决了问题，但是当每次有新服务加入又该如何修改nginx配置呢？nginx可以通过虚拟主机域名区分不同的服务，而每个服务通过upstream进行定义不同的负载均衡池，再加上location进行负载均衡的反向代理，在日常使用中只需要修改nginx.conf即可实现，但是在k8s中又该如何实现这种方式调度呢？
假设后端的服务开始只有ecshop，后面增加了bbs和member服务，那么又该如何将这2个服务加入到nginx Pod进行调度呢？总不能每次手动改后者Rolling update前端nginx pod吧！所以k8s引入了ingress，ingress就是为了解决这些问题而生的，ingress包含两大组件：ingress controller和ingress。
- ingress
简单理解就是你原来需要修改nginx配置，然后配置各种域名对应哪个Service，现在把这个动作抽象出来，变成一个Ingress对象，你可以用yaml创建，每次不要去修改nginx了，直接改yaml然后创建/更新就行了，那么问题又来了：nginx该如何处理呢？ - ingress controller
这个东西就是解决nginx如何处理这个问题的，ingress controller通过与kubernetes API交互，动态的去感知集群中Ingress规则变化，然后读取它，按照它自己的模板生成一段nginx配置，再写到nginx Pod中，最后reload以下，工作流程如下图
图表说明：
首先我们有一个外部的负载均衡器externalLB把请求调度到一个nodePort类型的Service(ingress-nginx)上，然后nodePort类型的Service(ingress-nginx)又把它调度到内部的叫做ingressController的Pod上，ingressCtroller根据ingress中的定义(虚拟主机还是URL)，每一组主机名或者URL都对应后端的Pod资源，并且用Service分组(图中的<service>site1/<service>site2只是用来做分组用的)
二、Ingress Nginx部署
使用Ingress功能步骤：
-
安装部署ingressCtroller Pod
-
部署ingress-nginx service
-
部署后端服务
-
部署ingress
从前面的描述我们知道，Ingress可以使用yaml的方式进行创建，从而得知Ingress也是标准的K8S资源，其定义的方式也可以使用kubectl explain
进行查看：[root@master manifests]# kubectl explain ingress KIND: Ingress VERSION: extensions/v1beta1 DESCRIPTION: Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend. An Ingress can be configured to give services externally-reachable urls, load balance traffic, terminate SSL, offer name based virtual hosting etc. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds metadata <Object> Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata spec <Object> Spec is the desired state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status status <Object> Status is the current state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
2.1. 部署ingress controller
2.1.1. 下载ingress相关yaml文件
[root@master ~]# mkdir ingress-nginx
[root@master ~]# cd ingress-nginx/
[root@master ingress-nginx]# for filr in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml udp-services-configmap.yaml default-backend.yaml;do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file;done
[root@master ingress-nginx]# ls -l | awk '{print $9}'
configmap.yaml #用于为nginx从外部注入配置的
default-backend.yaml #配置默认后端服务
namespace.yaml #创建独立的名称空间
rbac.yaml #用于集群角色授权
tcp-services-configmap.yaml
udp-services-configmap.yaml
with-rbac.yaml
2.1.2. 创建ingress-nginx名称空间
[root@k8s-master ingress-nginx]# cat namespace.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx --- [root@k8s-master ingress-nginx]# kubectl apply -f namespace.yaml namespace/ingress-nginx created
2.1.3. 创建ingress controller的Pod
[root@master ingress-nginx]# kubectl apply -f . configmap/nginx-configuration created deployment.extensions/default-http-backend created service/default-http-backend created namespace/ingress-nginx configured serviceaccount/nginx-ingress-serviceaccount created clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole configured role.rbac.authorization.k8s.io/nginx-ingress-role created rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding configured configmap/tcp-services created configmap/udp-services created deployment.extensions/nginx-ingress-controller created
此处遇到一个问题，新版本的Kubernetes在安装部署中，需要从k8s.grc.io仓库拉去所需镜像文件，但是由于国内防火墙问题导致无法正常拉去。
辛亏docker.io仓库对google的容器做了mirror，可以通过下列命令拉取相关镜像：
[root@node01 ~]# docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5 1.5: Pulling from mirrorgooglecontainers/defaultbackend-amd64 9ecb1e82bb4a: Pull complete Digest: sha256:d08e129315e2dd093abfc16283cee19eabc18ae6b7cb8c2e26cc26888c6fc56a Status: Downloaded newer image for mirrorgooglecontainers/defaultbackend-amd64:1.5 [root@node01 ~]# docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5 [root@node01 ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE mirrorgooglecontainers/defaultbackend-amd64 1.5 b5af743e5984 34 hours ago 5.13MB k8s.gcr.io/defaultbackend-amd64 1.5 b5af743e5984 34 hours ago 5.13MB
2.2. 部署ingress-nginx service
通过ingress-controller对外提供服务，现在还需要手动给ingress-controller建立一个servcie，接收集群外部流量。
2.2.1 下载ingress-controller service的yaml文件
[root@master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml [root@master ingress-nginx]# cat service-nodeport.yaml apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ---
2.2.2. 创建ingress-controller的service
[root@master ingress-nginx]# kubectl apply -f service-nodeport.yaml service/ingress-nginx created [root@master ingress-nginx]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.107.114.219 <none> 80/TCP 27m ingress-nginx NodePort 10.108.12.235 <none> 80:31875/TCP,443:32020/TCP 38s
2.3. 部署ingress
2.3.1. 查看ingress的配置清单选项
[root@master ingress-nginx]# kubectl explain ingress.spec KIND: Ingress VERSION: extensions/v1beta1 RESOURCE: spec <Object> DESCRIPTION: Spec is the desired state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status IngressSpec describes the Ingress the user wishes to exist. FIELDS: backend <Object> #定义后端有哪几个主机 A default backend capable of servicing requests that don't match any rule. At least one of 'backend' or 'rules' must be specified. This field is optional to allow the loadbalancer controller or defaulting logic to specify a global default. rules <[]Object> #定义规则 A list of host rules used to configure the Ingress. If unspecified, or no rule matches, all traffic is sent to the default backend. tls <[]Object> TLS configuration. Currently the Ingress only supports a single TLS port, 443. If multiple members of this list specify different hosts, they will be multiplexed on the same port according to the hostname specified through the SNI TLS extension, if the ingress controller fulfilling the ingress supports SNI.
2.3.2. 编写ingress的配置清单并创建ingress
[root@master ingress-nginx]# vim ingress-default-backend.yaml apiVersion: extensions/v1beta1 #api版本 kind: Ingress #清单类型 metadata: #元数据 name: ingress-default-backend #ingress名称 namespace: ingress-nginx #所属命名空间 annotations: #注解信息 kubernetes.io/ingress.class: "nginx" spec: #规格 rules: #定义后端转发规则 - host: default.backend.com #通过域名进行转发 http: paths: - path: #配置访问路径 backend: #配置后端服务 serviceName: default-http-backend servicePort: 80 [root@master ingress-nginx]# kubectl apply -f ingress-default-backend.yaml ingress.extensions/ingress-default-backend created
查看ingress-default-backend的详细信息：
[root@master ingress-nginx]# kubectl describe ingress -n ingress-nginx ingress-default-backend
Name: ingress-default-backend
Namespace: ingress-nginx
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
default.backend.com
default-http-backend:80 (<none>)
Annotations:
kubernetes.io/ingress.class: nginx
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 9m nginx-ingress-controller Ingress ingress-nginx/ingress-default-backend
进入nginx-ingress-controller进行查看是否注入了nginx配置：
[root@master ingress-nginx]# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-7db7c45b69-znpxp 1/1 Running 0 1h
nginx-ingress-controller-6bd7c597cb-fgdz8 1/1 Running 0 1h
[root@master ingress-nginx]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-6bd7c597cb-fgdz8 -- /bin/sh
$ cat nginx.conf
....
....
## start server default.backend.com
server {
server_name default.backend.com ;
listen 80;
set $proxy_upstream_name "-";
location / {
set $namespace "ingress-nginx";
set $ingress_name "ingress-default-backend";
set $service_name "default-http-backend";
set $service_port "80";
set $location_path "/";
rewrite_by_lua_block {
balancer.rewrite()
}
log_by_lua_block {
balancer.log()
monitor.call()
}
port_in_redirect off;
set $proxy_upstream_name "ingress-nginx-default-http-backend-80";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_tries 3;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
}
## end server default.backend.com
....
....
可以发现我们刚刚部署的ingress的的确确已经注入到nginx-ingress-controller的nginx配置中了
2.3.3. 通过ingress访问服务
修改本地host文件：
192.168.116.128 default.backend.com 192.168.116.129 default.backend.com
三、构建TLS站点
3.1. 添加tomcat服务
[root@master ingress-nginx]# vim tomcat-demo.yaml apiVersion: v1 kind: Service metadata: name: tomcat namespace: default spec: selector: app: tomcat release: stable ports: - name: http port: 8080 targetPort: 8080 - name: ajp port: 8009 targetPort: 8009 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-demo namespace: default spec: replicas: 3 selector: matchLabels: app: tomcat release: stable template: metadata: labels: app: tomcat release: stable spec: containers: - name: tomcat image: tomcat:8.5.34-jre8-alpine ports: - name: http containerPort: 8080 - name: ajp containerPort: 8009 [root@master ingress-nginx]# kubectl create -f tomcat-demo.yaml service/tomcat created deployment.apps/tomcat-demo created [root@master ingress-nginx]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 12d tomcat ClusterIP 10.108.64.162 <none> 8080/TCP,8009/TCP 8s [root@master ingress-nginx]# kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE tomcat-demo 3 3 3 0 16s [root@master ingress-nginx]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-demo-798cdf69b9-qk9pd 1/1 Running 0 7m tomcat-demo-798cdf69b9-t642t 1/1 Running 0 7m tomcat-demo-798cdf69b9-tlfq6 1/1 Running 0 7m
3.2. 准备证书并生成secret
准备证书：
[root@master ingress-nginx]# openssl genrsa -out tls.key 2048 Generating RSA private key, 2048 bit long modulus ............................+++ ..................+++ e is 65537 (0x10001) [root@master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.zwx.com
生成secret：
[root@master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key secret/tomcat-ingress-secret created [root@master ingress-nginx]# kubectl get secret NAME TYPE DATA AGE default-token-l9jws kubernetes.io/service-account-token 3 12d tomcat-ingress-secret kubernetes.io/tls 2 33s [root@master ingress-nginx]# kubectl describe secret tomcat-ingress-secret Name: tomcat-ingress-secret Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 1285 bytes tls.key: 1679 bytes
3.3. 创建ingress
[root@master ingress-nginx]# vim ingress-tomcat-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat-tls namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - tomcat.zwx.com secretName: tomcat-ingress-secret rules: - host: tomcat.zwx.com http: paths: - backend: serviceName: tomcat servicePort: 8080 [root@master ingress-nginx]# kubectl apply -f ingress-tomcat-tls.yaml ingress.extensions/ingress-tomcat-tls created
3.4. 访问测试
查看ingress-nginx 服务
[root@master ingress-nginx]# kubectl get svc -n ingress-nginx -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default-http-backend ClusterIP 10.107.114.219 <none> 80/TCP 4h app.kubernetes.io/name=default-http-backend,app.kubernetes.io/part-of=ingress-nginx ingress-nginx NodePort 10.108.12.235 <none> 80:31875/TCP,443:32020/TCP 3h app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
所以访问https://tomcat.zwx.com:32020
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Kubernetes - 配置Nginx-Ingress 作为服务发现
- kubernetes集群中使用ingress发布服务
- Kubernetes如何使用kube-dns实现服务发现
- kubernetes 简介：kube-dns 和服务发现
- 关于kubernetes 的 服务发现
- kubernetes最佳实践(四) - nginx-ingress-controller
- SpringCloud | Docker 学习系列 | Kubernetes 学习 将SpringCloud Eureka 服务注册与发现部署到docker中并放入到Kubernetes中管理
- kubernetes 服务发现和负载均衡
- Kubernetes(k8s)如何使用kube-dns实现服务发现
- kubernetes 服务发现与注册个人想法
- [Kubernetes] Kubernetes 集成 traefik ingress 实现 服务暴露
- Kubernetes如何使用kube-dns实现服务发现
- kubernetes学习记录（5）——服务发现机制与Cluster DNS的安装(无CA认证版)
- kubernetes 服务发现实现（kube-dns）
- Kubernetes(k8s)如何使用kube-dns实现服务发现
- 【Kubernetes】最佳实践1：kube-proxy与服务发现
- Kubernetes如何使用kube-dns实现服务发现
- Kubernetes Nginx Ingress Controller源码分析
- Docker自发现注册服务regd研发
- 【转】为什么不应该使用ZooKeeper做服务发现