k8s TLS bootstrap解析-k8s TLS bootstrap流程分析
k8s TLS bootstrap解析-k8s TLS bootstrap流程分析
概述
当k8s集群开启了TLS认证后,每个节点的kubelet组件都要使用由kube-apiserver的CA签发的有效证书才能与kube-apiserver通信;当节点非常多的时候,为每个节点都单独签署证书是一件非常繁琐而又耗时的事情。
此时k8s TLS bootstrap功能应运而生。
k8s TLS bootstrap功能就是让kubelet先使用一个预先商定好的低权限的bootstrap token连接到kube-apiserver,向kube-apiserver申请证书,然后kube-controller-manager给kubelet动态签署证书,后续kubelet都将通过动态签署的证书与kube-apiserver通信。
TLS bootstrap涉及组件相关参数
1.kube-apiserver
(1)
--client-ca-file:认证客户端证书的CA证书;
(2)
--enable-bootstrap-token-auth:设置为true则代表开启TLS bootstrap特性;
2.kube-controller-manager
(1)
--cluster-signing-cert-file、
--cluster-signing-key-file:用来签发kubelet证书的CA证书和私钥,这里的kubelet证书指的是用来跟kube-apiserver通信,kube-apiserver认证kubelet身份的证书,所以--cluster-signing-cert-file指定的值与kube-apiserver的--client-ca-file指定值一致,而私钥则也是对应的私钥;
(2)
--cluster-signing-duration:签发给kubelet的证书有效期;
3.kubelet
(1)
--bootstrap-kubeconfig:TLS bootstrap的配置文件,文件中一般包含bootstrap token和master url等信息;
(2)
--kubeconfig:在kubelet的CSR被批复并被kubelet取回时,一个引用所生成的密钥和所获得证书的kubeconfig文件会被写入到通过 --kubeconfig所指定的文件路径下,而证书和密钥文件会被放到--cert-dir所指定的目录中;
(3)
--rotate-certificates:开启证书轮换,kubelet在其现有证书即将过期时通过创建新的CSR来轮换其客户端证书。
详细流程解析
下面以kubeadm使用k8s TLS bootstrap将一个node节点加入已有的master为例,对TLS bootstrap部分进行详细流程解析。
1.RBAC相关操作
(1)生成bootstrap token,创建bootstrap token secret;
bootstrap token secret模板:
apiVersion: v1 data: auth-extra-groups: system:bootstrappers:kubeadm:default-node-token expiration: 2022-04-03T11:13:09+08:00 token-id: {token-id} token-secret: {token-secret} usage-bootstrap-authentication: "true" usage-bootstrap-signing: "true" kind: Secret metadata: name: bootstrap-token-{token-id} namespace: kube-system type: bootstrap.kubernetes.io/token
关于bootstrap token secret相关的格式说明:
secret的name格式为
bootstrap-token-{token-id}的格式;
secret的type固定为
bootstrap.kubernetes.io/token;
secret data中的token-id为6位数字字母组合字符串,token-secret为16位数字字母组合字符串;
secret data中的
auth-extra-groups定义了bootstrap token所代表用户所属的的group,kubeadm使用了
system:bootstrappers:kubeadm:default-node-token;
secret所对应的bootstrap token为
{token-id}.{token-secret};
bootstrap token secret示例:
apiVersion: v1 data: auth-extra-groups: system:bootstrappers:kubeadm:default-node-token expiration: 2022-04-03T11:13:09+08:00 token-id: abcdef token-secret: 0123456789abcdef usage-bootstrap-authentication: "true" usage-bootstrap-signing: "true" kind: Secret metadata: name: bootstrap-token-abcdef namespace: kube-system type: bootstrap.kubernetes.io/token
上述secret示例中,kubeadm生成的bootstrap token为
abcdef.0123456789abcdef,其代表的用户所在用户组为
system:bootstrappers:kubeadm:default-node-token;
(2)授予bootstrap token创建CSR证书签名请求的权限,即授予kubelet创建CSR证书签名请求的权限;
即创建ClusterRoleBinding -- kubeadm:kubelet-bootstrap
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:kubelet-bootstrap ... roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-bootstrapper subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:kubeadm:default-node-token
kubeadm生成的bootstrap token所代表的用户所在用户组为
system:bootstrappers:kubeadm:default-node-token,所以这里绑定权限的时候将权限绑定给了用户组
system:bootstrappers:kubeadm:default-node-token;
接下来看下被授予的权限ClusterRole -- system:node-bootstrapper
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:node-bootstrapper ... rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests verbs: - create - get - list - watch
(3)授予bootstrap token权限,让kube-controller-manager可以自动审批其发起的CSR;
即创建ClusterRoleBinding -- kubeadm:node-autoapprove-bootstrap
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:node-autoapprove-bootstrap ... roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:kubeadm:default-node-token
kubeadm生成的bootstrap token所代表的用户所在用户组为
system:bootstrappers:kubeadm:default-node-token,所以这里绑定权限的时候将权限绑定给了用户组
system:bootstrappers:kubeadm:default-node-token;
接下来看下被授予的权限ClusterRole -- system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:certificates.k8s.io:certificatesigningrequests:nodeclient ... rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/nodeclient verbs: - create
(4)授予kubelet权限,让kube-controller-manager自动批复kubelet的证书轮换请求;
即创建ClusterRoleBinding -- kubeadm:node-autoapprove-certificate-rotation
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:node-autoapprove-certificate-rotation ... roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes
kubelet创建的CSR用户名格式为
system:node:<name>,用户组为
system:nodes,所以kube-controller-manager为kubelet生成的证书所代表的用户所在用户组为
system:nodes,所以这里绑定权限的时候将权限绑定给了用户组
system:nodes;
接下来看下被授予的权限ClusterRole -- system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient ... rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/selfnodeclient verbs: - create
2.启动kubelet,开始TLS bootstrap
(0)根据bootstrap token以及master url等信息生成bootstrap-kubeconfig文件;
(1)启动kubelet,配置了kubeconfig文件目录,但kubeconfig文件为空,再指定bootstrap-kubeconfig文件为上述步骤生成的bootstrap-kubeconfig文件;
(2)kubelet发现配置的kubeconfig文件为空,则加载bootstrap-kubeconfig文件,读取其中的bootstrap token以及master url;
(3)kubelet使用bootstrap token与apiserver通信,创建CSR证书签名请求;
(4)kube-controller-manager批复CSR证书签名请求,为其签发相关证书;
(5)kubelet取回kube-controller-manager生成的相关证书,默认存放在/var/lib/kubelet/pki 目录下,然后根据证书生成kubeconfig文件,后续kubelet将使用该kubeconfig文件与kube-apiserver进行认证通信;
# ls /var/lib/kubelet/pki/ kubelet-client-2022-03-18-14-29-00.pem kubelet-client-current.pem kubelet.crt kubelet.key
kubelet-client-current.pem是个软链,指向kubelet-client-2022-03-18-14-29-00.pem文件,kubelet-client-2022-03-18-14-29-00.pem文件名记录的是证书创建时间,后续kubelet将会根据证书过期时间,在证书临过期前向kube-apiserver重新申请证书,然后自动轮换该证书;
# cat /etc/kubernetes/kubelet.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... server: https://192.168.1.10:6443 name: test-cluster contexts: - context: cluster: test-cluster user: system:node:test-cluster-node-1 name: system:node:test-cluster-node-1 current-context: system:node:test-cluster-node-1 kind: Config preferences: {} users: - name: system:node:test-cluster-node-1 user: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
3.kubelet自动轮换证书
(1)kubelet在证书接近于过期时自动向kube-apiserver请求更新证书;
(2)kube-controller-manager自动批复,为其签发新的证书;
(3)kubelet取回kube-controller-manager生成的相关证书,替换掉本地的旧证书,后续kubelet将使用新证书来与kube-apiserver进行认证通信;
总结
最后以一幅图来总结一下k8s TLS bootstrap的整个流程。
- mybatis源码解析 - 核心流程分析之编程模型构建
- MapReduce源码分析之作业Job状态机解析(一)简介与正常流程浅析
- hadoop源码解析之hdfs读取数据全流程分析
- Android程序包管理机制解析和PMS启动流程分析
- ARM9启动分析--存储器区分和启动流程解析
- uboot源码分析(1)uboot 命令解析流程简析
- Android恢复出厂设置流程分析【Android源码解析十】
- 24. SpringMVC_视图解析流程分析
- Hive SQL解析/执行计划生成流程分析
- Spring源码分析3 — spring XML配置文件的解析流程
- ibatis源码分析—运行流程解析(二)
- Chromium源码--视频播放流程分析(WebMediaPlayerImpl解析)
- Nutch学习笔记3:Nutch 1.7 版本 之 HtmlParser 解析流程分析
- BI系统中的分析流程解析与示例
- Android开机流程分析 -- init进程之配置文件解析
- [Django架构流程分析]请求处理机制其二:Django中间件的解析
- Kubernetes编排工具-helm源码分析(模板解析流程)
- Android启动流程分析(九) 解析init.rc的service
- linux引导流程解析与分析
- 开源工作流Fireflow源码分析之流程文件解析