您的位置：首页 > 其它

kubernetes 1.8.7 离线部署手册(一)

2018-02-09 13:34 399 查看
搭建kubernetes集群最大的麻烦其实不在于其复杂度(相对Openstack集群)而在于有GFW, 所以为了避免墙带来的麻烦, 也为了加深对kubernetes的理解, 这里将使用纯手工离线的方式进行部署(相应的文件我已经下载到百度云盘里面)。

环境介绍

下面Kubernetes集群搭建需要的版本信息:

软件版本
OSCentOS Linux release 7.3.1611 (Core)
Kubernetes1.8.7
Docker18.01.0-ce
Etcd3.2.7
我们将在四台CentOS系统的物理机上部署一个4个节点的kubernetes1.8.7集群(截止当前2018.1是最新版本):

IPRoleCPUMemory
192.168.204.3master/node2核4G
192.168.204.4node2核4G
192.168.204.6node2核4G
192.168.204.7node2核4G
这里master节点同时充当node节点, 因为后面一些在部署calico网络时采用的是daemonSet, 这样才能保证master和node节点内部网络通畅。

镜像准备

在进行集群搭建时, 由于很多镜像都需要翻墙下载, 因此请先准备一个私有仓库, 关于私有仓库的搭建请参考: {% post_link harbor %}, 如果还没搭建的请搭建然后继续.

在后面的安装过程中需要依赖一个基础镜像: pause, 它是根POD镜像, 用于管理POD的网络, 存储等一些共享资源:

gcr.io/google_containers/pause-amd64:3.0


如果自己能翻墙就自己下载后推送到自己的私有仓库, 如果自己翻不了墙, 我已帮忙下载, 请到国内kubernetes镜像tar包

docker load < pause-amd64.tar
docker tag gcr.io/google_containers/pause-amd64:3.0 {PRIVATE_REGISTRIY_ADDR}/{PROJECT}/pause-amd64:3.0
docker push {PRIVATE_REGISTRIY_ADDR}/{PROJECT}/pause-amd64:3.0


集群准备

在开始搭建集群前需要做一些基础准备:

同步集群的时间

理论上systemd系统都自带了时间同步和管理的工具, 使用timedatectl命令确认下, 确保NTP synchronized为yes:

[root@k8s-apiserver ~]# timedatectl
Local time: 六 2018-01-20 02:47:59 EST
Universal time: 六 2018-01-20 07:47:59 UTC
RTC time: 六 2018-01-20 07:47:59
Time zone: America/New_York (EST, -0500)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: no
Last DST change: DST ended at
日 2017-11-05 01:59:59 EDT
日 2017-11-05 01:00:00 EST
Next DST change: DST begins (the clock jumps one hour forward) at
日 2018-03-11 01:59:59 EST
日 2018-03-11 03:00:00 EDT


关闭SELinux

SELinux是系统上的沙盒机制, 为了尽快搭建出集群, 先关闭, 如果强调高安全, 可以等集群搭建成功后开启

# 确认SELinux的配置, 如果不是disabled, 请设置成disabled, 然后从启系统
[root@k8s-apiserver ~]# cat /etc/selinux/config  | grep ^SELINUX=
SELINUX=disabled
# 查看当前SELinux是否已经成功关闭
[root@k8s-apiserver ~]# getenforce
Disabled


关闭防火墙(搭建成功后可以慢慢开启)

systemd系统一般都采用firewalld作为防火墙, 同理如果没关闭 先关闭

[root@k8s-apiserver ~]# systemctl is-enabled firewalld
disabled
# 如果未关闭请执行
[root@k8s-apiserver ~]# systemctl stop  firewalld
[root@k8s-apiserver ~]# systemctl disable  firewalld
[root@k8s-apiserver ~]# systemctl is-enabled firewalld
disabled


集群部署

接下来进入集群的部署阶段, 部署分为如下几个阶段:

CA和证书准备

部署kubernetes客户端工具kubectl

node节点的TLS证书引导(TLS Bootstrap)配置准备

部署ETCD

部署master节点

部署node节点

集群测试

在进入部署之前, 先看看kubernetes组件的架构:



Kubernetes主要由以下几个核心组件组成：

etcd保存了整个集群的状态；

apiserver提供了资源操作的唯一入口，并提供认证、授权、访问控制、API注册和发现等机制；

controller manager负责维护集群的状态，比如故障检测、自动扩展、滚动更新等；

scheduler负责资源的调度，按照预定的调度策略将Pod调度到相应的机器上；

kubelet负责维持容器的生命周期，同时也负责Volume（CVI）和网络（CNI）的管理；

Container runtime负责镜像管理以及Pod和容器的真正运行（CRI）；

kube-proxy负责为Service提供cluster内部的服务发现和负载均衡；

CA和证书准备

kubernetes是一套分布式系统, 系统的各组件均使用TLS来进行身份的双向确认和通信加密, 本文档使用CloudFlare提供的PKI工具集 cfssl来生成Certificate Authority (CA)和管理证书(openssl已可以, 但是你需要一步一步的填写).

我们需要为每一个kubernetes的组件生成证书, 总结起来如下:

kubectl: 用户的CLI工具, 需要为用户生成访问的证书, 这里需要为admin用户颁发1张证书

master: apiserver, controller manager, scheduler, etcd 这4个服务都在master节点上, 因此为他们颁发1张证书

node: kubelet, kube-proxy 由于kubelet的证书是动态颁发的(TLS Bootstra), 因此这里仅需要为kube-proxy颁发1张证书

安装cfssl工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo


创建CA

首先我们创建一个目录pki, 后面的操作都在这个目录里面进行

mkdir pki && cd pki


首先创建CA的配置文件: ca-config.json

# 过期时间设置成了 87600h
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF


关于CA配置文件里面的参数说明:

ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；

signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；

server auth：表示client可以用该 CA 对server提供的证书进行验证；

client auth：表示server可以用该CA对client提供的证书进行验证；

创建CA证书签名请求

cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF


关于证书签名请求的参数说明:

“CN”：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；

“O”：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；

生成CA证书和私钥

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem


admin用户证书

后续kube-apiserver使用RBAC对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；

kube-apiserver 预定义了一些RBAC使用的RoleBindings, 如cluster-admin将Group system:masters与Role cluster-admin 绑定，该Role授予了调用kube-apiserver的所有API的权限, 意思是凡是system:masters Group的user都拥有cluster-admin的角色。 因此我们在使用kubectl命令时候，才拥有整个集群的管理权限(后面部署了客户端工具方可查看)；

因此admin用户证书申请的核心是指定Group: Group system:masters, 这样该用户就有访问APIServer的所有权限的.

签名请求参数如下:

cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF


生成签名证书:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem


master服务证书

cat > kubernetes-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.204.3",
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF


参数说明:

hosts: 如果hosts字段不为空则需要指定授权使用该证书的IP或域名列表，由于该证书后续被etcd集群和kubernetes master集群使用，所以上面分别指定了etcd集群、kubernetes master集群的主机IP和kubernetes服务的服务IP(一般是 kube-apiserver指定的 service-cluster-ip-range 网段的第一个IP，如 10.254.0.1)。

生成签名证书:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kubernetes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem


node服务证书(仅kube-proxy)

kube-apiserver预定义的RoleBinding system:node-proxier 将User system:kube-proxy与Role system:node-proxier绑定，该User具有调用kube-apiserver Proxy相关 API的权限(后面部署了客户端工具后方可查看).

因此证书里面通过CN指定user为: system:kube-proxy, 这该证书就是具有了proxy相应的权限.

cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF


生成证书:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem


分发证书

将上面生成好的证书分发到master和node节点, 作为kubernetes的配置文件:

# 因为我是在master上生成的证书, 仅需cp到master对于目录即可
$ mkdir -pv /etc/kubernetes/pki
mkdir: 已创建目录 "/etc/kubernetes"
mkdir: 已创建目录 "/etc/kubernetes/pki"
$ cp ~/pki/{admin-key.pem,admin.pem,ca-key.pem,ca.pem,kube-proxy-key.pem,kube-proxy.pem,kubernetes-key.pem,kubernetes.pem} /etc/kubernetes/pki/
$ ls /etc/kubernetes/pki/
admin-key.pem  ca-key.pem  kube-proxy-key.pem  kubernetes-key.pem
admin.pem      ca.pem      kube-proxy.pem      kubernetes.pem
# 在node节点上创建目录, 并且copy过去
$ ssh  root@192.168.204.6 'mkdir -pv /etc/kubernetes/pki'
$ scp  ~/pki/{admin-key.pem,admin.pem,ca-key.pem,ca.pem,kube-proxy-key.pem,kube-proxy.pem,kubernetes-key.pem,kubernetes.pem}  root@192.168.204.6:/etc/kubernetes/pki/


部署kubernetes客户端工具kubectl

客户端工具需要下载: kubernetes-client-linux-amd64.tar.gz(1.8.7), 但是由于墙的存在, 有些无法翻墙的朋友请访问我已经下载好的地址国内下载

获取包后进行客户端的安装:

tar -xzvf kubernetes-client-linux-amd64.tar.gz
cp kubernetes/client/bin/kube* /usr/bin/


客户端安装完成后, 需要配置访问凭证, 这里配置证书访问, 证书就是上面生成好的admin的证书

export KUBE_APISERVER="https://192.168.204.3:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER}
# 设置客户端认证参数
kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/pki/admin.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/pki/admin-key.pem
# 设置上下文参数
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin
# 设置默认上下文
kubectl config use-context kubernetes


生成的kubeconfig被保存到~/.kube/config文件, 该文件拥有对该集群的最高权限，请妥善保管。

由于我们集群的master节点还没有部署, 所以后面再进测试。

node节点的TLS证书引导(TLS Bootstrap)配置准备

kubelet、kube-proxy等Node机器上的进程与Master机器的kube-apiserver进程通信时需要认证和授权;

kubernetes 1.4开始支持由kube-apiserver为客户端生成TLS证书的TLS Bootstrapping功能，这样就不需要为每个客户端生成证书了; 该功能当前仅支持为kubelet生成证书;

创建TLS Bootstrapping Token

BOOTSTRAP_TOKEN 将被写入到kube-apiserver使用的token.csv文件和kubelet使用的bootstrap.kubeconfig文件.

export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
$ cat token.csv
fc9702212376b0c73ffc3db3d425227c,kubelet-bootstrap,10001,"system:kubelet-bootstrap"


创建kubelet bootstrapping kubeconfig文件

# 首先确认token环境变量的值
echo $BOOTSTRAP_TOKEN

# 确保KUBE_APISERVER设置正确
export KUBE_APISERVER="https://192.168.204.3:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig


–embed-certs 为 true 时表示将certificate-authority证书写入到生成的bootstrap.kubeconfig文件中;

设置客户端认证参数时没有指定秘钥和证书，后续由kube-apiserver自动生成；

创建kube-proxy kubeconfig文件

export KUBE_APISERVER="https://192.168.204.3:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kube-proxy \
--client-certificate=/etc/kubernetes/pki/kube-proxy.pem \
--client-key=/etc/kubernetes/pki/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig


设置集群参数和客户端认证参数时 –embed-certs 都为 true，这会将 certificate-authority、client-certificate 和 client-key 指向的证书文件内容写入到生成的 kube-proxy.kubeconfig 文件中；

kube-proxy.pem 证书中 CN 为 system:kube-proxy，kube-apiserver 预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；

分发kubeconfig文件

先确认我们刚才生成的文件, 然后将token.csv复制给master节点, bootstrap.kubeconfig和kube-proxy.kubeconfig负责给node节点

[root@k8s-apiserver01 TLS_Bootstrapping]# ls
bootstrap.kubeconfig  kube-proxy.kubeconfig  token.csv
# 因为我在master节点上(注意master同时也充当node节点), 所以
$ cp  token.csv /etc/kubernetes/
$ cp ~/TLS_Bootstrapping/{bootstrap.kubeconfig,kube-proxy.kubeconfig} /etc/kubernetes
$ scp ~/TLS_Bootstrapping/{bootstrap.kubeconfig,kube-proxy.kubeconfig} root@192.168.204.6:/etc/kubernetes/


部署ETCD

etcd作为kubernetes集群的主数据库, 在安装kubernetes集群之前必须先安装和启动。

从etcd官网下载etcd的二进制文件压缩包, 由于不翻墙下载速度会非常慢, 因此我已提前下载到百度网盘

TLS证书确认

需要为etcd集群创建加密通信的TLS证书, 之前创建的kubernetes证书的hosts字段列表中包含etcd部署地址的IP, 否则后续证书校验会失败;

[root@k8s-apiserver01 pki]# ls /etc/kubernetes/pki/
admin-key.pem  ca-key.pem  kube-proxy-key.pem  kubernetes-key.pem
admin.pem      ca.pem      kube-proxy.pem      kubernetes.pem


在部署TLS Etcd时我们将要使用: ca.pem, kubernetes-key.pem, kubernetes.pem

部署与配置ETCD

从上面的百度网盘下载下etcd的二进制包, 然后开始安装:

$ tar vxf etcd-v3.2.7-linux-amd64.tar.gz
$ cp etcd-v3.2.7-linux-amd64/etcd* /usr/local/bin/


安装完成后配置成systemd的系统服务, 生成/usr/lib/systemd/system/etcd.service文件, 文件内容如下:

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
--name ${ETCD_NAME} \
--cert-file=/etc/kubernetes/pki/kubernetes.pem \
--key-file=/etc/kubernetes/pki/kubernetes-key.pem \
--peer-cert-file=/etc/kubernetes/pki/kubernetes.pem \
--peer-key-file=/etc/kubernetes/pki/kubernetes-key.pem \
--trusted-ca-file=/etc/kubernetes/pki/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/pki/ca.pem \
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster infra1=https://192.168.204.3:2380 \
--initial-cluster-state new \
--data-dir=${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


编写etcd服务的配置文件/etc/etcd/etcd.conf, 文件内容如下:

# [member]
ETCD_NAME=infra1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.204.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.204.3:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.204.3:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.204.3:2379"


启动ETCD

# 因为在etcd的配置文件里面我们指定了etcd数据目录为: /var/lib/etcd, 因此我们需要提前建好
$ mkdir /var/lib/etcd
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd


验证ETCD

etcdctl \
--ca-file=/etc/kubernetes/pki/ca.pem \
--cert-file=/etc/kubernetes/pki/kubernetes.pem \
--key-file=/etc/kubernetes/pki/kubernetes-key.pem \
cluster-health
member 74c10a33d24ef135 is healthy: got healthy result from https://192.168.204.3:2379 cluster is health


部署master节点

Master是Kubernetes的大总管，主要由apiserver、controller manager与scheduler组成, 用于管理所有node.

目前这三个组件需要部署在同一台机器上, 注意同时只能有一个kube-scheduler、kube-controller-manager进程处于工作状态，如果运行多个，则需要通过选举产生一个leader

下载最新版本的二进制文件

master的服务需要下载: kubernetes-server-linux-amd64.tar.gz(1.8.7), 但是由于墙的存在, 有些无法翻墙的朋友请访问我已经下载好的文件:国内下载

下载完成后, 安装master服务:

$ tar vxf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes
$ cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/


配置和启动kube-apiserver

service配置文件/usr/lib/systemd/system/kube-apiserver.service内容:

[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


由于master服务: kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy有一部分相同的配置, 因此抽离了1个config来公用这些配置: /etc/kubernetes/config文件的内容为:

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://192.168.204.3:8080"


apiserver配置文件/etc/kubernetes/apiserver内容为:

###
## kubernetes system config
##
## The following values are used to configure the kube-apiserver
##
#
## The address on the local server to listen to.
#KUBE_API_ADDRESS="--insecure-bind-address=sz-pg-oam-docker-test-001.tendcloud.com"
KUBE_API_ADDRESS="--advertise-address=192.168.204.3 --bind-address=192.168.204.3 --insecure-bind-address=192.168.204.3"
#
## The port on the local server to listen on.
#KUBE_API_PORT="--port=8080"
#
## Port minions listen on
#KUBELET_PORT="--kubelet-port=10250"
#
## Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.204.3:2379"
#
## Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
#
## default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota"
#
## Add your own!
KUBE_API_ARGS="--authorization-mode=Node,RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/pki/kubernetes.pem --tls-private-key-file=/etc/kubernetes/pki/kubernetes-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --service-account-key-file=/etc/kubernetes/pki/ca-key.pem --etcd-cafile=/etc/kubernetes/pki/ca.pem --etcd-certfile=/etc/kubernetes/pki/kubernetes.pem --etcd-keyfile=/etc/kubernetes/pki/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h"


启动kube-apiserver

$ systemctl daemon-reload
$ systemctl enable kube-apiserver
$ systemctl start kube-apiserver
$ systemctl status kube-apiserver
$ netstat -tlnup  | grep kube
tcp        0      0 192.168.204.3:6443      0.0.0.0:*               LISTEN      10625/kube-apiserve
tcp        0      0 192.168.204.3:8080      0.0.0.0:*               LISTEN      10625/kube-apiserve


配置和启动kube-controller-manager

创建kube-controller-manager的serivce配置文件: /usr/lib/systemd/system/kube-controller-manager.service, 内容如下:

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/local/bin/kube-controller-manager \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


配置文件/etc/kubernetes/controller-manager内容如下:

###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem  --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem --root-ca-file=/etc/kubernetes/pki/ca.pem --leader-elect=true"


启动服务:

$ systemctl daemon-reload
$ systemctl enable kube-controller-manager
$ systemctl start kube-controller-manager
$ netstat -tlnup | grep controll
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      10692/kube-controll


配置和启动kube-scheduler

创建kube-scheduler的serivce配置文件/usr/lib/systemd/system/kube-scheduler.service:

[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/local/bin/kube-scheduler \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


配置文件/etc/kubernetes/scheduler:

###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1"


启动服务:

$ systemctl daemon-reload
$ systemctl enable kube-scheduler
$ systemctl start kube-scheduler
$ netstat -tlnup | grep schedule
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      10745/kube-schedule


验证master节点功能

$ netstat -tlnup | grep kube
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      10745/kube-schedule
tcp        0      0 192.168.204.3:6443      0.0.0.0:*               LISTEN      10625/kube-apiserve
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      10692/kube-controll
tcp        0      0 192.168.204.3:8080      0.0.0.0:*               LISTEN      10625/kube-apiserve
$ kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}


master安装完成后, 我们就可以额测试kubectl了, 查看上面提到的clusterrolebinding的相关权限:

$ kubectl get clusterrolebinding system:node-proxier -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-01-19T01:23:53Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:node-proxier
resourceVersion: "75"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system%3Anode-proxier
uid: 6916e382-fcb7-11e7-ad2d-fa163e1ab5c5
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-proxier
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:kube-proxy
$ kubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-01-19T01:23:52Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "72"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
uid: 69072637-fcb7-11e7-ad2d-fa163e1ab5c5
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters


部署node节点

Node是主要执行容器实例的节点，可视为工作节点。他负责容器的启停。

注意: 因为kubernetes CNI标准的推出, 容器的网络后面会通过CNI插件单独部署, 所以此时的node节点创建出来的容器是不能夸主机通信的.

Kubernetes node节点包含如下组件：

Docker: cri的实现, docker的安装很简单。

kubelet: 负责维持容器的生命周期，同时也负责Volume（CVI）和网络（CNI）的管理, 通过二进制安装

kube-proxy: 负责为Service提供cluster内部的服务发现和负载均衡, 通过二进制安装

证书与Bootstrap TLS配置文件确认

如果该node节点已经分发了证书, 可以看到如下:

[root@k8s-node01 kubernetes]# ls /etc/kubernetes/pki
admin-key.pem  ca-key.pem  kubelet-client.crt  kubelet.crt  kube-proxy-key.pem  kubernetes-key.pem
admin.pem      ca.pem      kubelet-client.key  kubelet.key  kube-proxy.pem      kubernetes.pem


如果没分发证书, 到master节点将相应的证书copy过去

mkdir -pv /etc/kubernetes/pki
scp  root@192.168.204.3:~/pki/{admin-key.pem,admin.pem,ca-key.pem,ca.pem,kube-proxy-key.pem,kube-proxy.pem,kubernetes-key.pem,kubernetes.pem}  /etc/kubernetes/pki/
scp  root@192.168.204.3:~/TLS_Bootstrapping/{bootstrap.kubeconfig,kube-proxy.kubeconfig} /etc/kubernetes/


部署docker

curl -fsSL "https://get.docker.com/" > docker_install.sh
sh docker_install.sh --mirror Aliyun


同时为了加速docker官方镜像的下载速度, 配置加速器(因为私有仓库没有使用https, 因此这里同时将私有仓库地址也加上):

# 注意由于自己搭建的私有仓库没有使用SSL, 如果你也是请将192.168.204.15换成你自己的仓库地址
mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://v5d7kh0f.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.204.15"]
}
EOF


然后重启docker:

systemctl restart docker
systemctl status docker


下载最新的node节点的二进制包

node节点的二进制包: kubernetes-node-linux-amd64.tar.gz, 已下载到我们百度云盘: 国内下载

tar vxf kubernetes-node-linux-amd64.tar.gz
cd kubernetes && cp  ./node/bin/{kube-proxy,kubelet} /usr/local/bin/


部署kubelet

创建kubelet的service配置文件:/usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_POD_INFRA_CONTAINER \
$KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target


创建kubelet的配置文件/etc/kubernetes/kubelet(
注意请将--pod-infra-container-image的值更换成自己私有仓库对应pause的镜像地址
):

###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.204.6"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.204.6"
#
## location of the api-server
## COMMENT THIS ON KUBERNETES 1.8+
# KUBELET_API_SERVER="--api-servers=192.168.204.3:8080"
#
## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=192.168.204.15/kubernetes/pause-amd64:3.0"
#
## Add your own!
KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/pki --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --allow_privileged=true"


注意
: –allow_privileged=true 必须为true, 不然后面部署daemonSet的应用时会没权限, 比如下面的报错:

2月 02 01:45:43 k8s-apiserver01 kubelet[17396]: W0202 01:45:43.550410   17396 config.go:350] Pod[1] (calico-node-5cvkj_kube-system(b0cd277f-07e4-11e8-a15f-fa163ee6f9f6)) from api failed validation, ignoring: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy


kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求，需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper cluster 角色(role)， 然后 kubelet 才能有权限创建认证请求(certificate signing requests):

# 注意 请在master节点上执行, node节点没有配置客户端
cd /etc/kubernetes && kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap


启动服务:

# 启动前手动拉取镜像
docker pull index.tenxcloud.com/jimmy/pod-infrastructure:rhel7
# 关闭swap
swapoff -a
# 启动服务
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet


kubelet 首次启动时向kube-apiserver发送证书签名请求，必须通过后kubernetes系统才会将该Node加入到集群, 因此需要在master节点上对这些node的加入进行审计, 通过后会自动为这些node颁发证书

# 查看未授权的CSR请求
$ kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-D0qfDbLl4sA2uFOuEkEXt0pqYr0DjqCTNqxyaocvgq0   30m       kubelet-bootstrap   Pending
# 通过CSR请求
$ kubectl certificate approve node-csr-D0qfDbLl4sA2uFOuEkEXt0pqYr0DjqCTNqxyaocvgq0
# 查看才加入的node
$ kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
192.168.204.6   Ready     <none>    47s       v1.8.7


部署kub-proxy

Kube-proxy 是实现Service的关键组件, kube-proxy会在每台节点上执行, 然后监听API Server的Service与Endpoint资源对象的改变, 然后来依据变化执行iptables来实现网络的转发。

创建 kube-proxy 的service配置文件: /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


kube-proxy配置文件/etc/kubernetes/proxy

###
# kubernetes proxy config

# default config should be adequate

# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.204.6 --hostname-override=192.168.204.6 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"


启动服务:

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy
$ netstat -tlnup | grep kub
tcp        0      0 192.168.204.6:10250     0.0.0.0:*               LISTEN      12437/kubelet
tcp        0      0 192.168.204.6:10255     0.0.0.0:*               LISTEN      12437/kubelet
tcp        0      0 192.168.204.6:4194      0.0.0.0:*               LISTEN      12437/kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      12437/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      12776/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      12776/kube-proxy


剩下的2个node节点: 192.168.204.和192.168.204.14, 192.168.204.3如法炮制, 然后我们在master看看所有已经添加好的node节点:

[root@k8s-apiserver01 docker]# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
192.168.204.3   Ready     <none>    54m       v1.8.7
192.168.204.4   Ready     <none>    3d        v1.8.7
192.168.204.6   Ready     <none>    3d        v1.8.7
192.168.204.7   Ready     <none>    3d        v1.8.7


集群测试

我们通过部署一个nginx服务来验证集群是否正常工作:

为了加快验证的效果, 我们先在node节点上拉取nginx的镜像:

docker pull nginx


然后在master节点上部署服务，并暴露出来:

$ kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=nginx  --port=80
deployment "nginx" created
$ kubectl expose deployment nginx --type=NodePort --name=example-service
service "example-service" exposed
$ kubectl describe svc example-service
Name:                     example-service
Namespace:                default
Labels:                   run=load-balancer-example
Annotations:              <none>
Selector:                 run=load-balancer-example
Type:                     NodePort
IP:                       10.254.100.109
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30924/TCP
Endpoints:                172.30.52.2:80,172.30.52.3:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


然后访问暴露出来的服务:

[root@k8s-apiserver01 ~]# curl 192.168.204.6:30924
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>


总结

经过这篇博客, 我们成功部署了一个4个节点的kubernetes 1.8.7的集群, 但是此时集群的网络, DNS, DashBoard, 监控都还没完成, 在接下来的系列中将进行这些组件的部署。
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 点击这里给我发消息
标签： 
相关文章推荐
新的分享
章节导航