Kubeadm部署基于Calico K8s集群

1 环境

 2 内核调优

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

3 修改文件描述符

echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

hard limits自AIX 4.1版本开始引入。hard limits 应由AIX系统管理员设置，只有security组的成员可以将此值增大，用户本身可以减小此限定值,但是其更改将随着该用户从系统退出而失效

soft limits 是AIX核心使用的限制进程对系统资源的使用的上限值。此值可由任何人更改，但不能超出hard limits值。这里要注意的是只有security组的成员可使更改永久生效普通用户的更改在其退出系统后将失效

1)soft nofile和hard nofile示，单个用用户的软限制为1000，硬限制为1200，即表示单用户能打开的最大文件数量为1000，不管它开启多少个shell。

2）soft nproc和hard nproc 单个用户可用的最大进程数量,软限制和硬限制

3）memlock 一个任务锁住的物理内存的最大值（这里设置成无限制）

4  配置k8s yum 源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

5 配置docker yum源

cd /etc/yum.repos.d
wget https://download.docker.com/linux/centos/docker-ce.repo

6 时间同步 

一个集群内的时间同步必不可少

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service

ntpdate -u ntp.api.bz

7  关闭SELinux、防火墙

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

8 关闭系统的Swap

swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

9 配置hosts 解析

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.135.239	k8s-1001	 k8s-1001
172.31.135.237  k8s-1003     k8s-1003
172.31.135.238  k8s-1002     k8s-1002

10 配置节点免密登录

ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub  用户名字@192.168.x.xxx

11 安装依赖等

yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl lrzsz wget

12 配置ipvs模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

yum install ipset ipvsadm -y

13 安装docker 

yum list docker-ce --showduplicates | sort -r

yum install -y docker-ce-18.06.1.ce-3.el7

最号先不要安装最新版本
systemctl daemon-reload
systemctl enable docker
systemctl start docker

14 master 和node 节点安装 kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl  master
systemctl enable kubelet
暂不启动 kubelet

15 如果在国内请按照下面步骤进行

生成默认配置 

kubeadm config print init-defaults > /root/kubeadm.conf

修改 /root/kubeadm.conf，使用国内阿里的imageRepository: registry.aliyuncs.com/google_containers

下载镜像

kubeadm config images pull --config /root/kubeadm.conf

16 如过网络允许可以直接初始化集群

kubeadm init --kubernetes-version=v1.14.1 --pod-network-cidr=10.244.0.0/16

这里没有设置--service-cidr 是因为我们下面需要部署calico网络，calico会帮助我们设置service网络，如果此地设置了service网络会导致calico部署不成功

保存这段内容

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
--discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1

16 创建相关文件夹

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

18 如果以后还有机器加入集群如何获取token 和 hash值？

获得token和hash值
1）获取token
kubeadm token list
默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Token
kubeadm token create

2）获取hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

19 验证

kubectl get pods --all-namespaces

NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-pdf9r            0/1     Pending   0          6m50s
kube-system   coredns-fb8b8dccf-rngcz            0/1     Pending   0          6m50s
kube-system   etcd-k8s-1001                      1/1     Running   0          5m52s
kube-system   kube-apiserver-k8s-1001            1/1     Running   0          6m4s
kube-system   kube-controller-manager-k8s-1001   1/1     Running   0          5m51s
kube-system   kube-proxy-b8dhg                   1/1     Running   0          6m50s
kube-system   kube-scheduler-k8s-1001            1/1     Running   0          5m47s

coredns 是pending状态 先不用管它，因为这个没有网络插件的导致的

20 部署calico网络插件

官方文档

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml

我们需要修改calico文件

https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

vim calico.yaml

1）修改ipip模式关闭 和typha_service_name

- name: CALICO_IPV4POOL_IPIP
value: "off"

typha_service_name: "calico-typha"

calico网络，默认是ipip模式（在每台node主机创建一个tunl0网口，这个隧道链接所有的node容器网络，官网推荐不同的ip网段适合，比如aws的不同区域主机），

修改成BGP模式，它会以daemonset方式安装在所有node主机，每台主机启动一个bird(BGP client)，它会将calico网络内的所有node分配的ip段告知集群内的主机，并通过本机的网卡eth0或者ens33转发数据；

2）修改replicas

replicas: 1
revisionHistoryLimit: 2

3）修改pod的网段CALICO_IPV4POOL_CIDR

- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
4）如果手动下载镜像请查看calico.yaml 文件里面标注的镜像版本 否则可以直接执行会自动下载
5）部署calico
kubectl apply -f calico.yaml

6）查看
kubectl get po --all-namespaces
此时你会发现是pending状态是因为node节点还没有相关组件
7） 验证是否为bgp模式
# ip route show
default via 172.31.143.253 dev eth0
blackhole 10.244.0.0/24 proto bird
10.244.0.2 dev caliac6de7553e8 scope link
10.244.0.3 dev cali1591fcccf0f scope link
10.244.1.0/24 via 172.31.135.237 dev eth0 proto bird
10.244.2.0/24 via 172.31.135.238 dev eth0 proto bird
169.254.0.0/16 dev eth0 scope link metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.31.128.0/20 dev eth0 proto kernel scope link src 172.31.135.239

21 将node加入到节点

kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
--discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1

master上查看集群
[root@k8s-1001 ~]# kubectl get node
NAME       STATUS   ROLES    AGE    VERSION
k8s-1001   Ready    master   37m    v1.14.1
k8s-1002   Ready    <none>   99s    v1.14.1
k8s-1003   Ready    <none>   115s   v1.14.1
[root@k8s-1001 ~]# kubectl get node -o wide
NAME       STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-1001   Ready    master   37m    v1.14.1   172.31.135.239   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
k8s-1002   Ready    <none>   103s   v1.14.1   172.31.135.238   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
k8s-1003   Ready    <none>   119s   v1.14.1   172.31.135.237   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1

[root@k8s-1001 ~]# kubectl get pod -o wide -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
calico-node-8z92v                  2/2     Running   0          2m1s    172.31.135.238   k8s-1002   <none>           <none>
calico-node-k542k                  2/2     Running   0          7m32s   172.31.135.239   k8s-1001   <none>           <none>
calico-node-n4jgf                  2/2     Running   0          2m17s   172.31.135.237   k8s-1003   <none>           <none>
calico-typha-55968bfd7b-c5r4z      1/1     Running   0          7m33s   172.31.135.237   k8s-1003   <none>           <none>
coredns-fb8b8dccf-pdf9r            1/1     Running   0          37m     10.244.0.3       k8s-1001   <none>           <none>
coredns-fb8b8dccf-rngcz            1/1     Running   0          37m     10.244.0.2       k8s-1001   <none>           <none>
etcd-k8s-1001                      1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
kube-apiserver-k8s-1001            1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
kube-controller-manager-k8s-1001   1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
kube-proxy-b8dhg                   1/1     Running   0          37m     172.31.135.239   k8s-1001   <none>           <none>
kube-proxy-nvlmz                   1/1     Running   0          2m17s   172.31.135.237   k8s-1003   <none>           <none>
kube-proxy-rfb77                   1/1     Running   0          2m1s    172.31.135.238   k8s-1002   <none>           <none>
kube-scheduler-k8s-1001            1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>

22 开启kube-proxy  lvs

修改ConfigMap的kube-system/kube-proxy中的config.conf，`mode: "ipvs"`：

kubectl edit cm kube-proxy -n kube-system
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr
-> 172.31.135.239:6443          Masq    1      0          0
TCP  10.96.0.10:53 rr
-> 10.244.0.2:53                Masq    1      0          0
-> 10.244.0.3:53                Masq    1      0          0
TCP  10.96.0.10:9153 rr
-> 10.244.0.2:9153              Masq    1      0          0
-> 10.244.0.3:9153              Masq    1      0          0
TCP  10.111.3.127:5473 rr
-> 172.31.135.237:5473          Masq    1      0          0
UDP  10.96.0.10:53 rr
-> 10.244.0.2:53                Masq    1      0          0
-> 10.244.0.3:53                Masq    1      0          0

23 master默认是污点，pod是不会调度到master

1）开放master可被调度
kubectl taint node k8s-1001 node-role.kubernetes.io/master-
2）
如果要恢复 Master Only 状态
kubectl taint node k8s-1001 node-role.kubernetes.io/master="":NoSchedule

24 创建一个测试pod 验证集群

kubectl run net-test --image=alpine --replicas=2 sleep 360

# kubectl get pod -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
net-test-7d6d58cc8-78wd6   1/1     Running   0          6s    10.244.1.2   k8s-1003   <none>           <none>
net-test-7d6d58cc8-hjdhw   1/1     Running   0          6s    10.244.2.2   k8s-1002   <none>           <none>