kubernetes1.13.1集群结合ceph rbd部署最新版本jenkins
参考文档
https://blog.csdn.net/aixiaoyang168/article/details/79767649 https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes https://www.cnblogs.com/cocowool/p/kubernetes_statefulset.html https://www.cnblogs.com/cocowool/p/kubernetes_storage.html
文档目录
- kubernetes1.13.1+etcd3.3.10+flanneld0.10集群部署
- kubernetes1.13.1部署kuberneted-dashboard v1.10.1
- kubernetes1.13.1部署coredns
- kubernetes1.13.1部署ingress-nginx并配置https转发dashboard
- kubernetes1.13.1部署metrics-server0.3.1
- kubernetes1.13.1集群使用ceph rbd存储块
- kubernetes1.13.1集群结合ceph rbd部署最新版本jenkins
简介
jenkins-kubernetes-plugin
Jenkins plugin to run dynamic agents in a Kubernetes cluster.
Based on the Scaling Docker with Kubernetes article, automates the scaling of Jenkins agents running in Kubernetes.
The plugin creates a Kubernetes Pod for each agent started, defined by the Docker image to run, and stops it after each build.
Agents are launched using JNLP, so it is expected that the image connects automatically to the Jenkins master. For that some environment variables are automatically injected:
- JENKINS_URL: Jenkins web interface url
- JENKINS_SECRET: the secret key for authentication
- JENKINS_AGENT_NAME: the name of the Jenkins agent
- JENKINS_NAME: the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)
基本环境
k81集群1.13.1版本
[root@elasticsearch01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 10.2.8.34 Ready <none> 25d v1.13.1 10.2.8.65 Ready <none> 25d v1.13.1
ceph集群 luminous版本
[root@ceph01 ~]# ceph -s services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 mgr: ceph03(active), standbys: ceph02, ceph01 osd: 24 osds: 24 up, 24 in rgw: 3 daemons active
操作步骤
一、使用ceph rbd创建pv、pvc
官网使用的是自带创建pv与pvc这里使用的是手动创建
1、创建pv
[root@elasticsearch01 jenkins]# cat jenkins-pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-home-pv spec: capacity: storage: 40Gi accessModes: - ReadWriteOnce rbd: monitors: - '10.0.4.10:6789' - '10.0.4.13:6789' - '10.0.4.15:6789' pool: rbd-k8s image: cephimage3 user: admin secretRef: name: ceph-secret fsType: ext4 readOnly: false persistentVolumeReclaimPolicy: Recycle [root@elasticsearch01 jenkins]# kubectl create -f jenkins-pv.yaml persistentvolume/jenkins-home-pv created [root@elasticsearch01 jenkins]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE ceph-rbd-pv 20Gi RWO Recycle Bound default/ceph-rbd-pv-claim 22h jenkins-home-pv 40Gi RWO Recycle Available 4s
2、创建pvc
[root@elasticsearch01 jenkins]# cat jenkins-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-home-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi [root@elasticsearch01 jenkins]# kubectl create -f jenkins-pvc.yaml persistentvolumeclaim/jenkins-home-pvc created [root@elasticsearch01 jenkins]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-rbd-pv-claim Bound ceph-rbd-pv 20Gi RWO 22h jenkins-home-pvc Bound jenkins-home-pv 40Gi RWO 3s [root@elasticsearch01 jenkins]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE ceph-rbd-pv 20Gi RWO Recycle Bound default/ceph-rbd-pv-claim 22h jenkins-home-pv 40Gi RWO Recycle Bound default/jenkins-home-pvc 77s
二、跟进实际情况修改jenkins.yml文件
主要修改的配置从上到下分别是:
1、拉取镜像策略
imagePullPolicy: IfNotPresent
2、自动存储storage class改成voulumes的pvc方式实现
volumes: - name: jenkins-home persistentVolumeClaim: claimName: jenkins-home-pvc
3、ingress的host改成实际的
host: jenkins.minminmsn.com
4、ingres的tls证书改成实际的
tls: - hosts: - jenkins.minminmsn.com secretName: ingress-secret
5、具体如下
[root@elasticsearch01 jenkins]# cat jenkins.yml --- apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: jenkins labels: name: jenkins spec: serviceName: jenkins replicas: 1 updateStrategy: type: RollingUpdate template: metadata: name: jenkins labels: name: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins:lts-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 8080 - containerPort: 50000 resources: limits: cpu: 1 memory: 1Gi requests: cpu: 0.5 memory: 500Mi env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home readOnly: false livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 # ~2 minutes readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 # ~2 minutes securityContext: fsGroup: 1000 volumes: - name: jenkins-home persistentVolumeClaim: claimName: jenkins-home-pvc --- apiVersion: v1 kind: Service metadata: name: jenkins spec: # type: LoadBalancer selector: name: jenkins # ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7) #externalTrafficPolicy: Local ports: - name: http port: 80 targetPort: 8080 protocol: TCP - name: agent port: 50000 protocol: TCP --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: jenkins annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" kubernetes.io/tls-acme: "true" # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size nginx.ingress.kubernetes.io/proxy-body-size: 50m nginx.ingress.kubernetes.io/proxy-request-buffering: "off" # For nginx-ingress controller < 0.9.0.beta-18 ingress.kubernetes.io/ssl-redirect: "true" # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size ingress.kubernetes.io/proxy-body-size: 50m ingress.kubernetes.io/proxy-request-buffering: "off" spec: rules: - http: paths: - path: / backend: serviceName: jenkins servicePort: 80 host: jenkins.minminmsn.com tls: - hosts: - jenkins.minminmsn.com secretName: ingress-secret
三、创建状态集、svc、pod、ingress
1、创建rbac认证角色
[root@elasticsearch01 jenkins]# kubectl create -f service-account.yml serviceaccount/jenkins created role.rbac.authorization.k8s.io/jenkins created rolebinding.rbac.authorization.k8s.io/jenkins created
2、创建jenkins服务等
[root@elasticsearch01 jenkins]# kubectl create -f jenkins.yml statefulset.apps/jenkins created service/jenkins created ingress.extensions/jenkins created 4s [root@elasticsearch01 jenkins]# kubectl get pods NAME READY STATUS RESTARTS AGE busybox 1/1 Running 454 18d ceph-rbd-pv-pod1 1/1 Running 1 21h jenkins-0 0/1 ContainerCreating 0 7s [root@elasticsearch01 jenkins]# kubectl get pods NAME READY STATUS RESTARTS AGE busybox 1/1 Running 454 18d ceph-rbd-pv-pod1 1/1 Running 1 21h jenkins-0 1/1 Running 0 4m52s
四、通过ingress访问
获取ingress-nginx对外端口,https://jenkins.minminmsn.com:47215/访问即可,需要配置dns解析到pod所在node的ip
[root@elasticsearch01 jenkins]# kubectl get svc -n ingress-nginx|grep ingress-nginx ingress-nginx LoadBalancer 10.254.125.151 <pending> 80:33003/TCP,443:47215/TCP 14d
五、初始化jenkins
1、查找密码
[root@elasticsearch02 ~]# df -h|grep rbd /dev/rbd0 493G 163G 306G 35% /data /dev/rbd1 20G 45M 20G 1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage2 /dev/rbd2 40G 138M 40G 1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3 [root@elasticsearch02 ~]# cd //var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3 [root@elasticsearch02 rbd-k8s-image-cephimage3]# ls config.xml init.groovy.d jobs nodes secrets war copy_reference_file.log jenkins.CLI.xml logs plugins updates hudson.model.UpdateCenter.xml jenkins.install.UpgradeWizard.state lost+found secret.key userContent identity.key.enc jenkins.telemetry.Correlator.xml nodeMonitors.xml secret.key.not-so-secret users [root@elasticsearch02 rbd-k8s-image-cephimage3]# cat secrets/initialAdminPassword 92c145b796cc48b0af8b5ef0f7afce28
2、选择安装插件
3、创建初始管理账号
4、设置jenkins url默认https://jenkins.minminmsn.com:47215/
5、开始使用jenkins
6、jenkins控制台界面,主要配置都在系统管理中
总结
使用ceph rbd 这种只能读写一次的设备不能用在线上,线上应该使用分布式存储例如nfs,cephfs,glusterfs等,这里只是测试jenkins结合ceph,pv,pvc完成有状态pod的测试
- [Ceph001]基于CentOS7部署Ceph集群(版本10.2.2)
- ceph集群jewel版本部署osd激活权限报错-故障排查
- 【实践】基于CentOS7部署Ceph集群(版本10.2.2)
- 持续集成之⑤:jenkins结合脚本实现代码自动化部署及一键回滚至上一版本
- Ceph RBD为Kubernetes集群提供分布式数据存储
- 持续集成之Jenkins结合脚本实现代码自动化部署及一键回滚至上一版本
- 初试 Kubernetes 集群使用 Ceph RBD 块存储
- kubernetes1.13.1+etcd3.3.10+flanneld0.10集群部署
- Zookeeper 最新版本集群搭建部署
- 基于centos7.5安装部署最新luminous版 ceph集群部署整合openstack
- 持续集成之⑤:jenkins结合脚本实现代码自动化部署及一键回滚至上一版本
- CentOS 7.5安装部署Jewel版本Ceph集群
- ceph集群jewel版本部署osd激活权限报错-故障排查
- jenkins结合脚本实现代码自动化部署及一键回滚至上一版本
- 基于centos7.3安装部署jewel版本ceph集群实战演练
- 持续集成之⑤:jenkins结合脚本实现代码自动化部署及一键回滚至上一版本
- Apache Tomcat 8.0.9下载、安装、配置和部署(不是最新版本)
- Kubernetes集群中部署dashboard