Harbor-hlem镜像库重新部署后PV数据恢复
2020-07-15 18:53
375 查看
起因
开发反馈habor镜像库登陆不了，初步查看是证书过期了。
解决方案
之前Harbor-helm部署镜像库文档可以回顾链接https://minminmsn.com/middleware/698/
1.首先新建新证书的secret
[root@elasticsearch01 harbor-helm]# kubectl create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt
2.然后修改harbor-helm的value.yaml，把secretName替换下
[root@elasticsearch01 harbor-helm]# head -n 20 values.yaml expose: # Set the way how to expose the service. Set the type as "ingress", # "clusterIP" or "nodePort" and fill the information in the corresponding # section type: ingress tls: # Enable the tls or not. Note: if the type is "ingress" and the tls # is disabled, the port must be included in the command when pull/push # images. Refer to https://github.com/goharbor/harbor/issues/5291 # for the detail. enabled: true # Fill the name of secret if you want to use your own TLS certificate # and private key. The secret must contain keys named tls.crt and # tls.key that contain the certificate and private key to use for TLS # The certificate and private key will be generated automatically if # it is not set secretName: "ingress-secret2021" # By default, the Notary service will use the same cert and key as # described above. Fill the name of secret if you want to use a # separated one. Only needed when the type is "ingress".
3.最后使用helm upgrade更新版本
[root@elasticsearch01 harbor-helm]# helm upgrade minminmsn . -f values.yaml
到这个时候应该能解决需求，可是事与愿违，不知道哪儿除了问题，这时登陆Harbor证书问题是解决了，但是项目及库访问不了提示内部错误，看Pod的运行状态也都是Running。
最后打算使用helm先delete掉再install，但是这样创建的harbor看起来一切正常，实际上是个初始化环境，是自动生成的新PV并没有原来的数据。此时发现原来的PV还在，下面就开始找PV恢复的方案。
调整PV状态
1.查询此时PV与PVC状态
[root@elasticsearch01 harbor-helm]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 9h pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Released default/minminmsn-harbor-chartmuseum rbd 417d pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/minminmsn-harbor-jobservice rbd 417d pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Released default/minminmsn-harbor-registry rbd 417d pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/database-data-minminmsn-harbor-database-0 rbd 417d pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/data-minminmsn-harbor-redis-0 rbd 417d [root@elasticsearch01 harbor-helm]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-minminmsn-harbor-redis-0 Bound pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h database-data-minminmsn-harbor-database-0 Bound pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h minminmsn-harbor-chartmuseum Bound pvc-6c903857-c5f0-11ea-9386-52540089b2b6 50Gi RWO rbd 9h minminmsn-harbor-jobservice Bound pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h minminmsn-harbor-registry Bound pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6 500Gi RWO rbd 9h
2.修改PV状态
先把PV的状态由Released改变成
备注：默认创建的PV的回收策略是Delete就是用完就删除，之前特意把RECLAIM POLICY改为了Retain，在线修改PV回收策略可以参考文档https://minminmsn.com/cloud/1091/。否则这里Helm Delete后就会自动删除PV，就没有后来这篇PV数据恢复操作了。
在线编辑PV,需要把其中claimRef这段删除，这样状态就可以变成Available了。
claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: minminmsn-harbor-chartmuseum namespace: default resourceVersion: "91736092" uid: b31ec8ca-c649-11ea-9386-52540089b2b6 persistentVolumeReclaimPolicy: Retain
具体如下修改
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: ceph.com/rbd rbdProvisionerIdentity: ceph.com/rbd creationTimestamp: "2019-05-24T06:33:55Z" finalizers: - kubernetes.io/pv-protection name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 resourceVersion: "91736100" selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6 spec: accessModes: - ReadWriteOnce capacity: storage: 50Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: minminmsn-harbor-chartmuseum namespace: default resourceVersion: "91736092" uid: b31ec8ca-c649-11ea-9386-52540089b2b6 persistentVolumeReclaimPolicy: Retain rbd: image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905 keyring: /etc/ceph/keyring monitors: - 10.0.4.8:6789 pool: rbd-k8s secretRef: name: ceph-secret namespace: default user: admin storageClassName: rbd volumeMode: Filesystem status: phase: Released
3.其他四个PV同样操作
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6
4.查看效果
现在看PV的STATUS已经变成了Available，然后CLAIM也变空了，这样就可以在后面绑定使用了
[root@elasticsearch01 harbor-helm]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Available rbd 417d pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Available rbd 417d pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d
创建PVC
1.先设置好PVC及PV对应关系
[root@elasticsearch01 yaml]# cat minminmsn.pvc apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minminmsn-harbor-registry spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 2000Gi volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minminmsn-harbor-jobservice spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 20Gi volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minminmsn-harbor-chartmuseum spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 50Gi volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: database-data-minminmsn-harbor-database-0 spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 20Gi volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-minminmsn-harbor-redis-0 spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 20Gi volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"
2.创建PVC
[root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc persistentvolumeclaim/minminmsn-harbor-registry created persistentvolumeclaim/minminmsn-harbor-jobservice created persistentvolumeclaim/minminmsn-harbor-chartmuseum created persistentvolumeclaim/database-data-minminmsn-harbor-database-0 created persistentvolumeclaim/data-minminmsn-harbor-redis-0 created
3.检查PV与PVC
[root@elasticsearch01 yaml]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Bound default/minminmsn-harbor-chartmuseum rbd 417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/minminmsn-harbor-jobservice rbd 417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Bound default/minminmsn-harbor-registry rbd 417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/database-data-minminmsn-harbor-database-0 rbd 417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/data-minminmsn-harbor-redis-0 rbd 417d
[root@elasticsearch01 yaml]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-rbd-pv-claim Bound ceph-rbd-pv 20Gi RWO 540d
data-minminmsn-harbor-redis-0 Pending pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
database-data-minminmsn-harbor-database-0 Pending pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
minminmsn-harbor-chartmuseum Pending pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
minminmsn-harbor-jobservice Pending pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
minminmsn-harbor-registry Bound pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO rbd 12s
[root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registry
Name: minminmsn-harbor-registry
Namespace: default
StorageClass: rbd
Status: Bound
Volume: pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe...
pv.kubernetes.io/bind-completed: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2000Gi
Access Modes: RWO
VolumeMode: Filesystem
Events: <none>
Mounted By: <none>
使用Hlem重新部署Harbor镜像库
1.部署前先删除版本
[root@elasticsearch01 harbor-helm]# helm delete --purge minminmsn helm delete --purge minminmsn release "minminmsn" deleted
2.修改Harbor-helm的values.yaml中PVC相关值
注意existingClaim: ""由空值改成上面生成的PVC名字，注意对应关系，其他不变，具体变更如下
persistence: enabled: true # Setting it to "keep" to avoid removing PVCs during a helm delete # operation. Leaving it empty will delete PVCs after the chart deleted resourcePolicy: "keep" persistentVolumeClaim: registry: # Use the existing PVC which must be created manually before bound existingClaim: "minminmsn-harbor-registry" # Specify the "storageClass" used to provision the volume. Or the default # StorageClass will be used(the default). # Set it to "-" to disable dynamic provisioning storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 2000Gi chartmuseum: existingClaim: "minminmsn-harbor-chartmuseum" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 50Gi jobservice: existingClaim: "minminmsn-harbor-jobservice" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 20Gi # If external database is used, the following settings for database will # be ignored database: existingClaim: "database-data-minminmsn-harbor-database-0" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 20Gi # If external Redis is used, the following settings for Redis will # be ignored redis: existingClaim: "data-minminmsn-harbor-redis-0" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 20Gi
3.重新部署
[root@elasticsearch01 harbor-helm]# helm install . --name minminmsn NAME: minminmsn LAST DEPLOYED: Wed Jul 15 11:18:13 2020 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minminmsn-harbor-adminserver ClusterIP 10.254.58.23 <none> 80/TCP 1s minminmsn-harbor-chartmuseum ClusterIP 10.254.154.44 <none> 80/TCP 1s minminmsn-harbor-clair ClusterIP 10.254.25.107 <none> 6060/TCP 1s minminmsn-harbor-core ClusterIP 10.254.56.153 <none> 80/TCP 1s minminmsn-harbor-database ClusterIP 10.254.65.18 <none> 5432/TCP 1s minminmsn-harbor-jobservice ClusterIP 10.254.81.97 <none> 80/TCP 1s minminmsn-harbor-notary-server ClusterIP 10.254.99.90 <none> 4443/TCP 1s minminmsn-harbor-notary-signer ClusterIP 10.254.175.105 <none> 7899/TCP 1s minminmsn-harbor-portal ClusterIP 10.254.242.113 <none> 80/TCP 1s minminmsn-harbor-redis ClusterIP 10.254.127.40 <none> 6379/TCP 1s minminmsn-harbor-registry ClusterIP 10.254.158.222 <none> 5000/TCP,8080/TCP 1s ==> v1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE minminmsn-harbor-adminserver 1 1 1 0 1s minminmsn-harbor-chartmuseum 1 1 1 0 1s minminmsn-harbor-clair 1 0 0 0 1s minminmsn-harbor-core 1 0 0 0 1s minminmsn-harbor-jobservice 1 0 0 0 1s minminmsn-harbor-notary-server 1 0 0 0 1s minminmsn-harbor-notary-signer 1 0 0 0 1s minminmsn-harbor-portal 1 0 0 0 1s minminmsn-harbor-registry 1 0 0 0 1s ==> v1/StatefulSet NAME DESIRED CURRENT AGE minminmsn-harbor-database 1 1 1s minminmsn-harbor-redis 1 1 1s ==> v1beta1/Ingress NAME HOSTS ADDRESS PORTS AGE minminmsn-harbor-ingress core-harbor.minminmsn.com,notary-harbor.minminmsn.com 80, 443 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE minminmsn-harbor-adminserver-b5d58db8c-wmrbd 0/1 ContainerCreating 0 1s minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 0/1 Pending 0 1s minminmsn-harbor-clair-54465ff7dd-d7bxx 0/1 Pending 0 1s minminmsn-harbor-core-587cc5d9b5-2xxl9 0/1 Pending 0 1s minminmsn-harbor-jobservice-764bb697d-wsxqx 0/1 Pending 0 1s minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 0/1 Pending 0 1s minminmsn-harbor-notary-signer-8466d68f5b-klv76 0/1 Pending 0 1s minminmsn-harbor-database-0 0/1 Pending 0 1s minminmsn-harbor-redis-0 0/1 Pending 0 1s ==> v1/Secret NAME TYPE DATA AGE minminmsn-harbor-adminserver Opaque 4 1s minminmsn-harbor-chartmuseum Opaque 1 1s minminmsn-harbor-core Opaque 4 1s minminmsn-harbor-database Opaque 1 1s minminmsn-harbor-jobservice Opaque 1 1s minminmsn-harbor-registry Opaque 1 1s ==> v1/ConfigMap NAME DATA AGE minminmsn-harbor-adminserver 39 1s minminmsn-harbor-chartmuseum 24 1s minminmsn-harbor-clair 1 1s minminmsn-harbor-core 1 1s minminmsn-harbor-jobservice 1 1s minminmsn-harbor-notary-server 5 1s minminmsn-harbor-registry 2 1s NOTES: Please wait for several minutes for Harbor deployment to complete. Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com. For more details, please visit https://github.com/goharbor/harbor. 3.查看新生成Pods的信息 [root@elasticsearch01 harbor-helm]# kubectl get pods NAME READY STATUS RESTARTS AGE minminmsn-harbor-adminserver-b5d58db8c-wmrbd 0/1 ContainerCreating 0 9s minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 0/1 ContainerCreating 0 9s minminmsn-harbor-clair-54465ff7dd-d7bxx 0/1 Running 0 9s minminmsn-harbor-core-587cc5d9b5-2xxl9 0/1 Running 0 9s minminmsn-harbor-database-0 0/1 Init:0/1 0 9s minminmsn-harbor-jobservice-764bb697d-wsxqx 0/1 ContainerCreating 0 9s minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 0/1 ContainerCreating 0 9s minminmsn-harbor-notary-signer-8466d68f5b-klv76 0/1 ContainerCreating 0 9s minminmsn-harbor-portal-64cf8b9b69-xm8nl 0/1 ContainerCreating 0 8s minminmsn-harbor-redis-0 0/1 ContainerCreating 0 9s minminmsn-harbor-registry-755746c5bb-q8m55 0/2 ContainerCreating 0 8s
再等2分钟查看就上恢复了
[root@elasticsearch01 harbor-helm]# kubectl get pods NAME READY STATUS RESTARTS AGE jenkins-0 1/1 Running 0 62d rbd-provisioner-67b4857bcd-rjwlg 1/1 Running 0 61d minminmsn-harbor-adminserver-b5d58db8c-wmrbd 1/1 Running 1 2m33s minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 1/1 Running 0 2m33s minminmsn-harbor-clair-54465ff7dd-d7bxx 1/1 Running 1 2m33s minminmsn-harbor-core-587cc5d9b5-2xxl9 1/1 Running 1 2m33s minminmsn-harbor-database-0 1/1 Running 0 2m33s minminmsn-harbor-jobservice-764bb697d-wsxqx 1/1 Running 0 2m33s minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 1/1 Running 0 2m33s minminmsn-harbor-notary-signer-8466d68f5b-klv76 1/1 Running 0 2m33s minminmsn-harbor-portal-64cf8b9b69-xm8nl 1/1 Running 0 2m32s minminmsn-harbor-redis-0 1/1 Running 0 2m33s minminmsn-harbor-registry-755746c5bb-q8m55 2/2 Running 0 2m32s
4.Harbor控制验证
证书更新了项目也恢复了
https://core-harbor.minminmsn.com/harbor/projects
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- 在项目中部署redis企业级数据备份方案以及各种踩坑的数据恢复容灾演练
- 重新安装OS,ORACLEL软件用RMAN恢复数据
- 硬盘重新分区后有一个分区表信息丢失的数据恢复
- mysql服务器出问题重新安装与数据恢复
- 每次重新部署后hibernate会把数据库里表都drop掉，数据丢失
- windows操作系统重新安装后的数据恢复
- Exchange2010 重新/恢复部署案例
- viewpager嵌套fragment，滑动别的fragment之后，之前的fragment数据重新加载，恢复默认了
- 数据恢复+删除+重新分区
- 硬盘或移动硬盘被重新分区后，数据怎么恢复
- 备份环境的部署及实施续——第五部分 应用数据的备份和恢复
- 硬盘分区并且重新格式化之后的数据恢复
- 数据恢复+删除+重新分区
- Elasticsearch之重要核心概念（cluster（集群）、shards（分配）、replicas（索引副本）、recovery（据恢复或叫数据重新分布）、gateway（es索引的持久化存储方式）、discovery.zen（es的自动发现节点机制机制）、Transport（内部节点或集群与客户端的交互方式）、settings（修改索引库默认配置）和mappings）
- docker 镜像仓库 Harbor 部署 以及 跨数据复制
- 重新装Mysql后原来数据的恢复办法
- 重新分区后数据丢失怎么恢复？
- MongoDB远程主从部署下的全量数据同步及故障恢复策略
- 用winhex恢复500G的硬盘被重新分区后的数据
- 电脑ghostxp系统后不小心重新分区了的数据恢复法子