openstack 管理二十四 - ceph 与 vm 连接测试记录
2017-02-20 15:53
417 查看
目的
测试 ceph 与 vm 连接与使用
创建 vm
主机 128030 及 129094 是全新安装并利用 puppet 推送的 nova compute 主机计划在这两个主机上进行 vm 连接 ceph 测试
nova boot --flavor b2c_web_1core --image Centos6.3_1.3 --security_group default --nic net-id=9106aee4-2dc0-4a6d-a789-10c53e2b88c1 ceph-test01.sh.vclound.com --availability-zone nova:sh-compute-128030.sh.vclound.com
+--------------------------------------+------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-0000020d |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | 5DCHoj8ihwN6 |
| config_drive | |
| created | 2015-06-25T03:49:14Z |
| flavor | b2c_web_1core (5) |
| hostId | |
| id | 99d37977-a13a-4a8b-b8b1-e613a4959623 |
| image | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4) |
| key_name | - |
| metadata | {} |
| name | ceph-test01.sh.vclound.com |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | bb0b51d166254dc99bc7462c0ac002ff |
| updated | 2015-06-25T03:49:14Z |
| user_id | 226e71f1c1aa4bae85485d1d17b6f0ae |
+--------------------------------------+------------------------------------------------------+
nova boot --flavor b2c_web_1core --image Centos6.3_1.3 --security_group default --nic net-id=9106aee4-2dc0-4a6d-a789-10c53e2b88c1 ceph-test02.sh.vclound.com --availability-zone nova:sh-compute-129094.sh.vclound.com
+--------------------------------------+------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-0000020f |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | wHddAW33sFBE |
| config_drive | |
| created | 2015-06-25T03:51:03Z |
| flavor | b2c_web_1core (5) |
| hostId | |
| id | b433b227-14ab-4157-8f08-362ad680e35e |
| image | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4) |
| key_name | - |
| metadata | {} |
| name | ceph-test02.sh.vclound.com |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | bb0b51d166254dc99bc7462c0ac002ff |
| updated | 2015-06-25T03:51:03Z |
| user_id | 226e71f1c1aa4bae85485d1d17b6f0ae |
+--------------------------------------+------------------------------------------------------+instance 状态
[root@sh-controller-129022 ~(keystone_admin)]# nova list +--------------------------------------+-----------------------------+--------+------------+-------------+---------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------------+--------+------------+-------------+---------------------------+ | 99d37977-a13a-4a8b-b8b1-e613a4959623 | ceph-test01.sh.vclound.com | ACTIVE | - | Running | SH_DEV_NET=10.198.192.254 | | b433b227-14ab-4157-8f08-362ad680e35e | ceph-test02.sh.vclound.com | ACTIVE | - | Running | SH_DEV_NET=10.198.192.255 | +--------------------------------------+-----------------------------+--------+------------+-------------+---------------------------+
创建云盘
[root@sh-controller-129022 ~(keystone_admin)]# cinder create 50
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2015-06-25T06:11:58.840626 |
| display_description | None |
| display_name | None |
| encrypted | False |
| id | 8516fb02-b578-4e57-9678-d30d2b0a6734 |
| metadata | {} |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# cinder create 50
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2015-06-25T06:12:07.151001 |
| display_description | None |
| display_name | None |
| encrypted | False |
| id | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f |
| metadata | {} |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# cinder create 50
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2015-06-25T06:12:14.321030 |
| display_description | None |
| display_name | None |
| encrypted | False |
| id | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 |
| metadata | {} |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+查询云盘
[root@sh-controller-129022 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 8516fb02-b578-4e57-9678-d30d2b0a6734 | available | None | 50 | None | false | | | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f | available | None | 50 | None | false | | | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 | available | None | 50 | None | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
连接云盘
[root@sh-controller-129022 ~(keystone_admin)]# nova volume-attach 99d37977-a13a-4a8b-b8b1-e613a4959623 8516fb02-b578-4e57-9678-d30d2b0a6734 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdc | | id | 8516fb02-b578-4e57-9678-d30d2b0a6734 | | serverId | 99d37977-a13a-4a8b-b8b1-e613a4959623 | | volumeId | 8516fb02-b578-4e57-9678-d30d2b0a6734 | +----------+--------------------------------------+ [root@sh-controller-129022 ~(keystone_admin)]# nova volume-attach 99d37977-a13a-4a8b-b8b1-e613a4959623 9d8aa395-5e6a-411a-9f19-6375f29e9f9f +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdd | | id | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f | | serverId | 99d37977-a13a-4a8b-b8b1-e613a4959623 | | volumeId | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f | +----------+--------------------------------------+ [root@sh-controller-129022 ~(keystone_admin)]# nova volume-attach b433b227-14ab-4157-8f08-362ad680e35e a5751c38-01c0-4f25-a02c-7d2a05d6ea36 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdc | | id | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 | | serverId | b433b227-14ab-4157-8f08-362ad680e35e | | volumeId | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 | +----------+--------------------------------------+
检测云盘
[root@sh-controller-129022 ~(keystone_admin)]# nova show 99d37977-a13a-4a8b-b8b1-e613a4959623
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | sh-compute-128030.sh.vclound.com |
| OS-EXT-SRV-ATTR:hypervisor_hostname | sh-compute-128030.sh.vclound.com |
| OS-EXT-SRV-ATTR:instance_name | instance-0000020d |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-06-25T03:49:26.000000 |
| OS-SRV-USG:terminated_at | - |
| SH_DEV_NET network | 10.198.192.254 |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-25T03:49:14Z |
| flavor | b2c_web_1core (5) |
| hostId | 8b5b75df8b0271d739323f1373b7363d432bb9c68b079ab3e94e1c1a |
| id | 99d37977-a13a-4a8b-b8b1-e613a4959623 |
| image | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4) |
| key_name | - |
| metadata | {} |
| name | ceph-test01.sh.vclound.com |
| os-extended-volumes:volumes_attached | [{"id": "8516fb02-b578-4e57-9678-d30d2b0a6734"}, {"id": "9d8aa395-5e6a-411a-9f19-6375f29e9f9f"}] | <-挂载两个云盘
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | bb0b51d166254dc99bc7462c0ac002ff |
| updated | 2015-06-25T03:49:26Z |
| user_id | 226e71f1c1aa4bae85485d1d17b6f0ae |
+--------------------------------------+--------------------------------------------------------------------------------------------------+
[root@sh-controller-129022 ~(keystone_admin)]# nova show b433b227-14ab-4157-8f08-362ad680e35e
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | sh-compute-129094.sh.vclound.com |
| OS-EXT-SRV-ATTR:hypervisor_hostname | sh-compute-129094.sh.vclound.com |
| OS-EXT-SRV-ATTR:instance_name | instance-0000020f |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-06-25T03:52:05.000000 |
| OS-SRV-USG:terminated_at | - |
| SH_DEV_NET network | 10.198.192.255 |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-25T03:51:03Z |
| flavor | b2c_web_1core (5) |
| hostId | a5239c63509fa00ab056ca701363538ecc0afe41d8f886f82b345b4d | <- 挂载一个
| id | b433b227-14ab-4157-8f08-362ad680e35e |
| image | Centos6.3_1.3 (7ec6eb66-b8a2-41e9-bbb5-b1e7ce1efed4) |
| key_name | - |
| metadata | {} |
| name | ceph-test02.sh.vclound.com |
| os-extended-volumes:volumes_attached | [{"id": "a5751c38-01c0-4f25-a02c-7d2a05d6ea36"}] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | bb0b51d166254dc99bc7462c0ac002ff |
| updated | 2015-06-25T03:52:05Z |
| user_id | 226e71f1c1aa4bae85485d1d17b6f0ae |
+--------------------------------------+----------------------------------------------------------+检测 cinder 状态
[root@sh-controller-129022 ~(keystone_admin)]# cinder list +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+ | 8516fb02-b578-4e57-9678-d30d2b0a6734 | in-use | None | 50 | None | false | 99d37977-a13a-4a8b-b8b1-e613a4959623 | | 9d8aa395-5e6a-411a-9f19-6375f29e9f9f | in-use | None | 50 | None | false | 99d37977-a13a-4a8b-b8b1-e613a4959623 | | a5751c38-01c0-4f25-a02c-7d2a05d6ea36 | in-use | None | 50 | None | false | b433b227-14ab-4157-8f08-362ad680e35e | +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
测试
测试云盘读写[root@ceph-test01 ~]# pvcreate /dev/vdc /dev/vdd Physical volume "/dev/vdc" successfully created Physical volume "/dev/vdd" successfully created [root@ceph-test01 ~]# vgcreate myvg /dev/vdc /dev/vdd Volume group "myvg" successfully created [root@ceph-test01 ~]# lvcreate -i 2 -n mylv -l 100%FREE myvg Using default stripesize 64.00 KiB Logical volume "mylv" created [root@ceph-test01 ~]# yum install -y xfsprogs.x86_64 > /dev/null 2>&1 [root@ceph-test01 ~]# mkfs.xfs /dev/myvg/mylv meta-data=/dev/myvg/mylv isize=256 agcount=16, agsize=1638256 blks = sectsz=512 attr=2, projid32bit=0 data = bsize=4096 blocks=26212096, imaxpct=25 = sunit=16 swidth=32 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal log bsize=4096 blocks=12800, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph-test01 ~]# mount /dev/myvg/mylv /mnt [root@ceph-test01 ~]# df -h 文件系统 容量 已用 可用 已用%% 挂载点 /dev/vda1 20G 1.1G 18G 6% / tmpfs 939M 0 939M 0% /dev/shm /dev/mapper/myvg-mylv 100G 33M 100G 1% /mnt
查询当前 ceph 情况
[root@sh-ceph-128213 ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 290T 290T 4615M 0 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 99157G 0 volumes 1 37220k 0 99157G 28 <- 留意当前默认下 ceph 只有 28 个对象
在 vm 上执行 操作
[root@ceph-test01 ~]# dd if=/dev/zero of=/mnt/1.img bs=1M count=700000 dd: 正在写入"/mnt/1.img": 设备上没有空间 记录了102309+0 的读入 记录了102308+0 的写出 107278180352字节(107 GB)已复制，982.231 秒，109 MB/秒
监控 ceph 存储空间
[root@sh-ceph-128212 var]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 290T 290T 337G 0.11 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 99033G 0 volumes 1 102399M 0.03 99033G 25622 <- dd 在客户端有 1 个文件, 但在 ceph 中会出现 [25622-28] 个文件 2.5 万个小文件
监控 ceph 物理存储空间
[root@sh-ceph-128212 ceph-0]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/mapper/centos-root 50G 1.8G 49G 4% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 18M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/sda2 494M 123M 372M 25% /boot /dev/mapper/centos-home 3.6T 33M 3.6T 1% /home /dev/sdb1 3.7T 3.8G 3.7T 1% /var/lib/ceph/osd/ceph-0 /dev/sdc1 3.7T 4.2G 3.7T 1% /var/lib/ceph/osd/ceph-1 /dev/sdd1 3.7T 4.3G 3.7T 1% /var/lib/ceph/osd/ceph-2 /dev/sdf1 3.7T 4.2G 3.7T 1% /var/lib/ceph/osd/ceph-3 /dev/sdg1 3.7T 4.1G 3.7T 1% /var/lib/ceph/osd/ceph-4 /dev/sdh1 3.7T 4.2G 3.7T 1% /var/lib/ceph/osd/ceph-5 /dev/sde1 3.7T 4.2G 3.7T 1% /var/lib/ceph/osd/ceph-6 /dev/sdi1 3.7T 3.7G 3.7T 1% /var/lib/ceph/osd/ceph-7 /dev/sdj1 3.7T 3.9G 3.7T 1% /var/lib/ceph/osd/ceph-8 /dev/sdk1 3.7T 4.6G 3.7T 1% /var/lib/ceph/osd/ceph-9 <- 参考已用空间
发现之前 dd 的文件是被打散, 分布在各个不同的磁盘中, 而且分布数量也不一定是一样, 没有在 /var/lib/ceph/osd/ceph* 目录中找到一个具体的文件名, 只看见一堆被打散的小文件
卸载云盘
[root@ceph-test01 ~]# rm -rf /mnt/1.img [root@ceph-test01 ~]# umount /mnt
检测 ceph 空间
[root@sh-controller-128022 cinder]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 290T 290T 304G 0.10 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 99044G 0 volumes 1 102399M 0.03 99044G 25622 <- 删除 1.img 文件 不会直接影响 ceph 磁盘容量
卸载 openstack 中云盘
[root@sh-controller-129022 ~(keystone_admin)]# nova volume-detach 99d37977-a13a-4a8b-b8b1-e613a4959623 8516fb02-b578-4e57-9678-d30d2b0a6734 [root@sh-controller-129022 ~(keystone_admin)]# nova volume-detach 99d37977-a13a-4a8b-b8b1-e613a4959623 9d8aa395-5e6a-411a-9f19-6375f29e9f9f
删除 openstack 云盘
[root@sh-controller-129022 ~(keystone_admin)]# cinder delete 8516fb02-b578-4e57-9678-d30d2b0a6734 [root@sh-controller-129022 ~(keystone_admin)]# cinder delete 9d8aa395-5e6a-411a-9f19-6375f29e9f9f
查询 ceph 存储空间
[root@sh-controller-128022 cinder]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 290T 290T 15961M 0 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 99131G 0 volumes 1 37220k 0 99131G 24 <- 之前产生的 1img 的对象被删除了 [root@sh-ceph-128212 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/mapper/centos-root 50G 1.8G 49G 4% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 18M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/sda2 494M 123M 372M 25% /boot /dev/mapper/centos-home 3.6T 33M 3.6T 1% /home /dev/sdb1 3.7T 58M 3.7T 1% /var/lib/ceph/osd/ceph-0 <-参考, 磁盘也恢复了之前大小 /dev/sdc1 3.7T 62M 3.7T 1% /var/lib/ceph/osd/ceph-1 /dev/sdd1 3.7T 55M 3.7T 1% /var/lib/ceph/osd/ceph-2 /dev/sdf1 3.7T 61M 3.7T 1% /var/lib/ceph/osd/ceph-3 /dev/sdg1 3.7T 63M 3.7T 1% /var/lib/ceph/osd/ceph-4 /dev/sdh1 3.7T 56M 3.7T 1% /var/lib/ceph/osd/ceph-5 /dev/sde1 3.7T 56M 3.7T 1% /var/lib/ceph/osd/ceph-6 /dev/sdi1 3.7T 63M 3.7T 1% /var/lib/ceph/osd/ceph-7 /dev/sdj1 3.7T 59M 3.7T 1% /var/lib/ceph/osd/ceph-8 /dev/sdk1 3.7T 60M 3.7T 1% /var/lib/ceph/osd/ceph-9
证明, 删除云盘能够会自动释放空间
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- openstack 管理三十九 - cinder 连接多个 ceph 存储方法
- openstack 管理二十三 - nova compute 连接 ceph 集群
- 管理蓝牙SDP记录和游戏服务器之间连接
- 理解 OpenStack + Ceph （8）: 基本的 Ceph 性能测试工具和方法
- openstack 命令行管理十三 - 外部网络[外部连接专用]管理 (备忘)
- 实践作业1：测试管理工具实践（小组作业）每日任务记录2
- 测试一下数据库记录客户端上一个跳转的连接地址
- 记一次VNC远程连接Linux问题解决记录（5900端口测试、KDE桌面安装）
- 测试管理之小记录
- 用c++编学生管理系统，连接到sql，在显示记录按钮点后出现错误
- 电话记录管理的制作1（连接数据库的方法们）
- 记一次VNC远程连接Linux问题解决记录（5900端口测试、KDE桌面安装）
- openstack controller ha测试环境搭建记录（六）——配置keystone
- openstack controller ha测试环境搭建记录（十）——配置neutron（控制节点）
- openstack controller ha测试环境搭建记录（十三）——配置cinder（控制节点）
- openstack controller ha测试环境搭建记录（十五）——创建实例
- openstack controller ha测试环境搭建记录（二）——配置corosync和pacemaker
- openstack连接ceph不成功
- 连接oracle数据库，执行ssh命令，然后把查询结果反写数据库-批量处理管理测试设备