Linux运维昱笔记（十三）：基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移
2020-01-14 10:13
190 查看
基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移
NFS+DRBD+Keepalived+KVM动态迁移演示
学习环境：
| 主机名 | IP | 安装服务 |
|---|---|---|
| KVM01 | 192.168.1.112 | KVM |
| KVM01 | 192.168.1.113 | KVM |
| NFS | 192.168.1.111 | nfs+drbd+keepalived |
| NFS2 | 192.168.1.114 | nfs+drbd+keepalived |
VMware 虚拟机中 linux 开机前需要勾选Inter VT-x/EPT 或 AMD/RVI(v)
在CentOS7的系统光盘镜像中，已经提供了安装KVM所需软件，通过部署基于光盘镜像的本地yum源，直接使用yum安装所需软件即可。
[root@localhost ~] yum -y groupinstall -y "GNOME Desktop" #安装GNOME桌面环境 [root@localhost ~] yum -y install qemu-kvm #KVM模块 [root@localhost ~] yum -y install qemu-kvm-tools #KVM调试工具，可不安装 [root@localhost ~] yum -y install qemu-img #qemu组件，创建磁盘，启动虚拟机 [root@localhost ~] yum -y install bridge-utils #网络支持工具 [root@localhost ~] yum -y install libvirt #虚拟机管理工具 [root@localhost ~] yum -y install virt-manager #图形界面管理虚拟机 [root@localhost ~] yum -y install qemu-kvm qemu-kvm-tools virt-install qemu-img bridge-utils libvirt virt-manager
检测KVM模块安装是否成功
[root@localhost ~] lsmod | grep kvm kvm_intel 174841 0 kvm 578518 1 kvm_intel irqbypass 13503 1 kvm
注意：使用yum安装完桌面后，将系统默认的运行target修改为graphical.target。重启系统后进入图形界面，若不修改系统的默认运行target，重启时可能会出错。（init 5）
开启服务，安装完成后还需要开启libvirtd服务
[root@localhost ~] systemctl start libvirtd [root@localhost ~] systemctl enable libvirtd
设置KVM网络默认是NAT。
[root@KVM01 ~] tail -4 /etc/hosts 192.168.1.112 KVM01 192.168.1.113 KVM02 192.168.1.111 NFS 192.168.1.114 NFS2 [root@KVM01 ~] scp /etc/hosts 192.168.1.111:/etc [root@KVM01 ~] scp /etc/hosts 192.168.1.113:/etc [root@KVM01 ~] scp /etc/hosts 192.168.1.114:/etc
NFS SERVER：配置共享目录 /data/kvm-share
[root@NFS ~] cat /etc/exports
/data/kvm-share 192.168.1.112(rw,sync,no_root_squash)
/data/kvm-share 192.168.1.113(rw,sync,no_root_squash)
[root@NFS ~] mkdir -p /data/kvm-share
[root@NFS ~] systemctl start nfs
- DRBD部署
DRBD详细解说
实验初始配置：所有主机关闭防火墙与selinux
[root@localhost ~] iptables -F [root@localhost ~] systemctl stop firewalld [root@localhost ~] systemctl disable firewalld [root@localhost ~] setenforce 0 [root@localhost ~] sed -i '/SELINUX/ s/enforcing/disabled/g' /etc/sysconfig/selinux
[root@NFS ~]rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm [root@NFS ~]yum install -y drbd84-utils kmod-drbd84 [root@NFS ~]modprobe drbd [root@NFS ~]lsmod | grep drbd drbd 397041 0
- DRBD配置(NFS和NFS2)
[root@NFS ~]cp /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.bak
[root@NFS ~]vim /etc/drbd.d/global_common.conf
global {
usage-count yes;
}
common {
protocol C;
handlers {
}
startup {
wfc-timeout 240;
degr-wfc-timeout 240;
outdated-wfc-timeout 240;
}
disk {
on-io-error detach;
}
net {
cram-hmac-alg md5;
shared-secret "testdrbd";
}
syncer {
rate 30M;
}
}
[root@NFS ~]vim /etc/drbd.d/r0.res
resource r0 {
on NFS {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.111:7898; #端口可以自己定义
meta-disk internal;
}
on NFS2 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.114:7898;
meta-disk internal;
}
}
- 在两台机器上添加DRBD磁盘(NFS和NFS2)
在NFS机器上添加一块10G的硬盘作为DRBD，分区为/dev/sdb1，不做格式化，并在本地系统创建/data目录，不做挂载操作。
fdisk -l 磁盘 /dev/sdb：10.7 GB, 10737418240 字节，20971520 个扇区 Units = 扇区 of 1 * 512 = 512 bytes 扇区大小(逻辑/物理)：512 字节 / 512 字节 I/O 大小(最小/最佳)：512 字节 / 512 字节 fdisk /dev/sdb 依次输入"n->p->1->回车->回车->w" //分区创建后，再次使用"fdisk /dev/sdb"，输入p，即可查看到创建的分区. mkdir /data
- 在两台机器上分别创建DRBD设备并激活r0资源(NFS和NFS2)
[root@NFS ~]mknod /dev/drbd0 b 147 0 [root@NFS ~]drbdadm create-md r0 Writing meta data... initializing activity log NOT initialized bitmap New drbd meta data block successfully created. [root@NFS ~]drbdadm create-md r0 You want me to create a v08 style flexible-size internal meta data block. There appears to be a v08 flexible-size internal meta data block already in place on /dev/vdd1 at byte offset 10737340416 Do you really want to overwrite the existing v08 meta-data? [need to type 'yes' to confirm] yes //这里输入"yes" Writing meta data... initializing activity log NOT initialized bitmap New drbd meta data block successfully created. 启动drbd服务（注意：需要主从共同启动方能生效） [root@NFS ~]systemctl start drbd [root@NFS ~]ps -ef|grep drbd root 81611 7779 0 20:11 pts/0 00:00:00 grep --color=auto drbd [root@NFS ~]cat /proc/drbd version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018- 11-03 01:26:55 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10484380 [root@NFS2 ~]: cat /proc/drbd version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018- 11-03 01:26:55 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10484380 由上面两台主机的DRBD状态查看结果里的ro:Secondary/Secondary表示两台主机的状态都是备机状态，ds是磁盘状态，显示的状态内容为“不一致”，这是因为DRBD无法判断哪一方为主机，应以哪一方的磁盘数据作为标准。
- 接着将NFS主机配置为DRBD的主节点
[root@NFS ~]drbdsetup /dev/drbd0 primary --force [root@NFS ~] cat /proc/drbd version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:163528 nr:0 dw:0 dr:165648 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10320852 [>....................] sync'ed: 1.7% (10076/10236)M finish: 0:05:15 speed: 32,704 (32,704) K/sec [root@NFS2 ~]# cat /proc/drbd version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----- ns:0 nr:1221508 dw:1221508 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:9262872 [=>..................] sync'ed: 11.7% (9044/10236)M finish: 0:03:51 speed: 40,056 (34,900) want: 41,040 K/sec ro在主从服务器上分别显示 Primary/Secondary和Secondary/Primary ds显示UpToDate/UpToDate 表示主从配置成功
- 挂载DRBD(NFS)
先格式化/dev/drbd0 [root@ NFS ~] mkfs.xfs /dev/drbd0 meta-data=/dev/drbd0 isize=512 agcount=4, agsize=655274 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=2621095, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 创建挂载目录，然后执行DRBD挂载 [root@ NFS ~] mkdir /data [root@ NFS ~] mount /dev/drbd0 /data [root@NFS ~] df -Th 文件系统 类型 容量 已用 可用 已用% 挂载点 /dev/drbd0 xfs 10G 33M 10G 1% /data 特别注意： server01节点上不允许对DRBD设备进行任何操作，包括只读，所有的读写操作只能在server01节点上进行。 只有当server01节点挂掉时，server02节点才能提升为server01节点
- DRBD主备故障切换测试(可不做)
模拟NFS节点发生故障，NFS2接管并提升为NFS
下面是在NFS主节点上操作记录
[root@ NFS ~] cd /data/
[root@ NFS data] touch test{1..5}
[root@NFS data] ls
test1 test2 test3 test4 test5
[root@ NFS data] cd ../
[root@ NFS /] umount /data
[root@ NFS /] drbdsetup /dev/drbd0 secondary //将Primary主机设置为DRBD的备节点。在实际生产环境中，直接在NFS2主机上提权（即设置为主节点）即可。
[root@ NFS /] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:10497043 nr:0 dw:12663 dr:10489714 al:16 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
注意：这里实际生产环境若NFS主节点宕机，在NFS2状态信息中ro的值会显示为Secondary/Unknown,只需要进行DRBD提权操作即可。
下面是在NFS2 备份节点上操作记录
先进行提权操作，即将server02手动升级为DRBD的主节点
[root@NFS2 ~] drbdsetup /dev/drbd0 primary
[root@NFS2 ~] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:10497043 dw:10497043 dr:2120 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
然后挂在drbd
[root@NFS2 ~] mount /dev/drbd0 /data
[root@NFS2 ~] df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
/dev/mapper/centos-root xfs 50G 4.0G 47G 8% /
devtmpfs devtmpfs 894M 0 894M 0% /dev
tmpfs tmpfs 910M 0 910M 0% /dev/shm
tmpfs tmpfs 910M 9.9M 900M 2% /run
tmpfs tmpfs 910M 0 910M 0% /sys/fs/cgroup
/dev/sr0 iso9660 4.3G 4.3G 0 100% /media/cdrom
/dev/sda1 xfs 1014M 179M 836M 18% /boot
/dev/mapper/centos-home xfs 27G 37M 27G 1% /home
tmpfs tmpfs 182M 0 182M 0% /run/user/0
/dev/drbd0 xfs 10G 33M 10G 1% /data
发现DRBD挂载目录下已经有了之前在远程NFS主机上写入的内容
[root@NFS2 ~] ls /data
test1 test2 test3 test4 test5
NFS部署
- 在两台主机上安装nfs(NFS和NFS2)
[root@NFS ~] yum install rpcbind nfs-utils [root@NFS ~] vim /etc/exports /data/kvm-share 192.168.1.112(rw,sync,no_root_squash) /data/kvm-share 192.168.1.113(rw,sync,no_root_squash) ~ [root@ NFS ~] systemctl start rpcbind [root@ NFS ~] systemctl start nfs
- 在两台kvm主机上验证(KVM01和KVM02)
[root@KVM01 ~] showmount -e 192.168.1.114 Export list for 192.168.1.114: /data/kvm-share 192.168.1.113,192.168.1.112 [root@KVM01 ~] showmount -e 192.168.1.111 Export list for 192.168.1.111: /data/kvm-share 192.168.1.113,192.168.1.112 [root@KVM02 /] showmount -e 192.168.1.111 Export list for 192.168.1.111: /data/kvm-share 192.168.1.113,192.168.1.112 [root@KVM02 /] showmount -e 192.168.1.114 Export list for 192.168.1.114: /data/kvm-share 192.168.1.113,192.168.1.112
keepalived部署
- 两台机器yum安装keepalived（NFS 和NFS2）
[root@NFS /] yum -y install keepalived [root@NFS2 /] yum -y install keepalived
- 更改NFS的配置文件并启动
[root@NFS ~] vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id DRBD_MASTER
}
vrrp_script chk_nfs {
script "/etc/keepalived/check_nfs.sh"
interval 2
weight -5
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 51
priority 100
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nfs
}
notify_master /etc/keepalived/notify_master.sh //当此机器为keepalived的master角色时执行这个脚本
virtual_ipaddress {
192.168.1.100
}
}
[root@NFS ~] vim /etc/keepalived/check_nfs.sh
#!/bin/sh
###检查nfs可用性：进程和是否能够挂载
/usr/bin/systemctl status nfs &>/dev/null
if [ $? -ne 0 ];then
###卸载drbd设备
umount /dev/drbd0
###将drbd主降级为备
drbdadm secondary r0
#关闭keepalived
/usr/bin/systemctl stop keepalived
fi
[root@NFS ~] chmod 755 /etc/keepalived/check_nfs.sh
[root@NFS ~] vim /etc/keepalived/notify_master.sh
#!bin/bash
/usr/sbin/drbdadm primary r0 &>/dev/null
/usr/bin/mount /dev/drbd0 /data &>/dev/null
/usr/bin/systemctl restart nfs &>/dev/null`
[root@NFS ~] chmod 755 /etc/keepalived/notify_master.sh
[root@NFS ~] systemctl restart keepalived
查看vip
[root@NFS ~] ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.1.111/24 brd 192.168.1.255 scope global noprefixroute ens32
inet 192.168.1.100/32 scope global ens32
- 更改NFS2的配置文件
[root@NFS2 ~] vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id DRBD_MASTER
}
vrrp_script chk_nfs {
script "/etc/keepalived/check_nfs.sh"
interval 2
weight -5
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nfs
}
notify_master /etc/keepalived/notify_master.sh //当此机器为keepalived的master角色时执行这个脚本
virtual_ipaddress {
192.168.1.100
}
}
[root@NFS2 ~] vim /etc/keepalived/check_nfs.sh
#!/bin/sh
###检查nfs可用性：进程和是否能够挂载
/usr/bin/systemctl status nfs &>/dev/null
if [ $? -ne 0 ];then
###卸载drbd设备
umount /dev/drbd0
###将drbd主降级为备
drbdadm secondary r0
#关闭keepalived
/usr/bin/systemctl stop keepalived
fi
[root@NFS2 ~] chmod 755 /etc/keepalived/check_nfs.sh
[root@NFS2 ~] vim /etc/keepalived/notify_master.sh
#!bin/bash
/usr/sbin/drbdadm primary r0 &>/dev/null
/usr/bin/mount /dev/drbd0 /data &>/dev/null
/usr/bin/systemctl restart nfs &>/dev/null
[root@NFS2 ~] chmod 755 /etc/keepalived/notify_master.sh
[root@NFS2 ~] systemctl restart keepalived
验证nfs高可用
- 模拟nfs服务停机
[root@NFS ~] systemctl stop nfs [root@NFS ~] ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.1.111/24 brd 192.168.1.255 scope global noprefixroute ens32 //VIP不在了 [root@NFS ~] cat /proc/drbd version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:10501164 nr:6171 dw:22955 dr:10496179 al:16 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 //drbd变为Secondary/Primary节点
- 查看NFS2(有了vip 变为primary/secondary)
[root@NFS2 ~] ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.1.114/24 brd 192.168.1.255 scope global noprefixroute ens32 inet 192.168.1.100/32 scope global ens32 [root@NFS2 ~] cat /proc/drbd version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:6171 nr:10501164 dw:10507335 dr:7581 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 [root@NFS2 ~] ls /data/ aaaa test1 test2 test3 test4 test5
KVM
两台KVM连接共享存储：双击localhost（qemu） 点击 存储 点击 加号
源主机：192.168.1.112 在共享存储上创建虚拟机
过程比较慢，需要耐心等待。
KVM迁移：打开virt-manager 选择文件 选择Add Connection
源主机连接目标主机
[root@KVM01 ~]# yum -y install openssh-askpass
[root@KVM02 ~]# yum -y install openssh-askpass
[root@KVM01 ~]# virsh shutdown centos7.0
域 centos7.0 被关闭
[root@KVM01 ~]# virsh edit centos7.0
[root@KVM01 ~]# virsh start centos7.0
域 centos7.0 已开始
[root@KVM01 ~]# virsh list --all
Id 名称 状态
2 centos7.0 running
迁移成功
- 点赞 1
- 收藏
- 分享
- 文章举报
火锅#2019
发布了17 篇原创文章 · 获赞 27 · 访问量 900
私信
关注
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Linux运维笔记-文档总结-Apache基本配置的介绍和语言支持，虚拟主机，访问控制的实现
- CentOS 7运维管理笔记(6)----Apache 基于 IP 的虚拟主机配置
- 20171107L09-01老男孩Linux运维实战培训-Lamp系列-Apache服务生产实战应用指南04-基于IP的虚拟主机实战配置
- 网站运维：windows、Linux设置Apache的虚拟主机实现本地网址访问
- kvm 静态迁移、动态迁移（基于nfs和后端镜像）
- KVM基于NFS的动态迁移
- CentOS 7运维管理笔记(7)----Apache 基于端口的虚拟主机配置
- linux下nginx实现虚拟主机（3种方法：基于域名、基于端口、基于ip地址）
- CentOS 7运维管理笔记(8)----Apache基于域名的虚拟主机配置
- 虚拟化学习笔记-KVM 虚拟机在物理主机之间迁移的实现（p2p）
- CentOS 7运维管理笔记(8)----Apache基于域名的虚拟主机配置
- 基于Linux的虚拟主机搭建
- linux http虚拟主机的实现（3种方法）
- Linux运维昱笔记（十三）：Memcached介绍和小集群部署实验
- apache_php_tomcat基于主机名的多虚拟主机整合笔记
- IP及主机名设置_Linux运维笔记
- KVM 虚拟机在物理主机之间迁移的实现
- Linux高可用(HA)集群笔记heartbeat+drbd+nfs
- Linux高可用(HA)集群笔记heartbeat+drbd+nfs
- Linux工程实践学习笔记——基于主机系统的多客户即时通讯/聊天室 Day(1)