Xen 启动客户虚拟机的过程分析

Thanks to paravirtualisation, many of the lowlevel

steps involved in a conventional x86 OS boot process (such as

interactions with the BIOS, and the use of 16-bit real mode) can be

elided.

The main steps in the domain building process are shown in Figure 1, with privileged operations (as defined later in §3.1) shown as bold arrows. Firstly, the kernel image (and, optionally, an initial ramdisk) are loaded from disk into the domain builder’s memory. The image is parsed in order to extract the executable code region, and obtain any parameter values. Following this, the builder requests that the hypervisor allocates physical memory for the new VM. This is not depicted as a privileged operation, because it does not affect the confidentiality or integrity of any VM, but it can lead to denial of service, so its use should be restricted. The kernel image is then loaded into the new VM, using a direct foreign mapping. Following this, the initial page tables for the new VMare calculated and written into the new VM, using multiple direct foreign mappings. Finally, the new VM is launched, by setting the virtual CPU registers to a start-of-day configuration, and unpausing the VM.

The domain builder is implemented by the xc linux build function and its callees in libxc, a C library that provides low-level control over VMs. It is implemented in an object-oriented fashion, passing a struct xc dom image object between steps to communicate intermediate results. Direct foreign mappings are implemented by the xc map foreign range function, and setting the virtual CPU registers is performed by the xc set vcpucontext function. The libxc library is linked into the xend management daemon, which is implemented in Python. xend implements the Xen API, which is an XML-RPC-based API that several tools use to manage a Xen-based system [35]

The Dom0 kernel is included in the TCB, because Dom0 is privileged, and the kernel can make privileged hypercalls. The domain builder process must also be included in the TCB, because it writes to the address space of the guest VM and configures its processor state: if it should malfunction, the security of S could be compromised.

However, the domain builder accesses the guest address space using the privileged command driver, which is exposed to all processes that run with administrator privileges in Dom0.

xl create is the command used in dom0 to initiate the creation

of a VM. Its main execution steps are the following: (1)

Checking the consistency of the VM configuration parameters

(e.g. the existence of the VM file system). This step is

performed at the dom0 level. (2) Creating the VM management

data structures to make the VM manageable during its

lifetime. This step is performed both at the dom0 and hypervisor

levels. (3) Allocating resources to the VM (e.g. memory

pages are granted to the VM). This step is performed at

the hypervisor level. (4) Finally, executing the VM’s kernel/-

bootloader. The traditional boot process of an OS starts here.

This step is performed at the VM level.

For its part, the destruction of a VM is triggered using

either xl shutdown/destroy in dom0 or halt in the VM

itself. It follows the inverse path of the initialization process:

(1) VM’s operating system shutdown, performed at the VM

level. (2) VM resource freeing, performed at the hypervisor

level. (3) Freeing VM management data structures, performed

at both at the hypervisor and dom0 levels.

Intuitively, one can see that operations which are performed

at the VM level are out of the cloud provider’s

purview. Indeed, they depend on both the VM kernel (built

by its owner) and assigned resources (booked by its owner).

This paper focuses on the steps performed either at the hypervisor

or dom0 level.

本文档分析了xen启动时创建dom0的基本过程，并进行了一些简单的测试，记录了测试结果。

@ xen3.3.0/xen/arch/x86/setup.c

—- line 408 __start_xen(unsigned long mbi_p)

@ line 414, multiboot_info_t *mbi = __va(mbi_p);

mbi = multiboot_info, 类型multiboot_info_t, 位于include/xen/multiboot.h

@ line 415, modile_t mod = (module_t )__va(mbi->mods_addr);

module_t是一个位于multiboot.h中的struct，包含mod_start, mod_end, string, reserved等4个成员，其中mod_start, mod_end是地址, string是module [file] [para]这里的para字段(参数, 如果没有的话，就是NULL)，比如kernel的参数就是这么搞出来的cmdline(大概在997行的位置)。

这个mod是一个指针，取值于mbi_p的成员mods_addr，其实就对应了一个数组，这数组的每一项应该就是顺序地对应着grub启动xen的时候，每一个module行对应的文件载入内存后的信息，包括起始和结束地址，还有附加的参数。

@ line 989, dom0 = domain_create(0, 0, DOM0_SSIDREF);

这里调用create_domain函数，分配一个struct domain的空间，对其进行初始化，然后返回这个地址存入dom0。我们可以仿照这个方式创建另一个struct domain, 比如domB。domain_create函数位于commom/domain.c，声明于include/xen/sched.h，其原型为

struct domain *domain_create(

domid_t domid, unsigned int domcr_flags, ssidref_t ssidref)

第一个参数是domid，第二个参数是dom的类型，具体参见sched.h +336(好像有点出入?)，第三个参数是用于xen的ssid reference，具体用途和生成方式不明。DOM0_SSIDREF被定义为0x0。

@ line 990, if ( (dom0 == NULL) || (alloc_vcpu(dom0, 0, 0) == NULL) )

这里检查989行的domain_create是否成功，如果成功，则试图给该domain分配一个vcpu。

@ line 997, cmdline = … mod[0].string …

从mod指向的module_t数组的第一个元素中取出kernel的启动参数。默认第一个是内核，第二个是initrd。接下来，如果参数不为空，就将参数进行一定处理，附加一些额外的参数，诸如acpi等，这一段可以忽略, 直到1027。

@ line 1028 if ( (initrdidx > 0) && (initrdidx < mbi->mods_count) )….

initrdidx的初始值为1, 如果这个值小于grub的menu.lst里面对应的module的行数(kernel + initrd，默认应该需要2行module)，那么说明是有initrd的，那么就从mod[initrdidx]，也就是mod[1]，也就是第二个module语句对应的module_t中取出initrd被载入内存后的起始地址和总长度。

@ 1043, if (construct_dom0(dom0, …

这里是实际建立一个domain的函数了，传入的参数包括一个有效struct domain结构的指针，kernel起始地址和长度，initrd的起始地址和长度，还有kernel的启动参数。返回值为0表示启动成功，否则是失败。

@1061, domain_unpause_by_systemcontroller(dom0);

这个函数激活dom0对应的domain的每一个vcpu，使得domain可以实际运行起来。

—-以上为启动dom0涉及到的过程。

—-下面是进行的一些测试。

测试结果见图simple_test1.png

点击在新窗口中浏览此图片

首先是对grub的一些处理:

cd/boot echo 123 > mini-os.gz

$ vi grub/menu.lst

启动xen的几行大致是

title Ubuntu 9.04, kernel 2.6.26.1-xen-686

root (hd0,0) #因为测试机上/boot是单独的一个分区，位于(hd0,0)

kernel /xen.gz

module /vmlinuz-2.6.26-1-xen-686 root=/dev/sda5 ro splash quiet

module /initrd.img-2.6.26-1-xen-686

在其后又加入了一行

module /mini-os.gz

然后对setup.c进行了一些修改，包括：

在995行处加入一段初始化domB的代码

domB = domain_create(0x7777, 0, 0x7777);

if ( (domB == NULL) || (alloc_vcpu(domB, 0, 0) == NULL) )

panic(“Error creating domain B\n”);

domB->is_privileged = 1;

domB->target = NULL;

printk(“OK Creating Domain B\n”);

其中domB定义和声明的位置在dom0定义和声明的位置下方，分别是common/domain.c和include/xen/sched.h。

从途中可以看到，创建domB和为domB分配VCPU的过程都是正确的。

其中的0x7777是随便填的一个数字，没有特殊含义，也不保证此后可以正确运行。

在997行之后加入了一行

printk(“—–JUST A TEST\ncmdline = %s\n”, cmdline);

由图可以看出，cmdline的确指向了kernel的启动参数。

在1035行的位置加入如下几行，用于打印module的行数、第三个module行对应文件被载入后的地址、以及其前三个字符（及其ASCII码值）

printk(“mods_count = %d\n”, mbi->mods_count);

printk(“–start char @ [%x] = %c(%d), %c(%d), %c(%d)\n”,

mod[2].mod_start,

((char)mod[2].mod_start), ((char)mod[2].mod_start),

((char)mod[2].mod_start+1), ((char)mod[2].mod_start+1),

((char)mod[2].mod_start+2), ((char)mod[2].mod_start+2)

);

由图可以看出mods_count = 3，前三个字符是123，和mini-os.gz的内容完全一致。

在1043行前加入

printk(“Before construt_dom0\n”);

在1050行加入

printk(“After construt_dom0\n”);

EARLY_FAIL(“”);

由图可以看出construct_dom0的作用范围。EARLY_FAIL是一个定义于setup.c中的一个macro，调用printk输出调试信息以后，无限循环调用halt()。这样可以在此处停下来，看到所有的调试信息。

呼—-今天终于有了一点实质性的进展 :)