Low Level Design for Identification of the Guest OS Process from VMM：KVM

Low Level Design for Identification of the Guest OS Process from VMM：KVM

康华 ：主要从事 Linux 操作系统内核、虚拟机、Linux 技术标准、计算机安全、软件测试等领域的研究与开发工作，曾就职 MII-HP 软件实验室 、瞬联软件公司/MOTOROLA，现就职于Lenovo研究院 。其所合写的Linux专栏见http://www.csdn.net/subject/linux/。 如果需要可以联系通过 kanghua151@msn.com （MSN）联系我.

Background

In last article, I have given the mechanism description for identification of Guest OS for VMM, This time I implemented a prototype to verify it. I choose KVM as my test platform, for KVM is more clear and easy debug.(it is a module for Linux kernel) The goal of prototype is that user can appoint the certain guest OS to trace, and can get guest processes info at running time. Of cause guest OS do need any modification.

KVM Modify

The mainly medication of kvm is that : some interface for user operation the trace and a hook for trace modules to register their own call back function. (For different OS will use different trace modules )

1 Set a Hook in handler_cr ——ToTraceCr3 (vm process id) to intercept and capture Guest Os access CR3 register. And then, add & export a register function in vmx.c for my own module (gptrace.ko) to hook callback function: ToTraceCr3. (When kvm handle GOS exit result from EXIT_REASON_CR_ACCESS, it will call callback function “handler cr” to handle cr register access from GOS)
typedef int (Cr3TraceFunc)(struct kvm *kvm ,int reg);
Register_cr3_trace(Cr3TraceFunc *func)
{
Hook_ToTraceCr3 = func;
}
EXPORT_SYMBOL_GPL(Register_cr3_trace);

2 Add vm_id (pid of qemu process that Guest OS running in) in the kvm structure for identifying the vm process, so that we can trace the given vm.

3 Add trace_enable (vm trace enable)flag in the kvm structure to indication whether this vm trace is on.

4 Add a opaque pointer field in kvm structure for storing the struct gptrv(see below)

5 Add some ioctl entry for vm monitor operations
KVM_ENABLE_VM_TRACE (ioctl for /dev/kvm)
KVM_GET_VM_GPTRV(ioctl for /dev/kvm)
KVM_SET_VMID(for for vcpu fd)

Date Structure

The most important data structure is “struct gptrv” for per vm to store process trace record .gptrv is abbreviation for “guest process trace record vector” ,which data structure includes the guest process running trace record.
struct gptrv //guest process trace record vector
Struct gptritem{
unsigned long gpptaddr; //guest process page table directory address
unsigned long gpdaddr; //guest process descriptor address
int gpid; //guest process id
char gpname[30]; //guest process’s name
__64 begin_time; // first record time
__64 last_time; // last record time
}gptr[MAX_TRACE_NUMBER]; //how many process that can be record
int last; //last time running process
int curr; //current running process
}
Anther need to explain data is : PID_OFFSE/COMM_OFFSET ,which is the pid/comm field offset in the process descriptor, we will find out pid/comm value from guest process descriptor memory region in VMM. Because different version of kernel’s process descriptor structure is not same totally, so pid/comm. Field offset may be different in variety kernel version. Later I will try to find a way to detect kernel version from VMM, and choose the correct offset automatically.

Operations & Interface

1 Set the vm process id when vm launch
It is the KVM_SET_VMID ioctl entry for kvm_vm_ioctl. When Guest OS launch(actually qemu launch) , we should set qemu process id to it’s according kvm structure. I add it in kvm_qemu_create_context function . like that , ioctl(kvm_context->vm_fd, KVM_SET_VMID ,getpid())

2 Swith on/off the trace function on the given vm (interface)
It is the KVM_ENABLE_VM_TRACE ioctl entry for kvm_vm_ioctl for enabling or disabling trace by pass the given vm enable trace flag (0 means switch off;1 otherwise ).

3 List the process trace record of the given vm (interface)
It is KVM_GET_VM_GPTRV ioctl entry for kvm_vm_ioctl for fetch the process trace record.
4 Kill the given process of the given vm (interface) --- not implement now
Add a new ioctl entry for kvm_vm_ioctl for accept given process id of guest vm
Set “KILL” signal to the signal offset of process descriptor of given process
Set TIF flag to the thread info descriptor of given process

5 Trace the guest process switch
Check if the vm’s trace enable flag
IF No, return;
Check if the vm’s opaque is NULL
IF No, create gptrv for vm.
Check whether the value of CR3 has been record in previous round..
IF Yes, update last index.
Update it’s timestamp
Record the gpdaddr into the last process trace record.
Record the gpid/gpname into the last process trace record.
Update last process index
IF No,
Record the gpptaddr into current process trace record and update it’s timestamp
Record the gpdaddr into the last process trace record.
Record the gpid/gpname into the last process trace record.
Update last process index and current process index.

PS:
1 gpdaddr is obtained by two step :1 rsp&0xffff000 (result is the guest virtual address of the pointer to process descriptor ; fffff000 for 4k kernel stack;ffffe000 for 8k ) ;2 kvm_read_guest (vcpu, rsp&0xffff000,4, gpdaddr) (read the guest virtual address of process descriptor)
2gpid/gpname is obtained by one step: kvm_read_guest(vcpu,gpdaddr+PID_OFFSET/COMM_OFFSET,length,gptr->ptrt[].gpid/gpname)

6 Insmod our module
Register the my own callback function:ToTraceCr3 to ToTraceCr3 hook in kvm-intel.ko

Limitation

1 Now pid/comm. Offset is given by hard code. only availability for FC6. if you want to run anther Linux distribution or custom kernel , it can not work (now it is only a prototypeJ) .I will try to find a way to detect kernel version from VMM, and choose the correct offset automatically in next vertion.
2 Only can record 270 (MAX_TRACE_NUMBER ) live processes traces. For ioctl only can transfer the data less than 4 page size (16kb).

HOWTO

1 Download the source code
2 Enter the director of directory:kvm-24
3 execute (the same as build KVM):
./make clean
./configure –prefix=/usr/local/kvm
./make
./make install
./modprobe kvm-intel ( I use intel VT cpu)
./modprobe gptrace
4 launch a VM
./use/local/kvm/bin/qemu <your vm image>

5 For simpleness, I implement user tools for trace operation under <kvm>/user directory and using a modified main.c file under it. You can execute “make” under <kvm>/user and kvmctl is our own user tools now.
You can do below by it:
1 Swith on trace on a given vm
./kvmctl –E vmid (vmid is qemu process id ,you can get it form ps –aux|grep –I qemu)
2 Switch off trace on a given vm
./kvmctl –D vmid
3 Show guest process info of given vm
./kvmctl –S vmid
4 display usage of that tools
./kvmctl -h

补记: 我在sourceforge申请了一个名为VMM Guest OS Process Monitor的开源项目(http://vgpm.wiki.sourceforge.net/),欢迎对虚拟机有兴趣的朋友能来参加.