Xen has problems working with suspend to RAM (S3), and our solutions: KVM + Xen

I tested Xen v4.1 + Linux kernel 3.2.0-68 in Ubuntu 12.04.  Suspend is okay, but resumes did not.

My labmate Nilton found that: 

Xen 4.x has another issue that prevents it from resuming from S3. This threadhttp://lists.xenproject.org/archives/html/xen-devel/2013-06/msg02947.html
suggests Xen 4.2.3 with a set of patches on Linux 3.8.8 to fix that.

Our final solution is use KVM + Xen.  We run KVM as the hypervisor.  The reason of using KVM is that Ubuntu has better support for Suspend to RAM.  
We then start a KVM guest and installed Xen 3.4 on it. It didn't work at first.  Xen panics at probing a MSR feature of the CPU.  We commented out a line in Xen to disable the MSR probe!  Then it works:  (1) Xen can
boot as a KVM guest,  (2) we start new jettison VMs on Xen,  (3) VM suspend/resume works fine. 
Below is the panic message before commenting out the MSR probe line: 

(XEN) Xen version 3.4.0 (root@) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Wed Sep  3 09:55:51 EDT 2014 

(XEN) Latest ChangeSet: Wed Feb 24 10:59:37 2010 +0000 29409:a705ab9e2760 

(XEN) Command line: com1=38400,8n1 console=com1 nolapic 

(XEN) Video information: 

(XEN)  VGA is text mode 80x25, font 8x16 

(XEN) Disc information: 

(XEN)  Found 1 MBR signatures 

(XEN)  Found 1 EDD information structures 

(XEN) Xen-e820 RAM map: 

(XEN)  0000000000000000 - 000000000009dc00 (usable) 

(XEN)  000000000009dc00 - 00000000000a0000 (reserved) 

(XEN)  00000000000f0000 - 0000000000100000 (reserved) 

(XEN)  0000000000100000 - 00000000dfffd000 (usable) 

(XEN)  00000000dfffd000 - 00000000e0000000 (reserved) 

(XEN)  00000000feffc000 - 00000000ff000000 (reserved) 

(XEN)  00000000fffc0000 - 0000000100000000 (reserved) 

(XEN)  0000000100000000 - 0000001f99500000 (usable) 

(XEN) System RAM: 128916MB (132010600kB) 

(XEN) ACPI: RSDP 000FDA30, 0014 (r0 BOCHS ) 

(XEN) ACPI: RSDT DFFFD210, 0034 (r1 BOCHS  BXPCRSDT        1 BXPC        1) 

(XEN) ACPI: FACP DFFFFC10, 0074 (r1 BOCHS  BXPCFACP        1 BXPC        1) 

(XEN) ACPI: DSDT DFFFD630, 2589 (r1   BXPC   BXDSDT        1 INTL 20100528) 

(XEN) ACPI: FACS DFFFFBC0, 0040 

(XEN) ACPI: SSDT DFFFD3A0, 0283 (r1 BOCHS  BXPCSSDT        1 BXPC        1) 

(XEN) ACPI: APIC DFFFD290, 009A (r1 BOCHS  BXPCAPIC        1 BXPC        1) 

(XEN) ACPI: HPET DFFFD250, 0038 (r1 BOCHS  BXPCHPET        1 BXPC        1) 

(XEN) Domain heap initialised 

(XEN) Processor #0 6:2 APIC version 20 

(XEN) Processor #1 6:2 APIC version 20 

(XEN) Processor #2 6:2 APIC version 20 

(XEN) Processor #3 6:2 APIC version 20 

(XEN) Processor #4 6:2 APIC version 20 

(XEN) Processor #5 6:2 APIC version 20 

(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs 

(XEN) Using scheduler: SMP Credit Scheduler (credit) 

(XEN) Detected 2400.051 MHz processor. 

(XEN) Xen BUG at vmcs.c:87 

(XEN) ----[ Xen-3.4.0  x86_64  debug=n  Not tainted ]---- 

(XEN) CPU:    0 

(XEN) RIP:    e008:[<ffff828c8019f42a>] vmx_cpu_up+0x1aa/0x760 

(XEN) RFLAGS: 0000000000010086   CONTEXT: hypervisor 

(XEN) rax: 00000000fffffdff   rbx: 0000000000181000   rcx: 0000000000000483 

(XEN) rdx: 0000000000000200   rsi: 0000000000000200   rdi: 0000000000000001 

(XEN) rbp: ffff828c80277f28   rsp: ffff828c80277d68   r8:  00000000b2998688 

(XEN) r9:  000000000000003f   r10: 0000000011e57ed0   r11: 000000008028fea0 

(XEN) r12: ffff828c801f8c88   r13: 000000000000000c   r14: 0000000000000005 

(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000020b0 

(XEN) cr3: 00000000df87c000   cr2: 0000000000000000 

(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008 

(XEN) Xen stack trace from rsp=ffff828c80277d68: 

(XEN)    0000000000000010 0000000000000001 0000000000007ff1 0000000000000000 

(XEN)    ffff8300ffe00000 ffff8300df881ff0 000ffffffffff000 0000000000000800 

(XEN)    ffff828c80277f28 ffff828c801f8c88 000000000000000c 0000000000100000 

(XEN)    ffff83000008bf60 ffff828c801a3191 0000062300000800 0000000000000000 

(XEN)    ffff828c801f8c80 ffff828c8017e6cf 8080202100000800 00000623078bfbfd 

(XEN)    ffff828c80277f28 ffff828c801f8c80 ffff828c801f8c88 ffff828c8017e392 

(XEN)    ffff828c80277f28 ffff831f994f6000 0000000000000000 ffff828c8021f3d1 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 00000000002ba3c0 

(XEN)    ffffff8c8027c000 0008bf6000000000 000000000008bf60 0000000000000000 

(XEN)    ffff83000008bfc0 00000000074a99c0 0000000000000000 ffffffff00000000 

(XEN)    0000000800000000 000000010000006e 0000000000000003 00000000000002f8 

(XEN)    0000000000000000 0000000000000000 0000000000067e9c 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff828c801000b5 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000 

(XEN) Xen call trace: 

(XEN)    [<ffff828c8019f42a>] vmx_cpu_up+0x1aa/0x760 

(XEN)    [<ffff828c801a3191>] start_vmx+0xd1/0x1c0 

(XEN)    [<ffff828c8017e6cf>] init_intel+0x15f/0x310 

(XEN)    [<ffff828c8017e392>] identify_cpu+0xd2/0x250 

(XEN)    [<ffff828c8021f3d1>] __start_xen+0x16a1/0x27c0 

(XEN)    [<ffff828c801000b5>] __high_start+0xa1/0xa3 

(XEN)     

(XEN)  

(XEN) **************************************** 

(XEN) Panic on CPU 0: 

(XEN) Xen BUG at vmcs.c:87 

(XEN) ****************************************