lost and found ( for me ? )

KVM nested : how to confirm “nested” works

I sometimes wonder my nested KVM environment works correctly or not.
Here is my trial and error logs when checking nested VMs.

[ success case ]

KVM host: Fedora19 ( AMD Phenom(tm) II X4 925 Processor )
VM: Ubuntu13.04 ( disk path IDE, storage format qcow2 )

[ failed case ]

KVM host : Fedora 19 64bit (AMD Phenom(tm) II X4 925 Processor)
VM : CentOS 6.4 64bit

- on the KVM host

confirm nested option is enabled ( in case of Intel chip, option name is “kvm_intel” instad of “kvm_amd” )

KVM host CPU info
[root@localhost ~]# cat /etc/fedora-release;uname -ri;qemu-system-x86_64 -version;cat /proc/cpuinfo
Fedora release 19 (Schrödinger’s Cat)
3.10.7-200.fc19.x86_64 x86_64
QEMU emulator version 1.6.0, Copyright (c) 2003-2008 Fabrice Bellard
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Phenom(tm) II X4 925 Processor
stepping : 2
microcode : 0x10000db
cpu MHz : 1600.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate npt lbrv svm_lock nrip_save
bogomips : 5599.72
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

enable nested
[root@localhost ~]# cat /sys/module/kvm_amd/parameters/nested
1

- on the VM, check CPU flags

[root@localhost ~]# egrep '(svm|vmx)' /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm 3dnowext 3dnow up rep_good extd_apicid unfair_spinlock pni cx16 popcnt hypervisor cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw

- install kvm on the VM
[root@localhost ~]# yum groupinstall Virtualization
[root@localhost ~]# yum groupinstall "Virtualization Client"
[root@localhost ~]# yum install virt-install libvirt libvirt-python libvirt-client

reboot the VM
[root@localhost ~]# reboot

after booting, confirm kvm and kvm_intel or kvm_amd module has been loaded
on the VM

nnn, not enable nested.
on the VM
[root@localhost ~]# lsmod | grep kvm
kvm                   317376  0

kvm-clock: Using msrs 4b564d01 and 4b564d00
kvm-clock: cpu 0, msr 0:1c25681, boot clock
Booting paravirtualized kernel on KVM
kvm-clock: cpu 0, msr 0:2216681, primary cpu clock
kvm-stealtime: cpu 0, msr 220e840
Switching to clocksource kvm-clock
kvm: no hardware support

[root@localhost ~]# virt-host-validate
 QEMU: Checking for hardware virtualization                                 : PASS
 QEMU: Checking for device /dev/kvm                                         : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization)
 QEMU: Checking for device /dev/vhost-net                                   : PASS
 QEMU: Checking for device /dev/net/tun                                     : PASS
  LXC: Checking for Linux >= 2.6.26                                         : PASS

on the KVM host ( Fedora 19 )
[root@localhost hattori]# virt-host-validate
 QEMU: Checking for hardware virtualization                                 : PASS
 QEMU: Checking for device /dev/kvm                                         : PASS
 QEMU: Checking for device /dev/vhost-net                                   : PASS
 QEMU: Checking for device /dev/net/tun                                     : PASS
  LXC: Checking for Linux >= 2.6.26                                         : PASS

shutdown the VM to change vCPU configuration.
on the VM
[root@localhost ~]# init 0

edit VM’s xml file
on the KVM host
# virsh edit “vm domain”

   <pae/>
 </features>
 <cpu mode='host-passthrough'>
 </cpu>
 <clock offset='utc'/>



start the VM
on the KVM host
virsh # start centos6-vm1-ganeti01

on the VM
okay.
[root@localhost ~]# egrep -i '(svm|vmx)' /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm 3dnowext 3dnow up rep_good extd_apicid unfair_spinlock pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw npt nrip_save
[root@localhost ~]# virt-host-validate
 QEMU: Checking for hardware virtualization                                 : PASS
 QEMU: Checking for device /dev/kvm                                         : PASS
 QEMU: Checking for device /dev/vhost-net                                   : PASS
 QEMU: Checking for device /dev/net/tun                                     : PASS
  LXC: Checking for Linux >= 2.6.26                                         : PASS
[root@localhost ~]# dmesg | grep -i kvm
kvm-clock: Using msrs 4b564d01 and 4b564d00
kvm-clock: cpu 0, msr 0:1c25681, boot clock
Booting paravirtualized kernel on KVM
kvm-clock: cpu 0, msr 0:2216681, primary cpu clock
kvm-stealtime: cpu 0, msr 220e840
Switching to clocksource kvm-clock
kvm: Nested Virtualization enabled
kvm: Nested Paging enabled
[root@localhost ~]#
[root@localhost ~]# lsmod | grep kvm
kvm_amd                41702  0
kvm                   317376  1 kvm_amd



run tiny linux image to confirm
download “ttylinux.img” from  “http://marketplace.c12g.com/appliance
login: root, password: password
on the VM, start the image.
[root@localhost ~]# /usr/libexec/qemu-kvm -hda ttylinux.img -m 128

error.
dmesg on the VM
handle_exit: unexpected exit_ini_info 0x80000008 exit_code 0x60
handle_exit: unexpected exit_ini_info 0x80000008 exit_code 0x60
handle_exit: unexpected exit_ini_info 0x80000008 exit_code 0x60

on the KVM host
kvm [7175]: vcpu0 unhandled rdmsr: 0xc0010001

Googling this error, this seems to be a bug..

I tried to solve this by changing disk type ( Virtio/IDE/SCSI with raw/qcow2 ),  but I was not able to find solutions…

[ success case ]

I changed a VM to Ubuntu13.04 VMs from CentOS VMs, I was able to do nested VM without errors.

KVM host: Fedora19 ( AMD Phenom(tm) II X4 925 Processor )
VM: Ubuntu13.04 ( disk path IDE, storage format qcow2 )

When using virtio as disk path, ubuntu vm does not start up due to kernel panic..)

build a VM on Ubuntu 13.04 VM running within Fedora 19 KVM

on the Ubuntu 13.04 VM
root@ubuntu:~# vmbuilder kvm ubuntu
2013-08-21 01:19:02,794 INFO    : Unmounting target filesystem
2013-08-21 01:19:06,103 INFO    : Calling hook: convert
2013-08-21 01:19:06,103 INFO    : Converting /tmp/tmpcYqKtA to qcow2, format ubuntu-kvm/tmpcYqKtA.qcow2
2013-08-21 01:19:25,803 INFO    : Calling hook: fix_ownership
2013-08-21 01:19:25,812 INFO    : Calling hook: deploy
2013-08-21 01:19:25,813 INFO    : Calling hook: fix_ownership
root@ubuntu:~# echo $?
0

start that VM ( Ubuntu VM )
root@ubuntu:~# cd ubuntu-kvm/
root@ubuntu:~/ubuntu-kvm# ls
run.sh  tmpcYqKtA.qcow2
root@ubuntu:~/ubuntu-kvm# cat run.sh
#!/bin/sh

exec kvm -m 128 -smp 1 -drive file=tmpcYqKtA.qcow2 "$@"
root@ubuntu:~/ubuntu-kvm# ./run.sh

Ubuntu 10.04 VM is running within Ubuntu 13.04 VM which is running Fedora 19 KVM.

on the KVM host ( Fedora 19 )
virsh # dumpxml 21
<domain type='kvm' id='21'>
 <name>ubuntu1304-vm3</name>
 <uuid>c4cefa40-6dcd-194e-5e87-8675024a5029</uuid>
 <memory unit='KiB'>1048576</memory>
 <currentMemory unit='KiB'>1048576</currentMemory>
 <vcpu placement='static'>1</vcpu>
 <resource>
   <partition>/machine</partition>
 </resource>
 <os>
   <type arch='x86_64' machine='pc-i440fx-1.6'>hvm</type>
   <boot dev='hd'/>
 </os>
 <features>
   <acpi/>
   <apic/>
   <pae/>
 </features>
 <cpu mode='custom' match='exact'>
   <model fallback='allow'>phenom</model>
   <vendor>AMD</vendor>
 </cpu>
 <clock offset='utc'/>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>restart</on_crash>
 <devices>
   <emulator>/usr/bin/qemu-kvm</emulator>
   <disk type='file' device='disk'>
     <driver name='qemu' type='qcow2'/>
     <source file='/var/lib/libvirt/images/ubuntu1304-vm3.qcow2'/>
     <target dev='hda' bus='ide'/>
     <alias name='ide0-0-0'/>
     <address type='drive' controller='0' bus='0' target='0' unit='0'/>
   </disk>
   <disk type='file' device='cdrom'>
     <driver name='qemu' type='raw' cache='none'/>
     <target dev='hdc' bus='ide'/>
     <readonly/>
     <alias name='ide0-1-0'/>
     <address type='drive' controller='0' bus='1' target='0' unit='0'/>
   </disk>
   <controller type='usb' index='0'>
     <alias name='usb0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
   </controller>
   <controller type='pci' index='0' model='pci-root'>
     <alias name='pci0'/>
   </controller>
   <controller type='ide' index='0'>
     <alias name='ide0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
   </controller>
   <controller type='virtio-serial' index='0'>
     <alias name='virtio-serial0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
   </controller>
   <interface type='network'>
     <mac address='52:54:00:a3:70:c7'/>
     <source network='network1'/>
     <target dev='vnet0'/>
     <model type='virtio'/>
     <alias name='net0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
   </interface>
   <serial type='pty'>
     <source path='/dev/pts/2'/>
     <target port='0'/>
     <alias name='serial0'/>
   </serial>
   <console type='pty' tty='/dev/pts/2'>
     <source path='/dev/pts/2'/>
     <target type='serial' port='0'/>
     <alias name='serial0'/>
   </console>
   <channel type='spicevmc'>
     <target type='virtio' name='com.redhat.spice.0'/>
     <alias name='channel0'/>
     <address type='virtio-serial' controller='0' bus='0' port='1'/>
   </channel>
   <input type='mouse' bus='ps2'/>
   <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
     <listen type='address' address='127.0.0.1'/>
   </graphics>
   <video>
     <model type='qxl' ram='65536' vram='65536' heads='1'/>
     <alias name='video0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
   </video>
   <memballoon model='virtio'>
     <alias name='balloon0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
   </memballoon>
 </devices>
 <seclabel type='dynamic' model='selinux' relabel='yes'>
   <label>system_u:system_r:svirt_t:s0:c89,c482</label>
   <imagelabel>system_u:object_r:svirt_image_t:s0:c89,c482</imagelabel>
 </seclabel>
</domain>



No comments:

Post a Comment

Note: Only a member of this blog may post a comment.