Restoring a saved vm client that cannot find its attached USB device

The title is a mouthfull, I know. I really want to talk about Arduino development in a vm client. In fact, this was going to be one of the issues I had found while doing that, but I later on felt it deserves its own article. Reason is that it might be helpful on its own.

So I have a KVM-based (with libvirt) vm host, vmhost. Running in it is a vm guest, desktop. vmhost is configured to use managesave to automagically save and restore vm clients when it reboots. I like doing that because it saves the state and all that exciting stuff. Now, I configured desktop to access an Arduino board using USB passthrough, which seemed to have worked. But then I needed to reboot vmhost. I expected all my vms to come back running, but that was not the case:

[root@vmhost tmp]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     nameserver                     running
 3     nagios                         running
 4     win7                           running
 5     puppet                         running
 -     desktop                        shut off

[root@vmhost tmp]#

As you can see, desktop did not come back. Let's see if I can persuade it to start manually:

[root@vmhost tmp]# virsh start desktop
error: Failed to start domain desktop
error: internal error Did not find USB device 2341:1 bus:1 device:5

[root@vmhost tmp]#

Ok, who is that usb guy?

[root@vmhost tmp]# virsh dumpxml desktop | grep 2341
[root@vmhost tmp]#

Hmmm, nobody matching that vendor ID. But, wait! If managedsave saved the vm client before vmhost shutdown, the saved image should be in /var/lib/libvirt/qemu/save/, right?

[root@vmhost tmp]# ls /var/lib/libvirt/qemu/save/
desktop.save  lost+found
[root@vmhost tmp]#

We are making progress! Now, /var/lib/libvirt/qemu/save/desktop.save contains, amongst other things, a copy of the config as it was when the file was created. So, if we cheat a bit, we can see that file has the following entries:

<hostdev mode='subsystem' type='usb' managed='yes'>
  <source missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='5'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='6'/>
  </source>
</hostdev>

which show two hostdev entries related to our Arduino device. We can also notice the first one is the one related to the

error: internal error Did not find USB device 2341:1 bus:1 device:5

error message.

Our priority right now is to get desktop to boot. The quickest way to do so is to add startupPolicy='optional' to the

<source missing='yes'>

lines, which would tell virsh that it is ok to boot if those hostdevs are not around.

[root@vmhost tmp]# virsh save-image-edit --file /var/lib/libvirt/qemu/save/desktop.save
State file /var/lib/libvirt/qemu/save/desktop.save edited.
[root@vmhost tmp]#

If all goes well, we should now be able to start desktop:

[root@vmhost tmp]# virsh start desktop
Domain desktop started
[root@vmhost tmp]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     nameserver                     running
 3     nagios                         running
 4     win7                           running
 5     puppet                         running
 9     desktop                        running

[root@vmhost tmp]#

Success! desktop is once again up and running! And, it is not a reboot. I will not take the time to show it, so you will have to have faith on my lies, but it restored the session and the state as it was before vmhost shut down. I really like that fact (hence putting it in italics), and think this is the most important point in this entire article.

Now that desktop is up and running, virsh dumpxml should work and can be used to verify this is the vm client we edited and then restored. In other words, we should see our USB hostdevs.

[root@vmhost tmp]# virsh dumpxml desktop | less
[...]
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source startupPolicy='optional' missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='5'/>
  </source>
  <alias name='hostdev0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source startupPolicy='optional' missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='6'/>
  </source>
  <alias name='hostdev1'/>
</hostdev>
[root@vmhost tmp]#

I do not know about you, but I think it is high time to remove those bastards. Our plan is to create a file similar to what we used to tell desktop about the USB device. Since we have two entries we want to delete, we need to be more specific. However, we can be lazy and just copy the hotsdev entries from above (we will only delete one of them at a time so we can show we can specify exactly which hostdev we want to excise) into a .xml file:

[root@vmhost tmp]# cat > arduino-out.xml << 'EOF'
<hostdev mode='subsystem' type='usb' managed='yes'>
<source startupPolicy='optional' missing='yes'>
<vendor id='0x2341'/>
<product id='0x0001'/>
<address bus='1' device='5'/>
</source>
<alias name='hostdev0'/>
</hostdev>
EOF

In a previous article, we showed how to use virsh attach-device to, well, add the USB device. Therefore, it should not surprise us that virsh detach-device does the opposite:

[root@vmhost tmp]# virsh detach-device desktop arduino-out.xml
Device detached successfully

[root@vmhost tmp]#

If it worked, we should only have one USB device with vendor ID=0x2341 here.

[root@vmhost tmp]# virsh dumpxml desktop | grep 2341
<vendor id='0x2341'/>
[root@vmhost tmp]# virsh dumpxml desktop | less
<hostdev mode='subsystem' type='usb' managed='yes'>
<source startupPolicy='optional' missing='yes'>
<vendor id='0x2341'/>
<product id='0x0001'/>
<address bus='1' device='6'/>
</source>
<alias name='hostdev1'/>
</hostdev>
[root@vmhost tmp]#

So, if we edit the arduino-out.xml file so it says device='6', and run virsh detach-device desktop arduino-out.xml again, the last USB entry related to the Arduino should finally be gone.

Moral of the Story

1. Use startupPolicy='optional' so vm client will restart even if it cannot find a USB device. This might also work with a PCI passthrough device but I have not checked yet.
   
2. We can edit the config of a saved vm.
   
3. All of our cocking about did not cause the vm client to reboot. It was restored to the exact status it was before we saved it.
   
4. This article was long enough to be its own thing.