Record of the UNIX Wars: May 2015

Thursday, May 28, 2015

Save/Suspend and Resume a VMware ESXi vm client command line style

Here is an interesting project: let's say you have one or more UPS (one per power supply) attached to your ESXi vm host (or hosts; this is completely scalable). Yes, it goes without saying providing uninterrupted power to your servers is a good idea. But, unless you are a large company chances are this power will only last so long. You can make it last even longer by having a plan that will decide in which order your physical servers will be shut down based on load and remaining power. That does mean shutting down your vm servers; for the sake of this discussion, we will assume they are ESXi-based.

I have seen interesting articles on shutting down ESXi hosts on case of power failure, but many assume you are monitoring the UPS through the ESXi host. That might be thinking small; what if that is not the case? What if you have a UPS or two on the bottom feeding the entire cabinet? Chances are you will be monitoring it from a host, be it a vm or not, that is running some monitoring program such as Nagios, that is set to do something in case of a power failure. Of course, if you have a monitoring vm you can talk to your UPS using either ethernet or USB passthrough depending on how sophisticated that model is. And it will decide when to tell our ESXi box it is time to shut down.

I do not know about you but I would like to gracefully save/shutdown the vm clients running in that host before that.

The plan is to have the host monitoring the UPS tell the ESXi host to run a shutdown procedure, which would need to first save the vm guests. And, once the vm server is back up and running, it would resume -- by its own accord or by the order of another server -- the saved vm clients. Yes, you will have to worry about how the monitoring and the ESXi hosts will talk to each other and how the client's clock will catch up, but for this article we will focus on creating a tool that only cares about saving and resuming all of the vm guests running in this ESXi box. We can expand later.

If we want to save the running vm clients, we probably should find out which ones are running. In a previous article we wrote a script to see if a given vm client is running, off, or saved. For the script we will be creating, we want to use something else, vmdumper. Here is what the help screen for the program says.

/tmp # vmdumper -h
vmdumper: [options]  
         -f: ignore vsi version check
         -h: print friendly help message
         -l: print information about running VMs
         -g: log specified text to the vmkernel log
/tmp #

Note the -l shows only the running vms, which is what we want to do. So, let's run that and see what it spits back (I will break them a bit so they will kinda fit the screen):

~ # /sbin/vmdumper -l
wid=264397      pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/boot2docker/boot2docker.vmx" uuid="56 4d 11 2b 63 bc 88 fb-d9 e1 
93 fc 69 36 66 45"  displayName="boot2docker"       vmxCartelID=264396
wid=13080       pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/Windows 2012/Windows 2012.vmx"       uuid="56 4d e7 cb 24 11 63 
13-04 0d 9b 41 08 f9 a3 be"  displayName="Windows 2012"      vmxCartelID=13079
wid=527962      pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/devcentos/devcentos.vmx"     uuid="56 4d d7 e8 25 6c de 91-09 38 
60 ce ab 5d 43 ca"  displayName="devcentos" vmxCartelID=527961
~ #

As you can see, it shows the path for the config file the vm guest is using (cfgFile, its name (displayName) and something called wid. And a few other things I do not feel like caring about. So, how do we save a vm anyway? We know we can start a vm using vim-cmd vmsvc/power.on, so maybe it sounds similar. Some frustrating searching later we find that http://www.vi-toolkit.com/wiki/index.php/Vmsvc/power.hibernate might be a candidate. Thing is it needs wmid as the argument. I will save some time and state (have faith, brother!) it can be obtained by

vim-cmd vmsvc/getallvms | grep "${displayName}" | awk '{ print "vmid=" $1}'

But, does it really work? We shall try with devcentos, which happens to have wmid=3 (again, I cheat because I have spent loads of time testing this):

/tmp # vim-cmd  vmsvc/power.hibernate 3
(vim.fault.ToolsUnavailable) {
   dynamicType = ,
   faultCause = (vmodl.MethodFault) null,
   msg = "Cannot complete operation because VMware Tools is not running in this virtual machine.",
}
/tmp #

And it does not seem to want to work. It needs VMware Tools, and I do not want to worry about it. So let's see what else we can use. After some looking I found vmdumper. To save devcentos we could do

/tmp # vmdumper 527962 suspend_vm
Suspending VM...
/tmp #

The weird number 527962 is the world id or wid for devcentos, which happens to be the first column in the output of vmdumper -l associated with that vm client.

Pet Peeve: If you remember the output of vmdumper -h, which should be the help page for that command, mentions nothing about suspend-vm. Good job, VMware! That does make me wonder what else you are not documenting...

Now my venting is done, let's see what we need.

We need the wid to shut down with vmdumper
We can resume (I tested already, and so can you!) the vm client using vim-cmd vmsvc/power.on. Thing is it needs wmid as the argument, which we figure out how to get above.
We then need a way to save wmid so when we can restore the saved vms. Probably saving the names of the vms would also be a nice touch.

So, here is the script I wrote to save and restore the running vms. As you can see, it is rather dumb since it is an all or nothing kinda deal. It is also unforgiving: if you run it again to save vms, the old /var/tmp/save_vms file will be overwritten. For what I wrote this script for, that is but a small annoyance.

cat > save_runningvms.sh  << 'EOF'
#!/bin/sh
IFS=$'\n'
USAGE="Usage: $0 {save|resume}"
SAVE_FILE=/var/tmp/save_vms

if [ "$#" == "0" ]; then
        echo "$USAGE"
        exit 1
fi

selection=$1

case $selection in
   # If we want to save them
   save )
      rm -f ${SAVE_FILE}

      # Find which vms are currently running
      for i in $(vmdumper -l \
         | awk ' BEGIN { FS = "\t" }; { print $1 ";" $5 }')
      do
         eval $i
         # Start saving them
         vmid=$(vim-cmd vmsvc/getallvms | grep "${displayName}" \
            | awk '{ print "vmid=" $1}')
         vmdumper $wid suspend_vm

         # Write list of saved guests in $SAVE_FILE
         echo $vmid ";" $i >> ${SAVE_FILE}
      done
      ;;
   # If we want to restore them
   resume )
      # Get list of saved guests
      for i in $(cat ${SAVE_FILE})
      do
         # Wake them up
         eval $i
         vim-cmd vmsvc/power.on $vmid
      done
      ;;
esac
EOF
chmod +x save_runningvms.sh

You will note that I avoid using Bashisms because the shell in busybox is closer to Bourne than Bash.

I think you probably want to see it running. So, let's run it. First we do some saving

/tmp # ./save_runningvms.sh save
Suspending VM...
Suspending VM...
Suspending VM...
/tmp #

Did it create the /var/tmp/save_vms file? If so, how does it look like?

/tmp # cat /var/tmp/save_vms
vmid=24 ; wid=5718058;displayName="boot2docker"
vmid=23 ; wid=5714001;displayName="Windows 2012"
vmid=3 ; wid=5715871;displayName="devcentos"
/tmp #

Ok, I am not convinced. You must be lying. Lemme go to the other vmhost, vmhost, and ping devcentos

[raub@vmhost tmp]# ping devcentos
PING devcentos.example.com (10.0.0.112) 56(84) bytes of data.
From vmhost.example.com (10.0.0.19) icmp_seq=2 Destination Host Unreachable
From vmhost.example.com (10.0.0.19) icmp_seq=3 Destination Host Unreachable
From vmhost.example.com (10.0.0.19) icmp_seq=4 Destination Host Unreachable
^C
--- devcentos.example.com ping statistics ---
7 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6125ms
pipe 3
[raub@vmhost tmp]#

Hmmmm, okay. But maybe it was off and you were lying to me. So, let's see about waking up the sleeping vms.

/tmp # ./save_runningvms.sh resume
Powering on VM:
Powering on VM:
Powering on VM:
/tmp #

And then pinging devcentos

[raub@vmhost tmp]# ping devcentos
PING devcentos.example.com (10.0.0.112) 56(84) bytes of data.
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=1 ttl=64 time=212 ms
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=2 ttl=64 time=0.316 ms
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=3 ttl=64 time=0.313 ms
^C
--- devcentos.example.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2078ms
rtt min/avg/max/mdev = 0.313/70.992/212.349/99.954 ms
[raub@vmhost tmp]#

I guess the script does work after all. What's the world coming to?

Monday, May 25, 2015

Get status of local VMware ESXi guests using command line

I have been talking a lot about KVM, and showing examples of how to control it from the command line. In fact, just a few posts ago we were talking about USB passthrough in KVM. I guess it is high time to do some in ESXi. After all, some of you -- yours truly included -- also have to deal with ESXi, which is VMware's product.

At this point in the presentation I can see some of you stomping your chests shouting "ESXi is Enterprise level product, not to be compared to the amateur hour likes of Kay-Vee-Em!". I have bad news for you, sunshine: they are all the same. They all can do live migration and clustering and so on. RedHat builds their enterprise turnkey solutions around KVM, which they also own. The differences between them are moving targets. And they are not the only games in town. Deal with it.

Back to the topic, yes ESXi has a nice, albeit C#-dependent (i.e. Windows only), GUI client... which they have been trying to get rid of for a while. But, sometimes I (I will take full blame for this) want to do something that is not available in the GUI. Now, I am aware I could be using PowerCLI, but what if I do want to have something running automagically in the ESXi host? To see what I mean, let's use a simple example: suposed we want to have a nice list of which vm clients ar ein this ESXi host and what they are up to (running/paused/etc). If you can't figure out why we would want to do that right now, hold onto your seat until the later parts of this article.

Of course this assumes you can ssh into the ESXi host.

The command we want to use is vm-support, and the option is -listmvs:

vm-support --listvms
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/mail_1/mail_1.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/Windows 2012/Windows 2012.vmx (Running)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/coreos/coreos.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devnetbsd/devnetbsd.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/centos64/centos64.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/boot2docker/boot2docker.vmx (Running)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/TheOnion/TheOnion.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devubuntu/devubuntu.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/freebsd/freebsd.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devcentos/devcentos.vmx (Registered)
#

As you can see, it shows where each VM disk image (yes, yes, I am lazy but this is just a little blog post. Focus, focus, focus) and a status of Running or Registered. It is a good start; we feel good about ourselves until we realize that Registered canmean the vm is turned off or just saved. Bummer.

Now I know that when you save a vm, it creates a file with a .vmss extension that stores the state of the vm when it was saved. Let me show an example: I know that devcentos is saved for I did that last week. If what I said before is not a lie, we should find a .vmss file:

# ls /vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devcentos/*.vmss
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devcentos/devcentos-aaf17b2a.vmss
#

For those of you who read this blog (why would you do that?), that sounds just like the .save file KVM uses to do the very same thing. Didn't I mention in the end of the day they are the same? So, what we need to do is if we find a vm labelled as Registered, we should also see if it has a .vmss file. If so, it is a saved vm. I could bore you with the details and trials, but here is what I got:

cat > vm_status.sh  << 'EOF'
#!/bin/sh
IFS=$'\n'

for i in $(vm-support --listvms)
do 
   vm_path=$(dirname "$i")
   vm_name=$(basename $vm_path)
   vm_status=$(echo "$i" | awk '{ print $NF}' )

   # Now check status of each vm
   case $vm_status in
      '(Running)' )
         echo "$vm_name is currently running"
         ;;
      '(Registered)' )
         test -f $vm_path/*.vmss \
            && echo "$vm_name is currently saved" \
            || echo "$vm_name is currently off"
         ;;
   esac

done
EOF
chmod +x vm_status.sh

It is not perfect, but it does what I want:

# ./vm_status.sh 
mail_1 is currently off
Windows 2012 is currently running
coreos is currently off
ctfbox is currently off
devnetbsd is currently off
centos64 is currently off
boot2docker is currently running
TheOnion is currently saved
devubuntu is currently off
freebsd is currently off
devcentos is currently saved
#

One thing you need to be aware is that ESXi uses busybox, so do not try to write a full blown Bash script; you will end very disappointed.

"So, what is the point of this script as it runs in the ESXi host?" you might ask, and you have a point. By itself it makes more sense to wire it in PowerCLI and run it that way. But, what if you want to have a script send you periodic reports of which vms are alive and which ones are off? Or what if you need to see if you need to start a given vm or just restore it? This script is small and a bit of a simpleton, but it shows the potential we have for writing proper scripts to be run in the ESXi host. I will show an even more practical example in an upcoming article.

Friday, May 15, 2015

USB Passthrough in KVM/libvirt 2: Problems and workarounds on Arduino development in a vm client

We will talk a bit about setting up an Arduino development environment. Originally I thought this was going to be a short article, but that was not the case. The more I worked on this the longer it became. So I got annoyed and put the part about setting USB passthrough and then dealing with waking up a vm client that lost its USB device after it was saved in their own articles. I want to have this article as focused on building an Arduino development environment in a vm guest as possible; we already talked about USB passthrough in general and even accessed a UPS as example.

Last time I did setup an Arduino development environment I used my laptop and my Arduino. That worked fine but required me to have my Arduino with me and the laptop just in case I decided to do something; which I did: I carried the board, cables, and even a USB drive with useful stuff in a little box in my backpack at all times. With time that got old, I now prefer to do remote development as much as I can get away with, so let's see if I can do it with the Arduino.

The Arduino boards I will be using here are the Funduino Uno/YourDuinoRoboRED and a Duemilanove.

The scientific reason I am using those boards is because that is what I have; I bought both of them with my own money. The first one I ever got was the Duemilanove; it is in fact the one I used to carry in my bag. Recently (as in last month) I bought the Funduino board. I decided to start this article by choosing this red board because it comes with a miniUSB port and included a USB cable just the right size so it can be precariously hung from vmhost. Maybe also because it is more colourful; you'll be the judge.

We will be adding an Arduino device to the vm client desktop, which is an Ubuntu Desktop vm in the vmhost called, for reasons the go beyond the topic of this article, vmhost, which runs KVM with libvirt.

Setting the mess up

The title of this article mentions USB passthrough. Main difference between that and PCI passthrough is that the USB one allows hotplugging. And that means we should be able to add the Arduino board to desktop while this vm client is running. At least that is the hope.

Without further ado, let's get busy.

We probably should start this by connecting the arduino board to on of vmhost's USB port. The picture on the right shows the actual board hanging in front of the actual vmhost.

As we did in an earlier article on USB passthrough, we should ask if vmhost knows about the Arduino board:

[raub@vmhost ~]$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 001 Device 003: ID 2341:0001 Arduino SA Uno (CDC ACM)
[raub@vmhost ~]$

The line we are interested in is

Bus 001 Device 003: ID 2341:0001 Arduino SA Uno (CDC ACM)

Note that it identifies itself as an Arduino Uno; I do not know if the real Arduino Uno also identifies itself the same way. Also note that

<vendor id='0x2341' />
<product id='0x0001' />

With that info we can now tell desktop about the Arduino board. As we have seen before, we can do it live by creating a xml config file containing info about the Arduino board:

cat > arduino.xml << 'EOF'
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x2341'/>
        <product id='0x0001'/>
      </source>
    </hostdev>
EOF

Attach it

[root@vmhost tmp]# virsh attach-device desktop ./arduino.xml
Device attached successfully

[root@vmhost tmp]#

Check if the vm client reports it as attached

raub@desktop:~$ lsusb
Bus 001 Device 004: ID 2341:0001 Arduino SA Uno (CDC ACM)
Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
raub@desktop:~$

Don't forget since we are accessing this board as a normal user, we need to be in the group that owns that device. I know that its USB port is shown as /dev/ttyACM0, so we need to find who owns it.

raub@desktop:~/Downloadz/Arduino$ ls -l /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 May  8 09:56 /dev/ttyACM0
raub@desktop:~/Downloadz/Arduino$ id
uid=1000(raub) gid=1000(raub) groups=1000(raub),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),107(lpadmin),124(sambashare),127(debian-tor)
raub@desktop:~/Downloadz/Arduino$

Get and install the Arduino SDK. Since I am doing all of this in a Linux vm, installing the SDK is really uncompressing the file, arduino-1.6.4-linux64.tar.xz at the time I wrote this article, into where you want to run the SDK from; I put it in ~/bin.
```
tar xJvf arduino-1.6.4-linux64.tar.xz
```
And run and configure the SDK. I connected to the vm running the SDK from my laptop, telling ssh to do X11 port forwarding:
```
ssh -X desktop
```
Once there, I started the Arduino SDK
```
./bin/arduino-1.6.4/arduino
```
which popped the usual GUI on my laptop screen thanks to the magic of X Windows. Now, do tell it that you are connected to an Arduino Uno board through port /dev/ttyACM0.

Now we need to test things up. I think one of the best programs to test an Arduino SDK setup is blink, which controls the little LED that a lot of boards have built-in. Here is the original code if you are too lazy to look it up:

/*
  Blink
  Turns on an LED on for one second, then off for one second, repeatedly.

  This example code is in the public domain.
 */

// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
int led = 13;

// the setup routine runs once when you press reset:
void setup() {
  // initialize the digital pin as an output.
  pinMode(led, OUTPUT);
}

// the loop routine runs over and over again forever:
void loop() {
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(1000);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);               // wait for a second
}

Upload the file and see if it blinks the way you told it to do so (1s on, 1s off).

And all seems to be nice an peachy; we should end this article here and pat ourselves on the back. But, this is where things start to go wrong.

Things Did Not Happen According to the Plan

Let's say I want to change the blink frequency in the blink program. Maybe I want to have the LE stay on 2s and off 1s. That part of the code would change like this:

void loop() {
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(2000);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);               // wait for a second
}

Great! now, let's upload the new code and stare at the LED.

And no changes.

We then look at the GUI and find the following messages:

Sketch uses 1,068 bytes (3%) of program storage space. Maximum is 32,256 bytes.
Global variables use 11 bytes (0%) of dynamic memory, leaving 2,037 bytes for local variables. Maximum is 2,048 bytes.
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 7 of 10: not in sync: resp=0x00
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 8 of 10: not in sync: resp=0x00
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 9 of 10: not in sync: resp=0x00
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 10 of 10: not in sync: resp=0x00
Problem uploading to board.  See http://www.arduino.cc/en/Guide/Troubleshooting#upload for suggestions.

Sometimes doing virsh detach-device and then reattaching it will work, but other times it does not, which leads to...

The Funduino board likes to disappear.

[root@vmhost tmp]# virsh attach-device desktop arduino.xml
error: Failed to attach device from arduino.xml
error: internal error Did not find USB device 2341:1

[root@vmhost tmp]#  lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
[root@vmhost tmp]#

By that I mean I still have not found what triggers that. Sometimes it is connected to vmhost for 3 days doing nothing but always showing on lsusb even if it is not attached to a vm client. And then, it is gone. Just like that. Only workaround I found was to physically unplug the Funduino out of vmhost and then plug it back in.

What happens if you use the Duemilanove?

I have to say I have not been able to replicate any of the above issues with the Duemilanove. It uses a different USB-to-Serial chip, so it is seen by lsusb as

Bus 001 Device 012: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC

I do not know if the chipset makes a difference or there is something in my Funduino board that is boink (maybe the chipset is more picky?). I have another Funduino board, but not an Arduino Uno, so I can only test so much. What I can say is as of now I will use the Duemilanove to do remote development, but when I want to take a board in the field -- say, attach to my car -- I will grab the Funduino one. After all it works fine if I just install new code once.

Update on the Funduino/RoboRED: Success!

So I got home last night, took my second Funduino out of its shrinkwrap, and tried it. It works as well as the Duemilanove:

If I disconnect it physically from vmhost, desktop gracefully reports it as gone. virsh dumpxml still reports an entry
```
[root@vmhost tmp]# virsh dumpxml desktop|grep 2341
        
        
[root@vmhost tmp]# 
```
but that could be how KVM does things.
If I unload it using
```
[root@vmhost tmp]# virsh detach-device desktop arduino.xml                      
Device attached successfully

[root@vmhost tmp]#
```
it does get removed from desktop's config file. If there is more than one entry, it might take a bit more time but it will come out.
If we did not physically remove the Arduino device as above, instead just using virsh detach-device, it will be removed from destkop's config and running lsusb in that vm client will no longer report it.
Once lsusb in vmhost shows the Arduino board back (say, you physically connected it back to that machine),
```
[root@vmhost tmp]# virsh attach-device desktop arduino.xml                      
Device attached successfully

[root@vmhost tmp]#
```
will once again make the device available for desktop.

Since the problem is on the first one; I will see about getting it replaced.

Thursday, May 14, 2015

Restoring a saved vm client that cannot find its attached USB device

The title is a mouthfull, I know. I really want to talk about Arduino development in a vm client. In fact, this was going to be one of the issues I had found while doing that, but I later on felt it deserves its own article. Reason is that it might be helpful on its own.

So I have a KVM-based (with libvirt) vm host, vmhost. Running in it is a vm guest, desktop. vmhost is configured to use managesave to automagically save and restore vm clients when it reboots. I like doing that because it saves the state and all that exciting stuff. Now, I configured desktop to access an Arduino board using USB passthrough, which seemed to have worked. But then I needed to reboot vmhost. I expected all my vms to come back running, but that was not the case:

[root@vmhost tmp]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     nameserver                     running
 3     nagios                         running
 4     win7                           running
 5     puppet                         running
 -     desktop                        shut off

[root@vmhost tmp]#

As you can see, desktop did not come back. Let's see if I can persuade it to start manually:

[root@vmhost tmp]# virsh start desktop
error: Failed to start domain desktop
error: internal error Did not find USB device 2341:1 bus:1 device:5

[root@vmhost tmp]#

Ok, who is that usb guy?

[root@vmhost tmp]# virsh dumpxml desktop | grep 2341
[root@vmhost tmp]#

Hmmm, nobody matching that vendor ID. But, wait! If managedsave saved the vm client before vmhost shutdown, the saved image should be in /var/lib/libvirt/qemu/save/, right?

[root@vmhost tmp]# ls /var/lib/libvirt/qemu/save/
desktop.save  lost+found
[root@vmhost tmp]#

We are making progress! Now, /var/lib/libvirt/qemu/save/desktop.save contains, amongst other things, a copy of the config as it was when the file was created. So, if we cheat a bit, we can see that file has the following entries:

<hostdev mode='subsystem' type='usb' managed='yes'>
  <source missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='5'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='6'/>
  </source>
</hostdev>

which show two hostdev entries related to our Arduino device. We can also notice the first one is the one related to the

error: internal error Did not find USB device 2341:1 bus:1 device:5

error message.

Our priority right now is to get desktop to boot. The quickest way to do so is to add startupPolicy='optional' to the

<source missing='yes'>

lines, which would tell virsh that it is ok to boot if those hostdevs are not around.

[root@vmhost tmp]# virsh save-image-edit --file /var/lib/libvirt/qemu/save/desktop.save
State file /var/lib/libvirt/qemu/save/desktop.save edited.
[root@vmhost tmp]#

If all goes well, we should now be able to start desktop:

[root@vmhost tmp]# virsh start desktop
Domain desktop started
[root@vmhost tmp]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     nameserver                     running
 3     nagios                         running
 4     win7                           running
 5     puppet                         running
 9     desktop                        running

[root@vmhost tmp]#

Success! desktop is once again up and running! And, it is not a reboot. I will not take the time to show it, so you will have to have faith on my lies, but it restored the session and the state as it was before vmhost shut down. I really like that fact (hence putting it in italics), and think this is the most important point in this entire article.

Now that desktop is up and running, virsh dumpxml should work and can be used to verify this is the vm client we edited and then restored. In other words, we should see our USB hostdevs.

[root@vmhost tmp]# virsh dumpxml desktop | less
[...]
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source startupPolicy='optional' missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='5'/>
  </source>
  <alias name='hostdev0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source startupPolicy='optional' missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='6'/>
  </source>
  <alias name='hostdev1'/>
</hostdev>
[root@vmhost tmp]#

I do not know about you, but I think it is high time to remove those bastards. Our plan is to create a file similar to what we used to tell desktop about the USB device. Since we have two entries we want to delete, we need to be more specific. However, we can be lazy and just copy the hotsdev entries from above (we will only delete one of them at a time so we can show we can specify exactly which hostdev we want to excise) into a .xml file:

[root@vmhost tmp]# cat > arduino-out.xml << 'EOF'
<hostdev mode='subsystem' type='usb' managed='yes'>
<source startupPolicy='optional' missing='yes'>
<vendor id='0x2341'/>
<product id='0x0001'/>
<address bus='1' device='5'/>
</source>
<alias name='hostdev0'/>
</hostdev>
EOF

In a previous article, we showed how to use virsh attach-device to, well, add the USB device. Therefore, it should not surprise us that virsh detach-device does the opposite:

[root@vmhost tmp]# virsh detach-device desktop arduino-out.xml
Device detached successfully

[root@vmhost tmp]#

If it worked, we should only have one USB device with vendor ID=0x2341 here.

[root@vmhost tmp]# virsh dumpxml desktop | grep 2341
<vendor id='0x2341'/>
[root@vmhost tmp]# virsh dumpxml desktop | less
<hostdev mode='subsystem' type='usb' managed='yes'>
<source startupPolicy='optional' missing='yes'>
<vendor id='0x2341'/>
<product id='0x0001'/>
<address bus='1' device='6'/>
</source>
<alias name='hostdev1'/>
</hostdev>
[root@vmhost tmp]#

So, if we edit the arduino-out.xml file so it says device='6', and run virsh detach-device desktop arduino-out.xml again, the last USB entry related to the Arduino should finally be gone.

Moral of the Story

Use startupPolicy='optional' so vm client will restart even if it cannot find a USB device. This might also work with a PCI passthrough device but I have not checked yet.
We can edit the config of a saved vm.
All of our cocking about did not cause the vm client to reboot. It was restored to the exact status it was before we saved it.
This article was long enough to be its own thing.

USB Passthrough in KVM/libvirt: How to talk to a UPS

I was writing an article about how to connect an Arduino to a kvm vm host so a vm guest can access it. Since there were some issues I would like to share, I decided to break it down in two parts. First would be this very article, dealing with setting up USB passthrough. In a future article we will talk about issues using Arduino with USB passthrough.

Last time I talked about passthrough was in a post about using grep to look for a vm client which uses PCI passthrough. I never really mentioned how to do the deed there because I assumed everyone knew how to do that in KVM. Should we talk about how to set that up? You tell me. Since I will be talking about using USB passthrough here, you can see if it is enough to understand and use PCI passthrough. If not, let me know and I will write a few lies about it.

Since I kept talking about the Arduino, I will not use it as the example here. Instead, an APC-brand UPS is chosen for the unbiased version that it is what I have here. Here's the idea: let's say you have a vm client running something like Nagios or Icinga in it. Why not use it to monitor the UPS and let us know (or initiate some automated operation) if it is in use and/or the remaining charge drops below a certain level?

We will be adding the APC to the vm client scan, which is an CentOS vm client/guest in the KVM-based with libvirt vm host called, for reasons the go beyond the topic of this article, vmhost.

Install and Setup

The UPS in question has a USB cable that is used to monitor/talk to it. Now I know as matter of fact that apcupsd works well with that setup, so we should install it in our nagios (I am using this as the vm client's name for the sake of lazyness) server. To make a long story short, since this is RedHat-based distro we can get it by
```
yum install apcupsd --enablerepo=epel
```

Let's make sure the vm can see the APC box. To do so, we first need to see if the vm host, affectionately known as vmhost because of some unexplained reason, know we plugged the console USB cable from the APC:

[root@vmhost tmp]# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
[root@vmhost tmp]#

You have to agree the line

Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply

is a dead giveaway that this is the APC-brad UPS. Now we identified the entry, we want to get the USB vendor and product IDs. That will be the 051d:0002, respectively.

Next we need to reconfigure scan so it knows it has a new USB device attached to it. A USB device is a seen by KVM as hostdev element. From the libvirt docs, a hostdev is the container that describe host devices. We could shutdown scan and edit its config file, but we should/better be able to do it live. First we create a little file containing the hostdev definition:
```
cat > apc.xml << 'EOF'
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source startupPolicy='optional'>
        <vendor id='0x051d'/>
        <product id='0x0002'/>
      </source>
    </hostdev>
EOF
```
You can see the vendor and product IDs, which we gathered from last step. The reason for the startupPolicy='optional' is that if the machine needs to reboot, or come back from being saved, it will do so even if the usb device is no longer reachable. Now, we should let the vm client know about the UPS:
```
[root@vmhost tmp]# virsh attach-device nagios apc.xml
Device attached successfully

[root@vmhost tmp]# 
```

Does our vm client see it now?

[raub@scan ~]$ lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
Bus 001 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
[raub@scan ~]$

It seems it is there, so we should test the connection using apcupsd

[root@scan ~]# service apcupsd start
grep: /etc/nologin: No such file or directory
Starting UPS monitoring:                                   [  OK  ]
[root@scan ~]# service apcupsd status
apcupsd (pid  19373) is running...
APC      : 001,036,0917
DATE     : 2015-05-14 12:30:43 -0400
HOSTNAME : scan.in.example.com
VERSION  : 3.14.10 (13 September 2011) redhat
UPSNAME  : ups1
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2015-05-14 12:30:42 -0400
MODEL    : Back-UPS RS 1500G
STATUS   : ONLINE
LINEV    : 124.0 Volts
LOADPCT  :  21.0 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  38.4 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
SENSE    : Medium
LOTRANS  : 088.0 Volts
HITRANS  : 147.0 Volts
ALARMDEL : 30 seconds
BATTV    : 27.0 Volts
LASTXFER : Automatic or explicit self test
NUMXFERS : 0
TONBATT  : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x07000008 Status Flag
SERIALNO : 3B1416X00241
BATTDATE : 2014-04-14
NOMINV   : 120 Volts
NOMBATTV :  24.0 Volts
NOMPOWER : 865 Watts
FIRMWARE : 865.L5 .D USB FW:L5
END APC  : 2015-05-14 12:30:47 -0400
[root@scan ~]#

Smells like our USB passthrough adventure worked.

We could go on and configure Nagios to do the UPS monitoring, but that will be left for another article; if you want to get ahead, a starting point would be to search for "nagios apcupsd". Remember, all we wanted here is to get the USB passthrough part working. For the next article we will talk about when things do not work as peachy.

Removing the USB device from the vm guest

If you want to remove it programatically,
```
virsh detach-device nagios apc.xml
```
will remove it from nagios's config. lsusb on the vm client (nagios) will show it is gone.

If you just physically pluck the USB device from vmhost, the client will report it as gone However, it will still be in the vm client's config file. Here is an example of a USB device with vendor ID=2341 that I physically plucked from more than once (and then put it back and added to the client):

[root@vmhost tmp]# virsh dumpxml desktop|grep 2341
        <vendor id='0x2341'/>
        <vendor id='0x2341'/>
[root@vmhost tmp]#

The solution for that is to keep removing it (virsh detach-device nagios apc.xml) until virsh dumpxml stops reporting it is there. And then add it again.

Let me show this step-by-step with the AUP example:

We start by not seeing the APC UPS in the vm client:

[root@scan ~]# lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
[root@scan ~]#

The vm host tells us there are 3 entries in the vm client's config:

[root@vmhost tmp]# virsh dumpxml nagios|grep  051d
        <vendor id='0x051d'/>
        <vendor id='0x051d'/>
        <vendor id='0x051d'/>
[root@vmhost tmp]#

Let's start removing them:

[root@vmhost tmp]# virsh detach-device nagios apc.xml
Device detached successfully

[root@vmhost tmp]# virsh detach-device nagios apc.xml
Device detached successfully

[root@vmhost tmp]# virsh dumpxml nagios|grep  051d
        <vendor id='0x051d'/>
[root@vmhost tmp]#

Sounds like we can't count. We need to do the deed 3 times, so we do it one last time and see if the device is finally gone:

[root@vmhost tmp]# virsh detach-device nagios apc.xml
Device detached successfully

[root@vmhost tmp]# virsh dumpxml nagios|grep  051d
[root@vmhost tmp]#

Success! Now we add it

[root@vmhost tmp]# virsh attach-device nagios apc.xml
Device attached successfully

[root@vmhost tmp]#

And check in the vm client if it is back

[root@scan ~]# lsusb
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
Bus 001 Device 009: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
[root@scan ~]#

If you save the vm client (virhs managedsave or virsh save), exciting things will happen.

Monday, May 11, 2015

On extracting the contents of a .msi file

Twice now I had to get the contents of a .msi file. First it was because I needed some specific drivers for a SCCM-based network interface driver upgraded (I will talk about that one in a future post). The most recent time I was trying to do a manual install of the Avocent Dsview4 plugin hoping it would then work with my Firefox web browser. This last one will take even longer until I am ready to comment on here, so do not hold your breath.

It seems one way to do the deed is to use 7zip. I tried, and did not go very far and I got frustrated. I then decided instead to use something that is already in the system. And, if my previous posts are any indicator, you would have guessed it would be something command line. And you would be quite right:

msiexec /a C:\Users\raub\Downloads\AvctViewerPlugin.msi /qn targetdir=C:\Users\raub\Downloads\pickles

Where

/qn: Run quietly, no windows nor dialog boxes are shown.
targetdir: I think that is self-explanatory. It is where you want to extract the files to.

This will create a directory called "Temp Folder" inside pickles.

PS C:\Users\raub\Downloads\pickles> ls


    Directory: C:\Users\raub\Downloads\pickles


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----          5/6/2015  12:34 PM            Temp Folder
-a---          5/6/2015  12:34 PM      40960 AvctViewerPlugin.msi


PS C:\Users\raub\Downloads\pickles>

which contains all the file extracted from the .msi file.

NOTES

This is really the most important part of this entire article. The command to extract it is not as important as its idiosyncrasies:

.msi file should not be in targetdir
Use full/absolute paths. If you do not believe me, try relative paths, specially with targetdir, and see what happens.
Do not need to be admin/root to run the above command. Remember, you are not installing anything. Also, try to make a point to only become admin if you really need it.
Yes, it put a copy of the original .msi file inside pickles.

Record of the UNIX Wars