Getting the profile name for installing a ESXi patch from command line

So I go to the vmware page for ESXi patches and downloaded the patch

ESXi670-201811001

Product:ESXi (Embedded and Installable) 6.7.0

Download Size:317.5 MB

But when I try to install it, I need a "patch" name, whatever that is

[root@vmhost2:/tmp] esxcli software profile update -d /vmfs/volumes/VOL1
/ISO/ESXi670-201811001.zip
Error: Missing required parameter -p|--profile

Usage: esxcli software profile update [cmd options]

Description:
  update                Updates the host with VIBs from an image profile in a
                        depot. Installed VIBs may be upgraded (or downgraded
                        if --allow-downgrades is specified), but they will
                        not be removed. Any VIBs in the image profile which
                        are not related to any installed VIBs will be added
                        to the host. WARNING: If your installation requires a
                        reboot, you need to disable HA first.

Cmd options:
  --allow-downgrades    If this option is specified, then the VIBs from the
                        image profile which update, downgrade, or are new to
                        the host will be installed. If the option is not
                        specified, then the VIBs which update or are new to
                        the host will be installed.
  -d|--depot=[  ... ]
                        Specifies full remote URLs of the depot index.xml or
                        server file path pointing to an offline bundle .zip
                        file. (required)
  --dry-run             Performs a dry-run only. Report the VIB-level
                        operations that would be performed, but do not change
                        anything in the system.
[...]
  -p|--profile=    Specifies the name of the image profile to update the
                        host with. (required)
  --proxy=         Specifies a proxy server to use for HTTP, FTP, and
                        HTTPS connections. The format is proxy-url:port.
[root@vmhost2:/tmp]

Maybe it is named after the file? Let's punt

[root@vmhost2:/tmp] esxcli software profile update -d /vmfs/volumes/VOL1/IS
O/ESXi670-201811001.zip  -p ESXi-6.7.0-201811001-standard
 [NoMatchError]
 No image profile found with name 'ESXi-6.7.0-201811001-standard'
         id = ESXi-6.7.0-201811001-standard
 Please refer to the log file for more details.
[root@dentistryesxi2:/tmp]

After feeling a bit frustrated, I found some docs on updating a host using image profiles with the following example:

esxcli --server=server_name software sources profile list --depot=http://webserver/depot_name

Well, we know even though the docs seem to imply the "depot" is a server somewhere with the patch bundles as .zip files, we can just download and give the path. So, let's see if that applies here too:

[root@vmhost2:/tmp] esxcli software sources profile list -d /vmfs/volume
s/VOL1/ISO/ESXi670-201811001.zip
Name                             Vendor        Acceptance Level  Creation Time        Modification Time
-------------------------------  ------------  ----------------  -------------------  -------------------
ESXi-6.7.0-20181104001-no-tools  VMware, Inc.  PartnerSupported  2018-11-08T08:39:27  2018-11-08T08:39:27
ESXi-6.7.0-20181104001-standard  VMware, Inc.  PartnerSupported  2018-11-08T08:39:27  2018-11-08T08:39:27
[root@dentistryesxi2:/tmp]

Aha! I was close but no cookies. I really would never have guessed it was 20181104001 instead of 201811001; that is why it pays to ask whoever knows the right answer, in this case the file itself. Let's try it:

[root@vmhost2:/tmp] esxcli software profile update -d /vmfs/volumes/VOL1/ISO/ESXi670-201811001.zip \
 -p ESXi-6.7.0-20181104001-standard
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true 
   VIBs Installed: VMW_bootbank_bnxtroce_20.6.101.0-20vmw.670.1.28.10302608, VMW_bootbank_brcmfcoe_11.4.1078.5-11vmw.670.1.28.10302608, VMW_bootbank_elxnet_11.4.1095.0-5vmw.670.1.28.10302608, VMW_bootbank_i40en_1.3.1-22vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMW_bootbank_ixgben_1.4.1-16vmw.670.1.28.10302608, VMW_bootbank_lpfc_11.4.33.3-11vmw.670.1.28.10302608, VMW_bootbank_lsi-mr3_7.702.13.00-5vmw.670.1.28.1030260
[...]
otbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-
2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VM
W_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb
-storage_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.0-0.0.8169922, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, VMware_bootbank_esx-xserver_6.7.0-0.0.8169922, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMwa
re_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMware_bootbank_
native-misc-drivers_6.7.0-0.0.8169922, VMware_bootbank_qlnativefc_3.0.1.0-5vmw.
670.0.0.8169922, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922
[root@vmhost2:/tmp]

Much better.

Remotely controlling ESXi with bash and without Ansible or Windows

I would expect the more knowledgeable ESXi admins amongst you will rightfully mention the proper way to run a remote command in a ESXi box is using the vSphere (Web) Client or the vSphere Command-Line Interface. I agree completely, which is why I will ignore the advice and see what else I can use.

Shattering a Dream

Just to let you know, I really really REally wanted to have used Ansible here. In fact, this was supposed to be a post on how to use Ansible to shut down an ESXi server, saving the running vm clients, and then restore those clients after the server was back up. I spent time laying things out but could not get it working. I will not go into details; but at first I thought my issues were due to ESXi not having python-simplejson, but I learned that assumption was wrong; Python in ESXi has json in it:

I will try again some other time. For now, I must put my pride away, accept I cannot be one of the cool kids running Ansible for everything, and see what I can do with what I have.

Back to the Drawing Board

Let's start by getting away from the computer (figuratively speaking unless you printed this) and think on what the goals of this mess are

1. Connect to the ESXi-based vm host, who henceforth shall be named vmhost2.
   
2. Tell vmhost2 to save all the running vm clients, preserving their state.
   
3. Shut vmhost2 down.
   
4. Connect back to vmhost2 after it is back up and running, or when we feel like.
   
5. Tell vmhost2 to resume all those saved vm clients.
   

We already wrote the script to do the suspending and removing business a while ago, so two of those goals have been taken care of.

My code usually starts as a (Linux) shell script unless that is not appropriate for the task (say I need a Windows-only solution), so let me do some thinking aloud in shell before going fancy.

So, we need to connect to vmhost2 in a somewhat secure way. How about ssh? We can create a key pair using ssh-keygen; I would suggest making it at least 4096 bits long. Now we just need to copy the key to vmhost2. I will break some security principles and say we will connect to vmhost2 as root; adjust this as needed. With that in mind, we put the key in /etc/ssh/keys-root/authorized_keys, which probably behaves like your garden-variety authorized_keys file, i.e. command restrictions like we mentioned in an previous article should work. So, let's test it by connecting to vmhost2 and seeing what junk we have in the root dir:

[raub@desktop ~]$ ssh -C -tt -o IdentityFile=keys/vmhost2 -o User=root vmhost2 ls
altbootbank      lib64            sbin             var
bin              local.tgz        scratch          vmfs
bootbank         locker           store            vmimages
bootpart.gz      mbr              tardisks         vmupgrade
dev              opt              tardisks.noauto
etc              proc             tmp
lib              productLocker    usr
Connection to vmhost2 closed.
[raub@desktop ~]$

I think we can use that. Some of you might have noticed we are being a bit explicit with our options, using -o SomeOption when a shortcut would do, but we should make it easier for us to see what is going on, specially if we plan on coding that. And we shall: here is a Bourne (!) shell version of what we just did

#!/bin/sh

user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

ssh -C -tt -o IdentityFile="$KEY"  -o PasswordAuthentication=no \
    -o PreferredAuthentications=publickey \
    -o ConnectTimeout=10 \
    -o User=$user $host \
    $what

Yes, we told it to run a few more commands while it was connected, but the principle is the same: we can send commands to the ESXi box through ssh. Now, those of you who are well-versed with ssh could not be remotely impressed even if you tried really hard. And I agree; I am just laying down the building blocks. If you have been in this blog before, you would know that is how I roll.

Note: I will be honest with you and say sending out command is great, but we need to get replies for this to really be useful. Later on we will revisit this, for now assume it is all rainbows and unicorns.

I think we should stop messing around and switch the Bash so we can use functions. Maybe something like this perhaps?

#!/bin/bash

send_remote_command ()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

send_remote_command $host $user $KEY $what

As others have pointed out, passing parameters to a bash function is not fun. No matter, let's try it out:

[raub@desktop esxisave]$ ./esxitest.sh
~ # whoami

Er, that was not supposed to happen. Did I just connect to the vmhost2?

~ # whoami
/bin/sh: whoami: not found
~ # pwd
/
~ # hostname
vmhost2.example.com.
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

What's going on?

[raub@desktop esxisave]$ bash -x esxisave.sh
+ user=root
+ host=vmhost2
+ KEY=keys/vmhost2
+ what='/bin/sh -c '\''hostname'\'' && /bin/sh -c '\''date'\'' && /bin/sh -c '\''uptime'\'''
+ send_remote_command vmhost2 root keys/vmhost2 /bin/sh -c ''\''hostname'\''' '&&' /bin/sh -c ''\''date'\''' '&&' /bin/sh -c ''\''uptime'\'''
+ local hostname=vmhost2
+ local username=root
+ local keyfile=keys/vmhost2
+ local command=/bin/sh
+ ssh -C -tt -o IdentityFile=keys/vmhost2 -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o ConnectTimeout=10 -o User=root vmhost2 /bin/sh
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Duh! Escaping quotes. I think we are overthinking this.

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

So we run it again:

[raub@desktop esxisave]$ ./esxitest.sh
vmhost2.example.com.
Sat Sept 26 05:05:00 UTC 2015
 05:05:00 up 04:50:20, load average: 0.01, 0.01, 0.01
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Much better. Now we could add other functions to our script; first thing that comes to mind is making the script able to upload a file. You see, a while ago we wrote a script to suspend and resume vm clients, we might as well find a way to upload it. Since we are using ssh, it would make sense to then ask scp for help. And we shall.

Let's put a file over there

We have established we can remotely execute commands in the ESXi box vmhost2 without adding any extra packages to out desktop. On the last paragraph we promised we would show how to upload our old script tho shutdown vmhost2; we better deliver it, and we shall do it using scp if for no other reason than we hinted we were going to use it. Now, based on what we have already tested using ssh, to upload the script we might do something like

scp -i keys/vmhost2.example.com root@vmhost2:/var/tmp/save_runningvms.sh .

For now we will be using /var/tmp/ since it tends to survive a reboot but it is still a temp directory; I am not ready yet to start cluttering vmhost2's OS drive. We could modify esxitest.sh to include uploading support, say

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"
 
    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

send_file()
{
    # Reading function parameters in bash is still not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local localfile="$4"
    local remotefile="$5" # a bit of a misowner; it is more like the full path

    scp -i "$keyfile" $localfile $username@$hostname:$remotefile
}

# Let's try our little function out
user=root
host="vmhost2.in.kushana.com"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

# Now we send a file
sourcepath="files/save_runningvms.sh" # Yes, relative path because we can
destinationpath="/var/tmp/save_runningvms.sh" # Absolute path when it makes sense

send_file $host $user $KEY $sourcepath $destinationpath

Let's try it out: First we will check if there is nobody on /var/tmp

[root@vmhost2:~] ls /var/tmp/
sfcb_cache.txt
[root@vmhost2:~]

There is somebody there but nobody relevant to this article. So let's run the new script:

[raub@desktop esxisave]$ ./esxisave.sh 
vmhost2
Fri Jan  1 05:04:47 UTC 2016
  5:04:47 up 64 days, 22:37:34, load average: 0.01, 0.01, 0.01
Connection to vmhost2.in.kushana.com closed.
save_runningvms.sh                            100%  918     0.9KB/s   00:00    
[raub@desktop esxisave]$ 

Did you noticed the very last line? It sure looks very scpish to me. Were all of our efforts in vain or we now have a new file in /var/tmp?

[root@vmhost2:~] ls /var/tmp/
save_runningvms.sh  sfcb_cache.txt
[root@vmhost2:~] 

Of course we could have loaded the file and then remotely set its permissions and executed it, all from our little bash script. That hopefully gives you lots of ideas of where to go next.

That is fine, but why can't I run this script by passing the parameters from command line with options so it knows when I want to send a command or a file? And why doesn't it process the return messages? Do you also want me to give you some warm Ovomaltine and fluff your pillow? Seesh! I am just giving a basic example, a building block to solve whatever task we are talking about in a given article; you can embellish it later. I am not here to hold your hand but (hopefully) set you in the right direction.

Save/Suspend and Resume a VMware ESXi vm client command line style

Here is an interesting project: let's say you have one or more UPS (one per power supply) attached to your ESXi vm host (or hosts; this is completely scalable). Yes, it goes without saying providing uninterrupted power to your servers is a good idea. But, unless you are a large company chances are this power will only last so long. You can make it last even longer by having a plan that will decide in which order your physical servers will be shut down based on load and remaining power. That does mean shutting down your vm servers; for the sake of this discussion, we will assume they are ESXi-based.

I have seen interesting articles on shutting down ESXi hosts on case of power failure, but many assume you are monitoring the UPS through the ESXi host. That might be thinking small; what if that is not the case? What if you have a UPS or two on the bottom feeding the entire cabinet? Chances are you will be monitoring it from a host, be it a vm or not, that is running some monitoring program such as Nagios, that is set to do something in case of a power failure. Of course, if you have a monitoring vm you can talk to your UPS using either ethernet or USB passthrough depending on how sophisticated that model is. And it will decide when to tell our ESXi box it is time to shut down.

I do not know about you but I would like to gracefully save/shutdown the vm clients running in that host before that.

The plan is to have the host monitoring the UPS tell the ESXi host to run a shutdown procedure, which would need to first save the vm guests. And, once the vm server is back up and running, it would resume -- by its own accord or by the order of another server -- the saved vm clients. Yes, you will have to worry about how the monitoring and the ESXi hosts will talk to each other and how the client's clock will catch up, but for this article we will focus on creating a tool that only cares about saving and resuming all of the vm guests running in this ESXi box. We can expand later.

If we want to save the running vm clients, we probably should find out which ones are running. In a previous article we wrote a script to see if a given vm client is running, off, or saved. For the script we will be creating, we want to use something else, vmdumper. Here is what the help screen for the program says.

/tmp # vmdumper -h
vmdumper: [options]  
         -f: ignore vsi version check
         -h: print friendly help message
         -l: print information about running VMs
         -g: log specified text to the vmkernel log
/tmp #

Note the -l shows only the running vms, which is what we want to do. So, let's run that and see what it spits back (I will break them a bit so they will kinda fit the screen):

~ # /sbin/vmdumper -l
wid=264397      pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/boot2docker/boot2docker.vmx" uuid="56 4d 11 2b 63 bc 88 fb-d9 e1 
93 fc 69 36 66 45"  displayName="boot2docker"       vmxCartelID=264396
wid=13080       pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/Windows 2012/Windows 2012.vmx"       uuid="56 4d e7 cb 24 11 63 
13-04 0d 9b 41 08 f9 a3 be"  displayName="Windows 2012"      vmxCartelID=13079
wid=527962      pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/devcentos/devcentos.vmx"     uuid="56 4d d7 e8 25 6c de 91-09 38 
60 ce ab 5d 43 ca"  displayName="devcentos" vmxCartelID=527961
~ #

As you can see, it shows the path for the config file the vm guest is using (cfgFile, its name (displayName) and something called wid. And a few other things I do not feel like caring about. So, how do we save a vm anyway? We know we can start a vm using vim-cmd vmsvc/power.on, so maybe it sounds similar. Some frustrating searching later we find that http://www.vi-toolkit.com/wiki/index.php/Vmsvc/power.hibernate might be a candidate. Thing is it needs wmid as the argument. I will save some time and state (have faith, brother!) it can be obtained by

vim-cmd vmsvc/getallvms | grep "${displayName}" | awk '{ print "vmid=" $1}'

But, does it really work? We shall try with devcentos, which happens to have wmid=3 (again, I cheat because I have spent loads of time testing this):

/tmp # vim-cmd  vmsvc/power.hibernate 3
(vim.fault.ToolsUnavailable) {
   dynamicType = ,
   faultCause = (vmodl.MethodFault) null,
   msg = "Cannot complete operation because VMware Tools is not running in this virtual machine.",
}
/tmp #

And it does not seem to want to work. It needs VMware Tools, and I do not want to worry about it. So let's see what else we can use. After some looking I found vmdumper. To save devcentos we could do

/tmp # vmdumper 527962 suspend_vm
Suspending VM...
/tmp # 

The weird number 527962 is the world id or wid for devcentos, which happens to be the first column in the output of vmdumper -l associated with that vm client.

Pet Peeve: If you remember the output of vmdumper -h, which should be the help page for that command, mentions nothing about suspend-vm. Good job, VMware! That does make me wonder what else you are not documenting...

Now my venting is done, let's see what we need.

1. We need the wid to shut down with vmdumper
   
2. We can resume (I tested already, and so can you!) the vm client using vim-cmd vmsvc/power.on. Thing is it needs wmid as the argument, which we figure out how to get above.
   
3. We then need a way to save wmid so when we can restore the saved vms. Probably saving the names of the vms would also be a nice touch.
   

So, here is the script I wrote to save and restore the running vms. As you can see, it is rather dumb since it is an all or nothing kinda deal. It is also unforgiving: if you run it again to save vms, the old /var/tmp/save_vms file will be overwritten. For what I wrote this script for, that is but a small annoyance.

cat > save_runningvms.sh  << 'EOF'
#!/bin/sh
IFS=$'\n'
USAGE="Usage: $0 {save|resume}"
SAVE_FILE=/var/tmp/save_vms

if [ "$#" == "0" ]; then
        echo "$USAGE"
        exit 1
fi

selection=$1

case $selection in
   # If we want to save them
   save )
      rm -f ${SAVE_FILE}

      # Find which vms are currently running
      for i in $(vmdumper -l \
         | awk ' BEGIN { FS = "\t" }; { print $1 ";" $5 }')
      do
         eval $i
         # Start saving them
         vmid=$(vim-cmd vmsvc/getallvms | grep "${displayName}" \
            | awk '{ print "vmid=" $1}')
         vmdumper $wid suspend_vm

         # Write list of saved guests in $SAVE_FILE
         echo $vmid ";" $i >> ${SAVE_FILE}
      done
      ;;
   # If we want to restore them
   resume )
      # Get list of saved guests
      for i in $(cat ${SAVE_FILE})
      do
         # Wake them up
         eval $i
         vim-cmd vmsvc/power.on $vmid
      done
      ;;
esac
EOF
chmod +x save_runningvms.sh

You will note that I avoid using Bashisms because the shell in busybox is closer to Bourne than Bash.

I think you probably want to see it running. So, let's run it. First we do some saving

/tmp # ./save_runningvms.sh save
Suspending VM...
Suspending VM...
Suspending VM...
/tmp # 

Did it create the /var/tmp/save_vms file? If so, how does it look like?

/tmp # cat /var/tmp/save_vms
vmid=24 ; wid=5718058;displayName="boot2docker"
vmid=23 ; wid=5714001;displayName="Windows 2012"
vmid=3 ; wid=5715871;displayName="devcentos"
/tmp # 

Ok, I am not convinced. You must be lying. Lemme go to the other vmhost, vmhost, and ping devcentos

[raub@vmhost tmp]# ping devcentos
PING devcentos.example.com (10.0.0.112) 56(84) bytes of data.
From vmhost.example.com (10.0.0.19) icmp_seq=2 Destination Host Unreachable
From vmhost.example.com (10.0.0.19) icmp_seq=3 Destination Host Unreachable
From vmhost.example.com (10.0.0.19) icmp_seq=4 Destination Host Unreachable
^C
--- devcentos.example.com ping statistics ---
7 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6125ms
pipe 3
[raub@vmhost tmp]# 

Hmmmm, okay. But maybe it was off and you were lying to me. So, let's see about waking up the sleeping vms.

/tmp # ./save_runningvms.sh resume
Powering on VM:
Powering on VM:
Powering on VM:
/tmp #

And then pinging devcentos

[raub@vmhost tmp]# ping devcentos
PING devcentos.example.com (10.0.0.112) 56(84) bytes of data.
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=1 ttl=64 time=212 ms
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=2 ttl=64 time=0.316 ms
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=3 ttl=64 time=0.313 ms
^C
--- devcentos.example.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2078ms
rtt min/avg/max/mdev = 0.313/70.992/212.349/99.954 ms
[raub@vmhost tmp]#

I guess the script does work after all. What's the world coming to?

Get status of local VMware ESXi guests using command line

I have been talking a lot about KVM, and showing examples of how to control it from the command line. In fact, just a few posts ago we were talking about USB passthrough in KVM. I guess it is high time to do some in ESXi. After all, some of you -- yours truly included -- also have to deal with ESXi, which is VMware's product.

At this point in the presentation I can see some of you stomping your chests shouting "ESXi is Enterprise level product, not to be compared to the amateur hour likes of Kay-Vee-Em!". I have bad news for you, sunshine: they are all the same. They all can do live migration and clustering and so on. RedHat builds their enterprise turnkey solutions around KVM, which they also own. The differences between them are moving targets. And they are not the only games in town. Deal with it.

Back to the topic, yes ESXi has a nice, albeit C#-dependent (i.e. Windows only), GUI client... which they have been trying to get rid of for a while. But, sometimes I (I will take full blame for this) want to do something that is not available in the GUI. Now, I am aware I could be using PowerCLI, but what if I do want to have something running automagically in the ESXi host? To see what I mean, let's use a simple example: suposed we want to have a nice list of which vm clients ar ein this ESXi host and what they are up to (running/paused/etc). If you can't figure out why we would want to do that right now, hold onto your seat until the later parts of this article.

Of course this assumes you can ssh into the ESXi host.

The command we want to use is vm-support, and the option is -listmvs:

vm-support --listvms
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/mail_1/mail_1.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/Windows 2012/Windows 2012.vmx (Running)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/coreos/coreos.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devnetbsd/devnetbsd.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/centos64/centos64.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/boot2docker/boot2docker.vmx (Running)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/TheOnion/TheOnion.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devubuntu/devubuntu.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/freebsd/freebsd.vmx (Registered)
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devcentos/devcentos.vmx (Registered)
#

As you can see, it shows where each VM disk image (yes, yes, I am lazy but this is just a little blog post. Focus, focus, focus) and a status of Running or Registered. It is a good start; we feel good about ourselves until we realize that Registered canmean the vm is turned off or just saved. Bummer.

Now I know that when you save a vm, it creates a file with a .vmss extension that stores the state of the vm when it was saved. Let me show an example: I know that devcentos is saved for I did that last week. If what I said before is not a lie, we should find a .vmss file:

# ls /vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devcentos/*.vmss
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/devcentos/devcentos-aaf17b2a.vmss
#

For those of you who read this blog (why would you do that?), that sounds just like the .save file KVM uses to do the very same thing. Didn't I mention in the end of the day they are the same? So, what we need to do is if we find a vm labelled as Registered, we should also see if it has a .vmss file. If so, it is a saved vm. I could bore you with the details and trials, but here is what I got:

cat > vm_status.sh  << 'EOF'
#!/bin/sh
IFS=$'\n'

for i in $(vm-support --listvms)
do 
   vm_path=$(dirname "$i")
   vm_name=$(basename $vm_path)
   vm_status=$(echo "$i" | awk '{ print $NF}' )

   # Now check status of each vm
   case $vm_status in
      '(Running)' )
         echo "$vm_name is currently running"
         ;;
      '(Registered)' )
         test -f $vm_path/*.vmss \
            && echo "$vm_name is currently saved" \
            || echo "$vm_name is currently off"
         ;;
   esac

done
EOF
chmod +x vm_status.sh

It is not perfect, but it does what I want:

# ./vm_status.sh 
mail_1 is currently off
Windows 2012 is currently running
coreos is currently off
ctfbox is currently off
devnetbsd is currently off
centos64 is currently off
boot2docker is currently running
TheOnion is currently saved
devubuntu is currently off
freebsd is currently off
devcentos is currently saved
#

One thing you need to be aware is that ESXi uses busybox, so do not try to write a full blown Bash script; you will end very disappointed.

"So, what is the point of this script as it runs in the ESXi host?" you might ask, and you have a point. By itself it makes more sense to wire it in PowerCLI and run it that way. But, what if you want to have a script send you periodic reports of which vms are alive and which ones are off? Or what if you need to see if you need to start a given vm or just restore it? This script is small and a bit of a simpleton, but it shows the potential we have for writing proper scripts to be run in the ESXi host. I will show an even more practical example in an upcoming article.

Running a web app from command line using powershell

Had a interesting task: we have an web-based program that is run by connecting to a given website. Arguments are passed in the url. So far so good. But we want to run the program at scheduled times, and that means it must be run automagically. And the host that will be used to call the program runs Windows. How about that for requirements?

I decided that would be a good exercise in powershell. Why doing it in powershell instead of something in visual-something-or-another? Well, the honest answer is that I suck at those visual thingies. However, instead of telling the truth I will instead say: since I came from a Linux/Unix background, I would prefer to use some sort of script if I can and save the bloatware associated with building a program for another time. If I were to do this in the Linux camp, I would probably do it in Bourne, tcsh, or bash. Or python if I felt it required some degree of sophistication. On the Windows camp, however, the closest thing to those shells is Powershell. And if you have used it, it ain't bad at all.

This might be a bit long since I will go over how this evolved; go grab some popcorn.

In Windows you can call a web browser from the command line feeding it the url for whatever site you want it to open. Something like this:

explorer "http://unixwars.blogspot.com"

would pop up this very site on your default browser. Problem with that is, well, you are loading a web browser. There are times you do want to pop a browser, but if you want to run something in the background that might not be a good idea. After all, would you want to be typing the most amazing program ever written in Ook! when suddenly a web browser comes up taking you to some website you never heard of? You know, like spammers love to do, but instead point to a page without boobs or free money offers. Even if you know it is the web-based program mentioned above being called, that would get old really quickly. So, we need another option.

Powershell allows you to call some .net libraries and classes. One of those classes is System.Net.WebRequest. Here's an example of how we could use it:

$request = [System.Net.WebRequest]::Create("http://www.example.com/nagios")
$reply = $request.GetResponse()

The first line creates the request to connect, in this case, to my nagios server. But, that does not send any traffic yet. We need to send the request, which is the job of the last line. It also captures the reply sent by the server. Now what should I get by sending a carefully improper request like that? An error message. What kind of message, you may ask. Let's see what you would get back using netcat (from my Linux laptop):

raub@black:~$ nc -v "www.example.com" 80
Connection to www.example.com 80 port [tcp/http] succeeded!
GET http://www.example.com/ HTTP/1.0
HTTP/1.1 400 Bad Request
Date: Mon, 06 Oct 2014 05:18:50 GMT
Server: Apache/2.2.15 (CentOS)
Content-Length: 311
Connection: close
Content-Type: text/html; charset=iso-8859-1





Bad Request
Your browser sent a request that this server could not understand.
Apache/2.2.15 (CentOS) Server at www.example.com Port 80
raub@black:~$ 

As you can see we get a 400 Error. So, the above commands should spit back a similar message. Note at this point I do not care about getting a proper reply; I just want to connect to the web server.

Once I do that, we should take a look at the Apache access.log:

10.0.0.102 - - [)6/Oct/2014:00:18:50 -0500] "GET http://ww.example.com/ HTTP/1.0" 400 311 "-" "-"

As you can see, the logs register when netcat reached the server. Now, let's see if we can repeat the deed using powershell. I am going to run the two lines I mentioned above but as a powershell script, which I shall call gimmesite.ps1:

PS C:\Users\Administrator\dev> cat .\gimmesite.ps1
$request = [System.Net.WebRequest]::Create("http://www.example.com")
$reply = $request.GetResponse()
PS C:\Users\Administrator\dev>

Let's try to run it:

PS C:\Users\Administrator\Documents\dev> .\gimmesite.ps1
.\gimmesite.ps1 : File C:\Users\Administrator\Documents\dev\gimmesite.ps1 cannot be loaded because running scripts is
disabled on this system. For more information, see about_Execution_Policies at http://go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ .\gimmesite.ps1
+ ~~~~~~~~~~~~~~~
    + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess
PS C:\Users\Administrator\Documents\dev>

That does not look very happy. As the error message tries to tell us, the windows box is setup not to run any powershell script. You would need to certify it, which is something I have yet to do. There is a workaround, however which is mentioned in this thread:

PS C:\Users\Administrator\dev> powershell -ExecutionPolicy ByPass -File .\gimmesite.ps1
PS C:\Users\Administrator\dev>

As you can see, no error messages this time since we asked for an exception. Did apache see out connection attempt?

==> /var/log/httpd/access_log <==
10.0.0.105 - - [06/Oct/2014:00:23:50 -0500] "GET / HTTP/1.1" 200 - "-" "-"

It seems we are making progress. Next step is a small correction in the url we are using. You see, the site we are using to test out is http://www.example.com/nagios, not http://www.example.com/. So we edit the gimmesite.ps1 script and try again:

PS C:\Users\Administrator\dev> powershell -ExecutionPolicy ByPass -File .\gimmesite.ps1
Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (401) Unauthorized."
At C:\Users\Administrator\dev\gimmesite.ps1:2 char:1
+ $reply = $request.GetResponse()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException

PS C:\Users\Administrator\dev>

The error message makes sense: you need login credentials to access my nagios page. If we remember the objective of the script we are trying to develop in this article -- a powershell script that runs a script in a webpage at certain intervals -- we realize we have no interest in any error message. So, we need to get rid of it. Now, powershell has some structures similar to Java and Python. The one of interest here is the try{}catch{} one. So, let's modify the script once more:

PS C:\Users\Administrator\dev> cat gimmesite.ps1
# Connect to some site we told it about
# Run it as
# powershell -ExecutionPolicy ByPass -File .\gimmesite.ps1

$source = "http://www.example.com/nagios"
try
{
        # Let's try to reach site with our noodly tentacles:
        $request = [System.Net.WebRequest]::Create($source)
        $reply = $request.GetResponse()
}
catch
{
        # I tried really hard to care about error messages but I failed
}
PS C:\Users\Administrator\dev>

I hope the comments help a bit understand what is going on. When we run that, error messages are caught by the catch{} statement. Since we do nothing with them (we could in a fancier program), they are never printed to the screen/stdin:

PS C:\Users\Administrator\dev> powershell -ExecutionPolicy ByPass -File .\gimmesite.ps1
PS C:\Users\Administrator\dev>

So we have a working script. How about running it at predetermined intervals? Windows has no cron command, but there are equivalent commands. In fact, powershell has a set of commands to schedule tasks... which I will not use. Since this blog entry is getting long and I am getting tired of typing, I will instead use schtasks, which has been around since Windows XP. Let me add a comment to the script describing how to use it:

PS C:\Users\Administrator\dev> cat gimmesite.ps1
# Connect to some site we told it about
# Run it as
# powershell -ExecutionPolicy ByPass -File .\gimmesite.ps1
# And from the cronjobbie
# Schtasks /create /tn "Connect to site" /sc daily /st 07:00 /tr "powershell -ExecutionPolicy ByPass -File .\gimmesite.ps1"

$source = "http://www.example.com/nagios"
try
{
        # Let's try to reach site with our noodly tentacles:
        $request = [System.Net.WebRequest]::Create($source)
        $reply = $request.GetResponse()
}
catch
{
        # I tried really hard to care about error messages but I failed
}
PS C:\Users\Administrator\dev>

The above comment shows how to setup a "schedule task" (fancy term for a cron job) that will be run every day at 7am. Since I want to test this, let's make it run every 10 minutes instead:

PS C:\Users\Administrator\Documents\dev> schtasks /create /tn "Connect to site" /sc minute /mo 10 /tr "powershell -execution policy bypass -file C:\Users\Administrator\Documents\dev\gimmesite.ps1"
WARNING: The task name "Connect to site" already exists. Do you want to replace it (Y/N)? y
SUCCESS: The scheduled task "Connect to site" has successfully been created.
PS C:\Users\Administrator\Documents\dev> schtasks /query|more

Folder: \
TaskName                                 Next Run Time          Status
======================================== ====================== ===============
Connect to site                          11/6/2014 12:46:00 PM  Ready

Folder: \Microsoft
TaskName                                 Next Run Time          Status
======================================== ====================== ===============
INFO: There are no scheduled tasks presently available at your access level.
[...]

You can see that it barked I already had created the task before. Since I want to change it, I told it to just replace it. You probably also noticed that the task is being identified by its name (the /tn option). I will not spend any time at all describing the different options available in schtasks; there are beautiful pages online describing that. Or you can do schtasks /query /?. One final point: the task name is rather important for it is how you refer to it if you have to manipulate it somehow. How about if we delete the task?

PS C:\Users\Administrator\Documents\dev> schtasks /delete /tn "Connect to site"
WARNING: Are you sure you want to remove the task "Connect to site" (Y/N)? y
SUCCESS: The scheduled task "Connect to site" was successfully deleted.
PS C:\Users\Administrator\Documents\dev>

I hope this is enough to get you started and give you evil ideas. I put a slightly more complex version of this script in github.

On directory/folder and group ownership

Easy problem here: we have this directory, say /export/projects/web which is supposedly owned by the group developers,

raub@banana:~$ ls -lhd /export/projects/web
drwxrwxr-x 19 bob developers 4.0K 2013-03-19 13:36 /export/projects/web
raub@banana:~$

The idea is that it is a shared folder, a place the developers can put share files amongst themselves without others being able to change/delete them. In other words, we want any file or directory created inside projects to inherit its group ID. At east that is the idea. In reality when any member of that group creates a file there, it is owned by that user's default group, not by developers.

raub@banana:~$ touch /export/projects/web/here
raub@banana:~$ ls -lh /export/projects/web
-rw-rw-r--. 1 raub raub       0 Mar 19 13:15 here
raub@banana:~$ 

We can do something about it. First we set the setgid bit to make sure /export/projects/web is owned by developers:

raub@banana:~$ chmod g+s /export/projects/web
raub@banana:~$ ls -lhd /export/projects/web
drwxrwxr-x 19 bob developers 4.0K 2013-03-19 13:36 /export/projects/web
raub@banana:~$ ls -lh /export/projects/web
-rw-rw-r--. 1 raub developers 0 Mar 19 13:15 here
raub@banana:~$

Then, we should find all files in that directory with different groups and set them to be owned by developers:

raub@banana:~$ for i in `find /export/projects/web -type f ! -group developers`; do chown :developers $i; done
raub@banana:~$

Did it work? Let's find out!

raub@banana:~$ touch /export/projects/web/here
raub@banana:~$ ls -lh /export/projects/web
-rw-rw-r--. 1 raub developers 0 Mar 19 13:15 here
-rw-rw-r--. 1 raub developers 0 Mar 19 13:15 there
raub@banana:~$ 

References:

http://superuser.com/questions/102253/how-to-make-files-created-in-a-directory-owned-by-directory-group
http://www.cyberciti.biz/faq/how-do-i-find-all-the-files-owned-by-a-particular-user-or-group/

Grep-based decisions

So I had an interesting problem today: I want to have a script that will decide what to do based
on whether a file has something or not. Specifically, I am running libvirt with KVM/QEMU and wanted to know if a given vm client was configured with PC passthrough. Now, there are a ton of ways to do that, but I wanted to do it using Bourne or bash and grep. You know, something like

root@vmhost:~# virsh dumpxml vmclient | grep "type='pci' managed='yes'"
    
root@vmhost:~# 

since if it has that pattern, it is passing PCI through. What I need then is to have grep look for that pattern and give me back some kind of return code that tells me if it is there or not.

At first I thought grep -q (quiet), as in

virsh dumpxml vmclient | grep -q "type='pci' managed='yes'" | echo $?

would work (you would need then to fish the return code using $?, which is why I put the echo $?). After all we only care whether it finds it or not. But the thing is that it would return 0 no matter it found the pattern or not. Major bummer.

Now, grep has this -c option that would count how many times it finds the pattern. Which probably leads to if it does not find it, the count should be 0, right? Let's test it out:

root@vmhost1:~# virsh dumpxml vmclient|grep -c "type='pci' managed='no'" 
0
root@vmhost1:~# virsh dumpxml vmclient |grep -c "type='pci' managed='yes'" 
1
root@vmhost1:~#

I think you can see where I am going with this. So, here is a scaled-down version of what I ended up writing:

#!/bin/sh
vmclient=$1
have_pci=$(virsh dumpxml ${vmclient} | grep -c "type='pci' managed='yes'" )

if [ "$have_pci" -eq 0 ]
then
   echo "${vmclient} is PCI passthrough-free. Rejoy!"
   # Do something interesting
else
   echo "${vmclient} has PCI passthrough. Be nice to it."
   # Do something interesting
fi

As you can see it puts our little research to work by detecting if the vm whose name you provide as command line argument ($1) is configured to do PCI passthrough or not. Since this is just a sample skeleton code, it just tells us whether it does or not (the if statement); we could then use that info to do something else. In fact, I am using that test in a larger script I wrote; I did not want to include it here because I wanted to focus on one thing.

I hope this might be useful to someone out there. I probably should talk about the pci passthrough madness, but let's leave that for another episode, shall we?

Restoring time on sleeping (linux) vms

One of the most annoying issues in a VM is keeping accurate time.When a vm client is saved/paused and then restored, its clock will be still set to whatever time was when it was paused. That can lead to many issues including not being able to login using Kerberos. Rebooting the machine will force the clock to be reset, but what if we do not want to (or cannot) reboot? What we could use a script that will monitor the drift and if it is too far -- say one minute off -- it will automagically adjust the clock in the vm client.
There are many virtualization packages out there, some of which (in no order) are VMWare ESX/ESXi, Virtual Box, kvm, and Microsoft Virtual Server. The first three I personally have used; if you want to see a more complete list I would suggest to try the Wikipedia entry on Hypervisors. Suffice to say some of those programs allow host machine (the physical machine running the program) expose its clock to the guest machine (or vm), others do not. So let's see how we can check and set the clock based on whether the guest can see the host clock or not. For this discussion I will stick to Linux because I feel lazy today.

Using the vm host clock

Let's say the vm program of your choice can pass the vm host's clock to the client (I do know that KVM can do that and believe VMWare ESXi can too), you could write something like this:

cat > /usr/local/bin/driftcheck << 'EOF'
#!/bin/sh
### Detect drift in the vm client vh most. If it is massive; adjust
### it.

HWCLOCK=/sbin/hwclock

# Max drift in seconds. Kerberos does not like time offsets > 5min, 
# so we set it to 1m = 1*60s
MAX_DRIFT=` echo "1*60" | bc -l`

VMHOST_TIME=`date +'%s' -d "$(${HWCLOCK} -r -u | cut -d' ' -f-7)"`
MY_TIME=`date +%s`
DRIFT=$( echo "${VMHOST_TIME} - ${MY_TIME}" | bc | tr -d -)

if [ "${DRIFT}" -gt "${MAX_DRIFT}" ]
then   
        ${HWCLOCK} -s -u
fi
EOF
chmod +x /usr/local/bin/driftcheck

What it does is get the hardware clock of this vm client, which actually is the vm host's clock. One thing this requires is the vm client is configured to take the vm host's clock in utc time. YMMV here, but if you are using kvm and libvit, you would have a line like this

<clock offset="utc"></clock>

somewhere in the xml file defining the client.
The commented lines inside the if statement are there just so when you test it out you can see what is going on. For production you probably want leave them commented out
Now, we would probably want to to have it being called often but not really crazy. So, for now let's say we create a cron job to run driftcheck every 5 minutes?

cat > /etc/cron.d/driftcheck << 'EOF'
*/5 * * * * root /usr/local/bin/driftcheck > /dev/null 2>&1
EOF

Of course, you should adjust it to fit your needs. Do note I put driftcheck in /usr/local/bin/; it just felt like a nice place this season.

Using NTP

As we mentioned above, sometimes we cannot (or will not) use the clock off the vm host. If we are using a ntp server, we are good. Ok, you might argue but, if the drift/skew is too long, ntpd will refuse to adjust the clock. And, even if we force it, it will take hours or even days!. Not if we nudge it a bit by using an old friend, ntpdate:

cat > /usr/local/bin/driftcheck << 'EOF'
#!/bin/sh
### Detect drift against a reliable time source. If it is massive;
### instead of relying on ntp (which will not work), do something
### a bit more drastic

NTPQ=/usr/bin/ntpq
NTPDATE=/usr/sbin/ntpdate

# Max drift in ms. Kerberos does not like time offsets > 5min, so
# we set it to 1min = 1*60*1000ms
MAX_DRIFT=` echo "60*1000" | bc -l`

# Find the ntp server we are using in this host
NTP_SERVER=`sed -ne '/^server/p' /etc/ntp.conf | awk '{ print $2 }'| head -1`
# Get current drift
DRIFT=`${NTPQ} -p ${NTP_SERVER}|grep '*'|awk '{print $9 }'`
[ -z $DRIFT ] && DRIFT=`${NTPQ} -p |tail -n +3| awk '{print $9 }'`
DRIFT=$( echo "${DRIFT}/1" | bc | tr -d -)

if [ "${DRIFT}" -gt "${MAX_DRIFT}" ]
then
        echo ${NTPDATE} -u ${NTP_SERVER} >> /tmp/ntp
        ${NTPDATE} -u ${NTP_SERVER} >> /tmp/ntp
fi
EOF
chmod +x /usr/local/bin/driftcheck

The reason we want to look for * is that according to the
NTP troubleshooting page, it is the source you are currently synchronized to.
The cron file will be the same as what we had in the hardware clock session, so I decided not to copy it.
And that is pretty much it. Remember they are just ideas. Those scripts should work as is but you can/should customize/adapt them. If you have any questions, concerns, or just want to confuse me, do leave a message.

References

• http://time.to.pullthepl.ug/blog/2012/8/29/easy-ntp-stats-without-ntpstat/
• https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.6_Technical_Notes/Known_Issues-kvm.html
• http://linux.derkeiler.com/Mailing-Lists/Fedora/2012-04/msg00204.html