Remotely controlling ESXi with bash and without Ansible or Windows

I would expect the more knowledgeable ESXi admins amongst you will rightfully mention the proper way to run a remote command in a ESXi box is using the vSphere (Web) Client or the vSphere Command-Line Interface. I agree completely, which is why I will ignore the advice and see what else I can use.

Shattering a Dream

Just to let you know, I really really REally wanted to have used Ansible here. In fact, this was supposed to be a post on how to use Ansible to shut down an ESXi server, saving the running vm clients, and then restore those clients after the server was back up. I spent time laying things out but could not get it working. I will not go into details; but at first I thought my issues were due to ESXi not having python-simplejson, but I learned that assumption was wrong; Python in ESXi has json in it:

I will try again some other time. For now, I must put my pride away, accept I cannot be one of the cool kids running Ansible for everything, and see what I can do with what I have.

Back to the Drawing Board

Let's start by getting away from the computer (figuratively speaking unless you printed this) and think on what the goals of this mess are

1. Connect to the ESXi-based vm host, who henceforth shall be named vmhost2.
   
2. Tell vmhost2 to save all the running vm clients, preserving their state.
   
3. Shut vmhost2 down.
   
4. Connect back to vmhost2 after it is back up and running, or when we feel like.
   
5. Tell vmhost2 to resume all those saved vm clients.
   

We already wrote the script to do the suspending and removing business a while ago, so two of those goals have been taken care of.

My code usually starts as a (Linux) shell script unless that is not appropriate for the task (say I need a Windows-only solution), so let me do some thinking aloud in shell before going fancy.

So, we need to connect to vmhost2 in a somewhat secure way. How about ssh? We can create a key pair using ssh-keygen; I would suggest making it at least 4096 bits long. Now we just need to copy the key to vmhost2. I will break some security principles and say we will connect to vmhost2 as root; adjust this as needed. With that in mind, we put the key in /etc/ssh/keys-root/authorized_keys, which probably behaves like your garden-variety authorized_keys file, i.e. command restrictions like we mentioned in an previous article should work. So, let's test it by connecting to vmhost2 and seeing what junk we have in the root dir:

[raub@desktop ~]$ ssh -C -tt -o IdentityFile=keys/vmhost2 -o User=root vmhost2 ls
altbootbank      lib64            sbin             var
bin              local.tgz        scratch          vmfs
bootbank         locker           store            vmimages
bootpart.gz      mbr              tardisks         vmupgrade
dev              opt              tardisks.noauto
etc              proc             tmp
lib              productLocker    usr
Connection to vmhost2 closed.
[raub@desktop ~]$

I think we can use that. Some of you might have noticed we are being a bit explicit with our options, using -o SomeOption when a shortcut would do, but we should make it easier for us to see what is going on, specially if we plan on coding that. And we shall: here is a Bourne (!) shell version of what we just did

#!/bin/sh

user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

ssh -C -tt -o IdentityFile="$KEY"  -o PasswordAuthentication=no \
    -o PreferredAuthentications=publickey \
    -o ConnectTimeout=10 \
    -o User=$user $host \
    $what

Yes, we told it to run a few more commands while it was connected, but the principle is the same: we can send commands to the ESXi box through ssh. Now, those of you who are well-versed with ssh could not be remotely impressed even if you tried really hard. And I agree; I am just laying down the building blocks. If you have been in this blog before, you would know that is how I roll.

Note: I will be honest with you and say sending out command is great, but we need to get replies for this to really be useful. Later on we will revisit this, for now assume it is all rainbows and unicorns.

I think we should stop messing around and switch the Bash so we can use functions. Maybe something like this perhaps?

#!/bin/bash

send_remote_command ()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

send_remote_command $host $user $KEY $what

As others have pointed out, passing parameters to a bash function is not fun. No matter, let's try it out:

[raub@desktop esxisave]$ ./esxitest.sh
~ # whoami

Er, that was not supposed to happen. Did I just connect to the vmhost2?

~ # whoami
/bin/sh: whoami: not found
~ # pwd
/
~ # hostname
vmhost2.example.com.
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

What's going on?

[raub@desktop esxisave]$ bash -x esxisave.sh
+ user=root
+ host=vmhost2
+ KEY=keys/vmhost2
+ what='/bin/sh -c '\''hostname'\'' && /bin/sh -c '\''date'\'' && /bin/sh -c '\''uptime'\'''
+ send_remote_command vmhost2 root keys/vmhost2 /bin/sh -c ''\''hostname'\''' '&&' /bin/sh -c ''\''date'\''' '&&' /bin/sh -c ''\''uptime'\'''
+ local hostname=vmhost2
+ local username=root
+ local keyfile=keys/vmhost2
+ local command=/bin/sh
+ ssh -C -tt -o IdentityFile=keys/vmhost2 -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o ConnectTimeout=10 -o User=root vmhost2 /bin/sh
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Duh! Escaping quotes. I think we are overthinking this.

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

So we run it again:

[raub@desktop esxisave]$ ./esxitest.sh
vmhost2.example.com.
Sat Sept 26 05:05:00 UTC 2015
 05:05:00 up 04:50:20, load average: 0.01, 0.01, 0.01
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Much better. Now we could add other functions to our script; first thing that comes to mind is making the script able to upload a file. You see, a while ago we wrote a script to suspend and resume vm clients, we might as well find a way to upload it. Since we are using ssh, it would make sense to then ask scp for help. And we shall.

Let's put a file over there

We have established we can remotely execute commands in the ESXi box vmhost2 without adding any extra packages to out desktop. On the last paragraph we promised we would show how to upload our old script tho shutdown vmhost2; we better deliver it, and we shall do it using scp if for no other reason than we hinted we were going to use it. Now, based on what we have already tested using ssh, to upload the script we might do something like

scp -i keys/vmhost2.example.com root@vmhost2:/var/tmp/save_runningvms.sh .

For now we will be using /var/tmp/ since it tends to survive a reboot but it is still a temp directory; I am not ready yet to start cluttering vmhost2's OS drive. We could modify esxitest.sh to include uploading support, say

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"
 
    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

send_file()
{
    # Reading function parameters in bash is still not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local localfile="$4"
    local remotefile="$5" # a bit of a misowner; it is more like the full path

    scp -i "$keyfile" $localfile $username@$hostname:$remotefile
}

# Let's try our little function out
user=root
host="vmhost2.in.kushana.com"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

# Now we send a file
sourcepath="files/save_runningvms.sh" # Yes, relative path because we can
destinationpath="/var/tmp/save_runningvms.sh" # Absolute path when it makes sense

send_file $host $user $KEY $sourcepath $destinationpath

Let's try it out: First we will check if there is nobody on /var/tmp

[root@vmhost2:~] ls /var/tmp/
sfcb_cache.txt
[root@vmhost2:~]

There is somebody there but nobody relevant to this article. So let's run the new script:

[raub@desktop esxisave]$ ./esxisave.sh 
vmhost2
Fri Jan  1 05:04:47 UTC 2016
  5:04:47 up 64 days, 22:37:34, load average: 0.01, 0.01, 0.01
Connection to vmhost2.in.kushana.com closed.
save_runningvms.sh                            100%  918     0.9KB/s   00:00    
[raub@desktop esxisave]$ 

Did you noticed the very last line? It sure looks very scpish to me. Were all of our efforts in vain or we now have a new file in /var/tmp?

[root@vmhost2:~] ls /var/tmp/
save_runningvms.sh  sfcb_cache.txt
[root@vmhost2:~] 

Of course we could have loaded the file and then remotely set its permissions and executed it, all from our little bash script. That hopefully gives you lots of ideas of where to go next.

That is fine, but why can't I run this script by passing the parameters from command line with options so it knows when I want to send a command or a file? And why doesn't it process the return messages? Do you also want me to give you some warm Ovomaltine and fluff your pillow? Seesh! I am just giving a basic example, a building block to solve whatever task we are talking about in a given article; you can embellish it later. I am not here to hold your hand but (hopefully) set you in the right direction.