Remotely controlling ESXi with bash and without Ansible or Windows

I would expect the more knowledgeable ESXi admins amongst you will rightfully mention the proper way to run a remote command in a ESXi box is using the vSphere (Web) Client or the vSphere Command-Line Interface. I agree completely, which is why I will ignore the advice and see what else I can use.

Shattering a Dream

Just to let you know, I really really REally wanted to have used Ansible here. In fact, this was supposed to be a post on how to use Ansible to shut down an ESXi server, saving the running vm clients, and then restore those clients after the server was back up. I spent time laying things out but could not get it working. I will not go into details; but at first I thought my issues were due to ESXi not having python-simplejson, but I learned that assumption was wrong; Python in ESXi has json in it:

I will try again some other time. For now, I must put my pride away, accept I cannot be one of the cool kids running Ansible for everything, and see what I can do with what I have.

Back to the Drawing Board

Let's start by getting away from the computer (figuratively speaking unless you printed this) and think on what the goals of this mess are

1. Connect to the ESXi-based vm host, who henceforth shall be named vmhost2.
   
2. Tell vmhost2 to save all the running vm clients, preserving their state.
   
3. Shut vmhost2 down.
   
4. Connect back to vmhost2 after it is back up and running, or when we feel like.
   
5. Tell vmhost2 to resume all those saved vm clients.
   

We already wrote the script to do the suspending and removing business a while ago, so two of those goals have been taken care of.

My code usually starts as a (Linux) shell script unless that is not appropriate for the task (say I need a Windows-only solution), so let me do some thinking aloud in shell before going fancy.

So, we need to connect to vmhost2 in a somewhat secure way. How about ssh? We can create a key pair using ssh-keygen; I would suggest making it at least 4096 bits long. Now we just need to copy the key to vmhost2. I will break some security principles and say we will connect to vmhost2 as root; adjust this as needed. With that in mind, we put the key in /etc/ssh/keys-root/authorized_keys, which probably behaves like your garden-variety authorized_keys file, i.e. command restrictions like we mentioned in an previous article should work. So, let's test it by connecting to vmhost2 and seeing what junk we have in the root dir:

[raub@desktop ~]$ ssh -C -tt -o IdentityFile=keys/vmhost2 -o User=root vmhost2 ls
altbootbank      lib64            sbin             var
bin              local.tgz        scratch          vmfs
bootbank         locker           store            vmimages
bootpart.gz      mbr              tardisks         vmupgrade
dev              opt              tardisks.noauto
etc              proc             tmp
lib              productLocker    usr
Connection to vmhost2 closed.
[raub@desktop ~]$

I think we can use that. Some of you might have noticed we are being a bit explicit with our options, using -o SomeOption when a shortcut would do, but we should make it easier for us to see what is going on, specially if we plan on coding that. And we shall: here is a Bourne (!) shell version of what we just did

#!/bin/sh

user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

ssh -C -tt -o IdentityFile="$KEY"  -o PasswordAuthentication=no \
    -o PreferredAuthentications=publickey \
    -o ConnectTimeout=10 \
    -o User=$user $host \
    $what

Yes, we told it to run a few more commands while it was connected, but the principle is the same: we can send commands to the ESXi box through ssh. Now, those of you who are well-versed with ssh could not be remotely impressed even if you tried really hard. And I agree; I am just laying down the building blocks. If you have been in this blog before, you would know that is how I roll.

Note: I will be honest with you and say sending out command is great, but we need to get replies for this to really be useful. Later on we will revisit this, for now assume it is all rainbows and unicorns.

I think we should stop messing around and switch the Bash so we can use functions. Maybe something like this perhaps?

#!/bin/bash

send_remote_command ()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

send_remote_command $host $user $KEY $what

As others have pointed out, passing parameters to a bash function is not fun. No matter, let's try it out:

[raub@desktop esxisave]$ ./esxitest.sh
~ # whoami

Er, that was not supposed to happen. Did I just connect to the vmhost2?

~ # whoami
/bin/sh: whoami: not found
~ # pwd
/
~ # hostname
vmhost2.example.com.
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

What's going on?

[raub@desktop esxisave]$ bash -x esxisave.sh
+ user=root
+ host=vmhost2
+ KEY=keys/vmhost2
+ what='/bin/sh -c '\''hostname'\'' && /bin/sh -c '\''date'\'' && /bin/sh -c '\''uptime'\'''
+ send_remote_command vmhost2 root keys/vmhost2 /bin/sh -c ''\''hostname'\''' '&&' /bin/sh -c ''\''date'\''' '&&' /bin/sh -c ''\''uptime'\'''
+ local hostname=vmhost2
+ local username=root
+ local keyfile=keys/vmhost2
+ local command=/bin/sh
+ ssh -C -tt -o IdentityFile=keys/vmhost2 -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o ConnectTimeout=10 -o User=root vmhost2 /bin/sh
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Duh! Escaping quotes. I think we are overthinking this.

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

So we run it again:

[raub@desktop esxisave]$ ./esxitest.sh
vmhost2.example.com.
Sat Sept 26 05:05:00 UTC 2015
 05:05:00 up 04:50:20, load average: 0.01, 0.01, 0.01
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Much better. Now we could add other functions to our script; first thing that comes to mind is making the script able to upload a file. You see, a while ago we wrote a script to suspend and resume vm clients, we might as well find a way to upload it. Since we are using ssh, it would make sense to then ask scp for help. And we shall.

Let's put a file over there

We have established we can remotely execute commands in the ESXi box vmhost2 without adding any extra packages to out desktop. On the last paragraph we promised we would show how to upload our old script tho shutdown vmhost2; we better deliver it, and we shall do it using scp if for no other reason than we hinted we were going to use it. Now, based on what we have already tested using ssh, to upload the script we might do something like

scp -i keys/vmhost2.example.com root@vmhost2:/var/tmp/save_runningvms.sh .

For now we will be using /var/tmp/ since it tends to survive a reboot but it is still a temp directory; I am not ready yet to start cluttering vmhost2's OS drive. We could modify esxitest.sh to include uploading support, say

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"
 
    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

send_file()
{
    # Reading function parameters in bash is still not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local localfile="$4"
    local remotefile="$5" # a bit of a misowner; it is more like the full path

    scp -i "$keyfile" $localfile $username@$hostname:$remotefile
}

# Let's try our little function out
user=root
host="vmhost2.in.kushana.com"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

# Now we send a file
sourcepath="files/save_runningvms.sh" # Yes, relative path because we can
destinationpath="/var/tmp/save_runningvms.sh" # Absolute path when it makes sense

send_file $host $user $KEY $sourcepath $destinationpath

Let's try it out: First we will check if there is nobody on /var/tmp

[root@vmhost2:~] ls /var/tmp/
sfcb_cache.txt
[root@vmhost2:~]

There is somebody there but nobody relevant to this article. So let's run the new script:

[raub@desktop esxisave]$ ./esxisave.sh 
vmhost2
Fri Jan  1 05:04:47 UTC 2016
  5:04:47 up 64 days, 22:37:34, load average: 0.01, 0.01, 0.01
Connection to vmhost2.in.kushana.com closed.
save_runningvms.sh                            100%  918     0.9KB/s   00:00    
[raub@desktop esxisave]$ 

Did you noticed the very last line? It sure looks very scpish to me. Were all of our efforts in vain or we now have a new file in /var/tmp?

[root@vmhost2:~] ls /var/tmp/
save_runningvms.sh  sfcb_cache.txt
[root@vmhost2:~] 

Of course we could have loaded the file and then remotely set its permissions and executed it, all from our little bash script. That hopefully gives you lots of ideas of where to go next.

That is fine, but why can't I run this script by passing the parameters from command line with options so it knows when I want to send a command or a file? And why doesn't it process the return messages? Do you also want me to give you some warm Ovomaltine and fluff your pillow? Seesh! I am just giving a basic example, a building block to solve whatever task we are talking about in a given article; you can embellish it later. I am not here to hold your hand but (hopefully) set you in the right direction.

Joining two circles/cylinders in openscad 1: Geometry

Warning!

This article has more to do with geometry than IT in general. You might possibly be scared of it, and I think that repulse to math in general and geometry specifically is a valid reaction. and I would suggest you to skip to the next posting. Or you might want to go on if just for the pretty pictures. That suits me fine also; you are the one with decider here.

You have been warned!

Recently I decided to learn how to use openscad. As the name kinda implies, it is a (open source) program to build 3D solids. Main difference with the usual suspects is that it is not interactive: you create a file using your favourite text editor (or sed for that matter) and then compile it. That might sound a bit annoying but it can be quite convenient if you are doing engineering objects as you can define the different object that make it as separate functions, which can even become libraries. However, it does not have some features you expect to find in such programs, like fileting. They are still working on providing a official solution to that, but as of right now it ain't there.

Now, sometimes you can go around these limitations by using a bit of geometry. Let me show what I mean by using a simple example: Let's say I want to build a bracket that will hold two pipes together (think chainlink fence gate) that looks like this sketch:

For now we will ignore the pipe holes and look at the external shape of the object. Since we are going to have two circles internally, it would be nice if the ends had a circly feel to it. So, we shall start with two circles around the pipe holes. To make it look pretty, and easy when we do get to build it, how about if the pipe holes are 90% in diameter of those circles we are using to make the body? You know, something like the picture (done in inkscape without making any justice to its potential) on the right.

So we have the ends of the body, but what about the sides? We need to connect the two ends somehow.

The easiest thing I can think of is to use those lines representing the radius of each circle, which happen to be both vertical. We could just connect them on each side (showing just the upper side to keep figure clean and since the object will be symmetric with respect to the horizontal, henceforth called X, axis).

I don't know about you, but even though it might work around the larger (right) circle, it would look odd on the smaller (left) circle: the surface would become parallel to the X axis just before going vertical and then it would make a sharp turn and start moving away of this axis. It just looks like it was put together in a hurry, and I think we can do better.

And how do you plan on doing that? you might ask. Or not. But this is my article so I will answer that question. Our goal is to have the side to transition gently away from the circle. I think we are looking for some kind of curve that started tangent to one circle and then, without much ado, end tangentially to the other circle. Kinda like the picture on the left, with some simple gentle curve in the middle. That does look pretty, but it requires us to determine where it will move away from each circle. In other words, we might need to do some fiddling.

I am lazy, so I would like to keep any fiddling to a minimal. Therefore, I want a solution that makes more work for the computer and less to me. I therefore need a curve that I will provide as few parameters as possible and it will still end up nice and pretty. Euclidean Geometry has the answer: the easiest way to come up with a curved surface that touches both circles tangentially mathematically with the minimal amount of input is to use a third circle.

Here is what I have in mind in a drawing (most of the green circle has been omitted so we would not focus on it): the green circle touches each of the other two circles tangentially as we want. I know the curve in the drawing does not look as nice as the one in the previous drawing, but where it touches and the shape of the curve it creates between those points is determined solely by its radius. Specifically, the bigger the green circle is the closer the line between the blue circles will be to a straight line. The other required information is determined by the two pipes we mentioned before, namely:

1. Radius of the two blue circles
   
2. Distance between the two blue circles.
   

So far so good, thanks to geometry. Let's see if we can continue relying on it and come up with equations that will make our life lazier. First thing we will need is some lines connecting the centers of the three circles (as I mentioned before, I am ignoring the other side due to the symmetry the object has) together. As to be expected, that defines a triangle of some arbitrary (we did not specify any relationship between the radius of the 3 different circles. We will revisit that later) shape, which in our little drawing is represented by the black triangle.

Before we continue, we must do some labeling. So, the centers of the 3 circles are called A, B, and C, which as we know are the vertexes of the triangle ABC. The sides of the triangle are labelled a, b, and c where b is opposite to B and so on. We can also name each circle after its center, so circle C is the green circle. Distance between C and side c is h.

From the picture we can also figure out the radius of each circle is being named and that a is tangent to circles B and C (and b is tangent to circles A and C). Finally, h divides c in the segments c1 and c2 and creates two right triangles. With those parameters defined, we need to start defining a few equations.

First we restate the relationship between the sides a, b, and c and known quantities:

Well, we really do not know c1 or c2, but can find them by applying Pythagoras on the left triangle:

Since we have c2, we can find c1. So, now we can locate the circle C. Next time we will put all these findings in openscad and then do some doodling.