Configuring an interface in Linux without(!) nmcli

In ages past, when you wanted to configure a network interface eth0 with MAC CC:00:FF:EE:12:34 with static IP 128.227.3.20 and using uranus (128.227.3.1) as the gateway in Red Hat Linux or its derivatives, you could have something like this

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << 'EOF'
DEVICE="eth0"
BOOTPROTO="static"
HWADDR="CC:00:FF:EE:12:34"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
DHCP_HOSTNAME=vmhost
IPADDR=128.227.3.20
NETMASK=255.255.255.0
GATEWAY=128.227.3.1
EOF

in your notes and cut-n-paste it as needed. The same would work for a vlan trunk (think 802.1q), where the interface using 128.227.3.20 is now associated to tagged vlan 3; all we needed is 3 files.

• Define the base interface (we could have used uuid instead of MAC address)

  cat /etc/sysconfig/network-scripts/ifcfg-eno1 << 'EOF'
  TYPE=Ethernet
  NAME="eno1"
  DEVICE="eno1"
  HWADDR="CC:00:FF:EE:12:34"
  BOOTPROTO="none"
  ONBOOT=yes

• Define the tagged vlan 3, and which bridge it is associated with

  cat /etc/sysconfig/network-scripts/ifcfg-eno1.3 << 'EOF'
  DEVICE="eno1.3"
  BOOTPROTO="none"
  NM_CONTROLLED="no"
  ONBOOT="yes"
  VLAN=yes
  BRIDGE=dmzbr
  EOF

• Define the bridge with static IP

  cat /etc/sysconfig/network-scripts/ifcfg-dmzbr << 'EOF'
  DEVICE=dmzbr
  TYPE=Bridge
  BOOTPROTO="static"
  NM_CONTROLLED="no"
  ONBOOT="yes"
  TYPE="Ethernet"
  DHCP_HOSTNAME=vmhost
  IPADDR=128.227.3.20
  NETMASK=255.255.255.0
  GATEWAY=128.227.3.1
  EOF

We could get away with two but the 3rd one is there because I like to use bridges. The awake reader will have noticed I switched from eth0 to eno1; I will leave that to the reader but the point here is the above applies to whatever networking naming convention you have to deal with.

from Imgflip Meme Generator

As some (those who do not fall asleep reading these posts) know, I like automation and one of my tools of choice is ansible. Creating the above files for a given host from network declarations in its host_vars/host is very convenient using ansible.

But, there is Network Manager. Until recently -- CentOS 8, Rocky 8, Alma 8 -- you could still tell Network Manager to leave these interfaces alone (the NM_CONTROLLED entry), but now I am building a rocky 9 host, I am being forced to use nmcli instead. And what would it take to nmcli all of that? Let's find out (I think I missed a step here, so don't trust this):

nmcli con add ifname dmzbr type bridge con-name dmzbr
nmcli con modify dmzbr ipv6.method disabled
nmcli connection modify dmzbr ipv4.address 192.168.3.20/24
nmcli connection modify dmzbr ipv4.gateway 192.168.3.1
nmcli connection modify dmzbr ipv4.dns 192.168.3.1
nmcli connection add type vlan con-name eno1.3 ifname eno1.3 dev eno1 id 3
nmcli connection modify eno1.3 master dmzbr slave-type bridge
nmcli connection up eno1.3
nmcli connection up dmzbr

And this should end up with something like this

[root@testbox ~]# nmcli con show
NAME       UUID                                  TYPE      DEVICE
DMZ        6a97eddf-ac72-45d9-ba29-d12c7d59b511  vlan      eno1.3
dmzBridge  ccffabb8-0c8a-47d8-a2d6-15cab0e9b53b  bridge    dmzbr
eno1       9b1b155f-8197-3a7a-a9cc-79cab8b92da1  ethernet  eno1
lo         2dd22561-0fd3-436a-9da0-a53c61d63848  loopback  lo
[root@testbox ~]#

Before you get excited, that UUID is not set in stone. If you are going to do this in Ansible, take a look at the official docs on the nmcli_module. Short version is you are doing all those nmcli commands I did before, which I know I need to check since I know I missed something (I changed the bridge name later). And that is the proper official way to do the deed.

And then there is you

Yep, there is me. I know I will make a mistake. So, let's take a look on this network manager thing. No matter what, the configuration of all of these network interfaces have to go somewhere? If I am unlucky, some kind of binary-only database like Microsoft. If I am lucky, a text file. Well, it turned out my luck still holds: the files are hidden in /etc/NetworkManager/system-connections/:

[root@testbox ~]# ls /etc/NetworkManager/system-connections/
dmzbr.nmconnection  eno1.nmconnection   DMZ.nmconnection
[root@testbox ~]# 

Let's take a look at eno1:

[root@testbox ~]# cat /etc/NetworkManager/system-connections/eno1.nmconnection 
[connection]
id=eno1
uuid=9b1b155f-8197-3a7a-a9cc-79cab8b92da1
type=ethernet
interface-name=eno1
timestamp=1711852227

[ethernet]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=eui64
method=disabled

[proxy]
[root@testbox ~]# 

admit it, it may have a different, more grandiose format than /etc/sysconfig/network-scripts/ifcfg-eno1 but it describes the same thing. What would it take to make my own files? Long story short. not much:

[root@testbox ~]# more /etc/NetworkManager/system-connections/dmzbr.nmconnection /etc/NetworkManager/system-connections/DMZ.nmconnection 
::::::::::::::
/etc/NetworkManager/system-connections/dmzbr.nmconnection
::::::::::::::
[connection]
id=dmzBridge
type=bridge
interface-name=dmzbr

[ethernet]

[bridge]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=default
method=disabled

[proxy]
::::::::::::::
/etc/NetworkManager/system-connections/DMZ.nmconnection
::::::::::::::
[connection]
id=DMZ
type=vlan
interface-name=eno1.3
master=dmzbr
slave-type=bridge

[ethernet]

[vlan]
flags=1
id=3
parent=eno1

[bridge-port]
[root@testbox ~]# 

Note I did not even bother to declare the timestamp or uuid; the later can be created on the fly as the interface comes online. What does that mean? I can keep a copy of these files in a safe place, In fact, I have the following file (that is the filename, I swear)

[root@testbox ~]# cat NICs/eno1-oh_shit.nmconnection 
[connection]
id=eno1
uuid=9b1b155f-8197-3a7a-a9cc-79cab8b92da1
type=ethernet
autoconnect-priority=-999
interface-name=eno1

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]
[root@testbox ~]# 

The idea of this file is if I screw the network up and need to get it back quickly, I copy that to /etc/NetworkManager/system-connections/eno1.nmconnection, make sure it is connected to an untagged vlan switchport, and then restart network manager. That will then get a dhcp IP address and off it goes.

What about Ansible?

The same way I created /etc/sysconfig/network-scripts/ifcfg-eno1.3 I can create these files; the template looks a bit different but everything else is similar. Heretic? Surely, but that is how I roll.

Configuring autofs in a generic way on redhat/debian derived distros in ansible

Let's say you want to install and configure autofs so to have network fileshares mounted on demand. That sounds like a good task for Ansible.

Installing AutoFS

The installing part as you know is fairly easy. In both RedHat and Debian derivatives the package name is autofs. Since we are not doing anything special we can create a rather generic Ansible task using the Ansible package module. What that means is instead of having to worry about having a version for, say, Ubuntu and RedHat, the package module uses whatever package manager is the default for the distro in question:

- name: setup autofs
  package:
    name: autofs
    state: latest

NOTE: I know package works in the redhat/debian derived distros, but I am not sure if it will work in other Linux flavours. I also would check the autofs package name for these other distros.

Configuring AutoFS

Next thing we want to do is ensure we are using the right NFS version. That is done finding the line beginning with mount_nfs_default_protocol and editing its value. In my case I want to make sure it is using NFS v4, which is the default anyway in a modern autofs package. So, why bother? Well, call me paranoid: I want to have exactly what I want. Or call it a simple example of using the Ansible lineinfile module. Or maybe someone is using NFS v3 and want to see how to change it.

- name: Ensure nfs v4
  lineinfile:
    path: /etc/autofs.conf
    regexp: '^mount_nfs_default_protocol '
    line: 'mount_nfs_default_protocol = 4'

Let's use this as an excuse to talk about lineinfile: the regexp here is looking for a line that starts with the string 'mount_nfs_default_protocol '; I wrote it in quotes because it includes the blank space after mount_nfs_default_protocol. Note that the search pattern also means "and anything else to the end of the line," so a line looking like this

mount_nfs_default_protocol could be something you do not want to touch

is fair game. I know usually in regexp you would end the query statement with (.*)$ to include everything to the end of the page, but just nod a lot and move on. Now if the looked like this:

#mount_nfs_default_protocol could be something you do not want to touch

the regexp would not work because it expects the line to begin with m. line defines what we want the line to look like. If it matches that, no changes made.

The next step is to define what we want to use autofs for. In my case, I want to mount user home directories off the fileserver. That means creating a /etc/auto.home file which describes which fileserver we are using. For this I suggest using the template module since we can define the name of the fileserver somewhere earlier in the playbook or in a config file (I am thinking here of a file in host_vars/ or group_vars/ associated with the host in question.). In my task file I use something like

- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs

which

1. Grabs the template templates/auto.home.j2 and puts it in /etc/auto.home
   
2. Sets the permission, ownership, and selinux parameters for /etc/auto.home. The _default means that if there is a default selinux setting for that file/directory we will use it.
   

And roles/common/templates/auto.home.j2 looks like this:

/etc/auto.home
#
# File: /etc/auto.home
#
*   -fstype=nfs4,hard,intr,rsize=8192,wsize=8192 {{ nfs_server }}:/home/&

where nfs_server is the name of the nfs server defined somewhere else.

Now, there are two ways to deal with it, the old autofs and the new one.

• Old autofs: In the old days, you would edit the /etc/auto.master file, adding a line underneath +auto.master that would tell us how to mount user home directory. In the following example,
  

  +auto.master
  /home   /etc/auto.home --timeout=300
  

  the bottom line is saying "if you notice someone trying to access a file/directory in /home, go to /etc/auto.home to see how to mount it. But, if there is no activity after 300 seconds, unmount that." I use timeout of 300 seconds; change it to fit your needs. Now we need to add that to /etc/auto.master, which we will do using the lineinfile module once more:

  - name: Enable auto.home in auto.master
    lineinfile:
      path: /etc/auto.master
      regexp: '^\/home'
      insertafter: '^\+auto.master'
      line: /home   /etc/auto.home --timeout=300

  As you can see, it is a little more complex than the previous task:
  

  • The regexp statement has to escape the /
    
  • insertafter is used to look for a line which matches that pattern and then insert/change the line we want below this pattern. This is useful when you have more than one line matching the regexp pattern or you want the line we are creating/replacing to be on a specific location. You see, without that if line does not exist, it is appended on the end of the file.
    
• New autofs: The more modern /etc/auto.master file has the following lines in it:

  #
  # Include /etc/auto.master.d/*.autofs
  # The included files must conform to the format of this file.
  #
  +dir:/etc/auto.master.d
  #

  Instead of editing the /etc/auto.master file, which might be overwritten by an upgrade, we simply throw a file inside the /etc/auto.master.d directory which is then loaded into the /etc/auto.master file. This file, say /etc/auto.master.d/home.autofs, looks very much like what we did in the old autofs example, main difference is that it is a file

  raub@desktop:~/dev/ansible$ cat roles/common/files/home.autofs
  /home   /etc/auto.home --timeout=300
  raub@desktop:~/dev/ansible$

  that needs to be uploaded using the Ansible file copy module:

  - name: Enable auto.home in auto.master.d
    copy:
      src: home.autofs
      dest: /etc/auto.master.d/home.autofs
      owner: root
      group: root
      serole: _default
      setype: _default
      seuser: _default
      mode: 0644

Which one to pick? You know your setup, so pick the one that fits your needs.

(Re)Starting autofs

The final step we do need to do is (re)start autofs service after all this configuring. We do that using handlers:

- name: start autofs
  service:
    name: autofs
    state: started
    enabled: yes

- name: restart autofs
  service:
    name: autofs
    state: restarted
    enabled: yes

The way I use them is to start autofs when package is installed (see the notify statement),

- name: setup autofs
  package:
    name: autofs
    state: latest
  notify: start autofs

and then restart it after finishing with auto.home

- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs

After I unleash ansible, I then ssh as the non-root user which is allowed to login to vmhost2 and then see if fileshare was mounted:

[raub@vmhost2 ~]$ df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                               16G     0   16G   0% /dev
tmpfs                                  16G     0   16G   0% /dev/shm
tmpfs                                  16G  8.9M   16G   1% /run
tmpfs                                  16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/vmhost-root               2.0G   71M  2.0G   4% /
/dev/mapper/vmhost-usr                4.0G  1.8G  2.2G  46% /usr
/dev/sda2                             976M  179M  731M  20% /boot
/dev/sda1                             200M  6.8M  194M   4% /boot/efi
/dev/mapper/vmhost-var                4.0G  376M  3.7G  10% /var
/dev/mapper/vmhost-vg_backup           10G  104M  9.9G   2% /var/lib/libvirt/qemu/save
fileserver.example.com:/home/raub     690G  629G   61G  92% /home/raub
tmpfs                                 3.2G     0  3.2G   0% /run/user/1001
[raub@vmhost2 ~]$
[raub@vmhost2 ~]$ systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor pres>
   Active: active (running) since Mon 2019-12-09 14:17:26 EST; 2min 39s ago
 Main PID: 28387 (automount)
    Tasks: 6 (limit: 26213)
   Memory: 3.3M
   CGroup: /system.slice/autofs.service
           └─28387 /usr/sbin/automount --foreground --dont-check-daemon
[raub@vmhost2 ~]$

I do not know about you but it seems we have a winner. I will put a cleaner version of this playbook and supporting files in my github account later on.

Capturing the output of a command sent through ssh within script without it asking to verify host key

You have noticed when you connect to a new server, if its key is not in ~/.ssh/known_hosts ssh will ask you to verify the key:

raub@desktop:~$ ssh -i $SSHKEY_FILE cc@"$server_ip"
The authenticity of host 'headless.example.com (10.0.1.160)' can't be established.
ECDSA key fingerprint is SHA256:AgwYevnTsG2m9hQLu/ROp+Rjj5At2HU0HVoGZ+5Ug58.
Are you sure you want to continue connecting (yes/no)? 

That is a bit of a drag if you want to run a script to connect to said server. Fortunately ssh has an option just for that, StrictHostKeyChecking, as mentioned in the man page:

ssh automatically maintains and checks a database containing identification for all hosts it has ever been used with. Host keys are stored in ~/.ssh/known_hosts in the user's home directory. Additionally, the file /etc/ssh/ssh_known_hosts is automatically checked for known hosts. Any new hosts are automatically added to the user's file. If a host's identification ever changes, ssh warns about this and disables password authentication to prevent server spoofing or man-in-the-middle attacks, which could otherwise be used to circumvent the encryption. The StrictHostKeyChecking option can be used to control logins to machines whose host key is not known or has changed.

Now we can rewrite the ssh command as

ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip"

and it will login without waiting for us to verify the key. Of course this should only be done when the balance between security and automation meets your comfort level. However, if we run the above from a script, it will connect and just stay there with the session open, not accept any commands. i.e. if we try to send a command, pwd in this case,

ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip"
pwd

it will never see the pwd. Only way to get to the pwd is to kill the ssh session, when it will then go to the next command but running it in the local host, not the remote one. If we want to run pwd in the host we ssh into, we need to pass it as an argument, i.e.

raub@desktop:/tmp$ pwd
/tmpraub@desktop:/tmp$ ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip" pwd
/home/raub
raub@desktop:/tmp$

which means it will connect, run said command, and then exit. But, how to do that from a script? The answer is to use eval or bash -c (or other available shell):

raub@desktop:/tmp$ moo="ssh -o \"StrictHostKeyChecking no\" -i $SSHKEY_FILE cc@$server_ip"
raub@desktop:/tmp$ dirlist=$(eval "$moo pwd")
raub@desktop:/tmp$ echo $dirlist
/home/raub
raub@desktop:/tmp$ dirlist2=$(sh -c "$moo pwd")
raub@desktop:/tmp$ echo $dirlist2
/home/raub
raub@desktop:/tmp$

Now, there are subtle differences between eval and bash -c. There is a good thread in stackexchange which explains them better than I can do.

Now why someone would want to do that?

How about doing some poor-man ansible, i.e. when host on the other side does not have python or has some other idiosyncrasy. Like my home switch, but that is another story...

Downloading a single file from github (using ansible perhaps)

I was going to setup Ansible to talk to one of my Windows servers. According to the Ansible page on Windows, the easiest way to install the windows side is to download the powershell script ConfigureRemotingForAnsible.ps1 and blindly run it on the windows server. It supposedly does all the magic to install and set things up. Should I run it? I don't know, but let's try to get it. We can inspect its code once we have the file.

NOTE: Setting up Ansible on windows is not the topic of this thread. All I want to show is a way to download a single file from a github repo.

Since I do not know how to do it, let's do some searching. And then we try each method out and see what's what.

Attempt I

There is a thread in stackoverflow called How to pull a single file from a server repository in Git? which
suggested using the git clone command as in

git clone https://github.com/igniterealtime/Openfire.git \
Openfire/src/java/org/apache/mina/management/MINAStatCollector.java

Let's try it out:

raub@desktop:/tmp/rmoo$ git clone https://github.com/igniterealtime/Openfire.git \
Openfire/src/java/org/apache/mina/management/MINAStatCollector.java
Cloning into 'Openfire/src/java/org/apache/mina/management/MINAStatCollector.java'...
remote: Counting objects: 107450, done.
remote: Compressing objects: 100% (53/53), done.
Receiving objects:  14% (15868/107450), 61.11 MiB | 209.00 KiB/s
[...]
remote: Total 107450 (delta 32), reused 31 (delta 16), pack-reused 107380
Receiving objects: 100% (107450/107450), 802.60 MiB | 8.23 MiB/s, done.
Resolving deltas: 100% (63893/63893), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo$ ls Openfire/src/java/org/apache/mina/management/MINAStatCollector.java
build/          i18n/     webadmin/     LICENSE.txt  README.md
dbutil/         src/      webadmintld/  Makefile
documentation/  starter/  xmppserver/   pom.xml
raub@desktop:/tmp/rmoo$

Er, does that look like it grabbed the right thing? For some reason I thought the .java file was, well, a file and not a bunch of files and directories. At least I could swear writing .java files in vi so they were text files. Maybe I am wrong, so let's see if we can get the file I really want:

raub@desktop:/tmp/rmoo$ git clone https://github.com/ansible/ansible.git ansible
/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps1
Cloning into 'ansible/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps
1'...
remote: Counting objects: 236787, done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 236787 (delta 33), reused 25 (delta 6), pack-reused 236712
Receiving objects: 100% (236787/236787), 73.53 MiB | 8.23 MiB/s, done.
Resolving deltas: 100% (152234/152234), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo$ ls
ansible/
raub@desktop:/tmp/rmoo$ ls ansible/blob/devel/examples/scripts/ConfigureRemotin$ForAnsible.ps1/
ansible-core-sitemap.xml  .gitattributes            RELEASES.txt
bin/                      .github/                  requirements.txt
CHANGELOG.md              .gitignore                ROADMAP.rst
CODING_GUIDELINES.md      .gitmodules               setup.py
contrib/                  hacking/                  shippable.yml
CONTRIBUTING.md           lib/                      test/
COPYING                   .mailmap                  ticket_stubs/
.coveragerc               Makefile                  tox.ini
docs/                     MANIFEST.in               VERSION
docsite_requirements.txt  MODULE_GUIDELINES.md      .yamllint
examples/                 packaging/
.git/                     README.md
raub@desktop:/tmp/rmoo$ ls ansible/blob/devel/examples/scripts/ConfigureRemotin$ForAnsible.ps1/
bin/        test/                     docsite_requirements.txt  ROADMAP.rst
contrib/    ticket_stubs/             Makefile                  setup.py
docs/       ansible-core-sitemap.xml  MANIFEST.in               shippable.yml
examples/   CHANGELOG.md              MODULE_GUIDELINES.md      tox.ini
hacking/    CODING_GUIDELINES.md      README.md                 VERSION
lib/        CONTRIBUTING.md           RELEASES.txt
packaging/  COPYING                   requirements.txt
raub@desktop:/tmp/rmoo$

I do not know about you, but that not look like what I really wanted: a single file. On a second thought, that sure looks like the root for the ansible git repo:

I don't know about you but the files and directories sure look familiar.

I guess it is time to try something else.

Attempt II

Let's try something else: in StackOverflow there is a thread called Retrieve a single file from a repository, which suggests

git clone --no-checkout --depth 1 git@github.com:foo/bar.git && cd bar && git show HEAD:path/to/file.txt

For this attempt, we will try to get the file ConfigureRemotingForAnsible.ps1 I want:

git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && cd ansible && \
git show HEAD:examples/scripts/ConfigureRemotingForAnsible.ps1

Thing is that will just spit the file to the screen, literally:

raub@desktop:/tmp/rmoo$ git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && \
cd ansible && git show HEAD:examples/scripts/ConfigureRemotingForAnsible.ps1
Cloning into 'ansible'...
remote: Counting objects: 5873, done.
remote: Compressing objects: 100% (4282/4282), done.
remote: Total 5873 (delta 962), reused 3652 (delta 660), pack-reused 0
Receiving objects: 100% (5873/5873), 7.13 MiB | 8.09 MiB/s, done.
Resolving deltas: 100% (962/962), done.
Checking connectivity... done.
#Requires -Version 3.0

# Configure a Windows host for remote management with Ansible
# -----------------------------------------------------------
#
# This script checks the current WinRM (PS Remoting) configuration and makes
# the necessary changes to allow Ansible to connect, authenticate and
# execute PowerShell commands.
#
[...]

We could improve that by saving the file into a file. But how? The quickest solution I can think of is to pipe it to a file:

file="ConfigureRemotingForAnsible.ps1" ; git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && cd ansible && $(git show HEAD:examples/scripts/$file > $file)

will put $file inside the ansible dir:

raub@desktop:/tmp/rmoo/ansible$ ls
ConfigureRemotingForAnsible.ps1
raub@desktop:/tmp/rmoo/ansible$

Of course we can do better, like placing it on the original pwd and delete the (now temporary) ansible dir. Something like

file="ConfigureRemotingForAnsible.ps1" ; git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && cd ansible && $(git show HEAD:examples/scripts/$file > ../$file) && cd .. && rm -rf ansible

should do just fine. But you do not have to believe on me; here's it in action:

raub@desktop:/tmp/rmoo$ file="ConfigureRemotingForAnsible.ps1" ; \
git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && \
cd ansible && $(git show HEAD:examples/scripts/$file > ../$file) && cd .. && rm -rf ansible
Cloning into 'ansible'...
remote: Counting objects: 5873, done.
remote: Compressing objects: 100% (4282/4282), done.
remote: Total 5873 (delta 962), reused 3652 (delta 660), pack-reused 0
Receiving objects: 100% (5873/5873), 7.13 MiB | 758.00 KiB/s, done.
Resolving deltas: 100% (962/962), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo$ ls
ConfigureRemotingForAnsible.ps1
raub@desktop:/tmp/rmoo$

Just to show this is not an accident, let's validate it by applying it to get the java source file we tried to get earlier.

raub@desktop:/tmp/rmoo$ file="MINAStatCollector.java" ; git clone --no-checkout --depth 1 https://github.com/igniterealtime/Openfire.git  && cd Openfire && $(git show HEAD:src/java/org/apache/mina/management/$file > $file)
Cloning into 'Openfire'...
remote: Counting objects: 5291, done.
remote: Compressing objects: 100% (4433/4433), done.
remote: Total 5291 (delta 807), reused 3394 (delta 479), pack-reused 0
Receiving objects: 100% (5291/5291), 92.80 MiB | 8.24 MiB/s, done.
Resolving deltas: 100% (807/807), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo/Openfire/Openfire$ head MINAStatCollector.java
package org.apache.mina.management;

import static org.jivesoftware.openfire.spi.ConnectionManagerImpl.EXECUTOR_FILTER_NAME;

import org.apache.mina.core.service.IoService;
import org.apache.mina.core.service.IoServiceListener;
import org.apache.mina.core.session.IdleStatus;
import org.apache.mina.core.session.IoSession;
import org.apache.mina.filter.executor.ExecutorFilter;
import org.apache.mina.filter.executor.OrderedThreadPoolExecutor;
raub@desktop:/tmp/rmoo/Openfire$

I think we can improve it by making it more generic, which might be the subject of another post. Or something; I've been dragging on finishing this article for a few weeks now so I just want it gone.

Now what about something completely Ansible-ly?

Er, this Article is getting way longer than I originally planned. I will put the ansible side on another. How about that for raising your expectations and then expertly crushing them?

Relative paths and roles in Ansible

Here is something I finally understood in Ansible that has been annoying the hell out of me for a while: relative paths. The idea of relative paths is so if you have to tell Ansible to grab a file -- like some config file or even an image or certificate -- and put it somehwere else you do not have to give the entire source path.

Imagine if you wrote your playbook in your Linux desktop which has a task that grabs the file /home/bob/bobs-development-stuff/server.stuff/ansible/roles/webservers/files/main-logo.png and put it somewhere in the target machine. That is a mouthfull of a path. But then since you are clever you have this playbook in an internal git repository and then check it out in your Mac laptop to do some work. Thing is the path now is /Users/bobremote/ansible/roles/webservers/files/main-logo.png and now your task ain't working no more. And that is the short version for why absolute paths in your ansible box are not a great idea.

So we use relative paths instead and the main question of the day is what are those paths relative to?

Do you remember when we talked about creating and uploading ssh key pairs? Well, there was a bit of a fudge right in the key: line

- name: Copy the public key to the user's authorized_keys
    # It also creates the dir as needed
    authorized_key:
      user: "{{user_name}}"
      key: '{{ lookup("file", "roles/files/" ~ user_name ~ ".ssh.pub") }}'
      state: present

On a second thought, it is really not a fudge but it hints on the answer to our question.All we are asking is for the task to grab a file inside the roles/files directory. The original layout for this playbook looks like this

ansible
 hosts
 ansible.cfg
 site.yml
 roles
  defaults
  files
   moose.ssh
   moose.ssh.pub
  handlers 
   main.yml
  meta
  tasks 
   main.yml
  template
  vars

So we really only have just one playbook inside the ansible directory. When we write in the roles/tasks/main.yml tasklist file to grab the .ssh.pub< file, it knows exactly how to get it. With time, we became clever and would like to plan ahead so decided to break the roles into the ones common to every server and the ones that build up on the common one and then do the specialization stuff, say website or database server. So, what we really want is to have a layout like this (inside the same old ansible dir)

site.yml
web.yml
group_vars
roles
 common
   defaults
   files
      epel.repo
      iptables-save
      moose.ssh
      moose.ssh.pub
   handlers 
      main.yml
   meta
   tasks 
     main.yml
   template
   vars
 website
   defaults
   files
   handlers
   meta
   tasks
     main.yml
   template
   vars
 db
   defaults
   files
   handlers
   meta
   tasks
     main.yml
   template
   vars

The playbook that copies the ssh key is in the common role, specifically roles/common/tasks/main.yml (yes I could break it out some more but let's not confuse things just yet and hurt my brain). And we expect if we write our function as

- name: Copy the public key to the user's authorized_keys
    # It also creates the dir as needed
    authorized_key:
      user: "{{user_name}}"
      key: '{{ lookup("file", "./files/" ~ user_name ~ ".ssh.pub") }}'
      state: present

our playbook to create roles/common/files/moose.ssh.pub and then upload to whatever machine we are building. Instead it gives us the following friendly message:

fatal: [ansibletest -> localhost]: FAILED! => {"changed": true, "cmd": "ssh-
keygen -f ./files/\"moose\".ssh -t rsa -b 4096 -N \"\" -C \"moose\"", "delta": "0:
00:01.056746", "end": "2016-11-30 15:55:19.618226", "failed": true, "rc": 1, "s
tart": "2016-11-30 15:55:18.561480", "stderr": "Saving key \"./files/moose.ssh\
" failed: No such file or directory", "stdout": "Generating public/private rsa
 key pair.", "stdout_lines": ["Generating public/private rsa key pair."], "warni
ngs": []}

Clearly we have a bit of a path problem. Don't believe me? Let's ask where the playbook thinks it is running from

- name: Find out where I am locally speaking
    local_action: shell pwd > /tmp/here

All we are doing here is getting the path this is being executed from and saving in a file so we can see it. And see it we shall:

raub@desktop:~/dev/ansible$ cat /tmp/here
/home/raub/dev/ansible
raub@desktop:~/dev/ansible$ 

My guess is that if we use relative paths, they are all from where we ran the playbook from, /home/raub/dev/ansible in my case. Which really sucks! Now, there is an environmental variable in Ansible called role_path, which I found being mentioned in the docs which should show the path to the role we are currently running. Let's try it out

- name: Find out where I am locally speaking
    debug: msg="Role path is supposed to be {{ role_path }} "

which gives

TASK [website : Find out where I am locally speaking] ******************************
ok: [ansibletest] => {
    "msg": "Role path is supposed to be /home/raub/dev/ansible/roles/website "
}

So, the solution is to rewrite our little function as

- name: Copy the public key to the user's authorized_keys
    # It also creates the dir as needed
    authorized_key:
      user: "{{user_name}}"
      key: '{{ lookup("file", "{{ role_path }}/files/" ~ user_name ~ ".ssh.pub") }}'
      state: present

so the path becomes /home/raub/dev/ansible/roles/website/files/moose.ssh.pub

Creating a user with random password using Ansible

So I need to create an user in a machine so I can then have a script that will log into this machine and backup its database. Which database is that, you might ask? For this discussion it does not matter besides that it is running in a Linux box. But, if you want a more specific example, we could be backing up a sqlite database since we talked about how to do the deed before.

Anyway, the plan is to have the script ssh into the database server and grab the backup. Since we are using ssh in the script, we might as well use key pair authentication. Now I have learned that by default if you create a user and do not assign it a password, you will not be able to login as said user using key pair authentication. You can turn that off but I would rather not. Instead, since I am creating said user programmatically, I can give it a long random password.

Now that we have a plan, let's see how to do it in an ansible playbook. I will present only the relevant bits since I do not know how you do your playbooks. So, we could create a user using something like

- name: Create user_name
    user:
      name={{user_name}}
      groups={{user_groups}}
      shell=/bin/bash
      state=present
      append=yes

which would create user user_name who will also belong to the groups defined in user_groups, where user_name and user_groups are variables defined somewhere earlier in the show. And this would create a user without a password, which would do us no good. Nor would us make the playbook stop and ask us to enter a password. We said earlier we are going to create a random password, so let's see if we can make something random enough for our needs.

I plan on generating this random password in the machine we are running ansible on, not the target machine. One of the reasons is that I want to use the Linux command mkpassword to create the password hash (note it is being called using the shell command. So, I will use a local_action to do the deed. For instance, let's say I want the password to be pickles and encoded using SHA-512 hash (mild encryption). I could accomplish it by writing

- name: generate random password for new user
    local_action: shell  mkpasswd --method=SHA-512 pickles
    register: user_pw

This would create a hash, say

$6$rIcep9bGJTUpE$s8pka1dX6gWfyeNfi8YLaqrg/85tgtpJv809AUmO2jHhMQbSUnuNSloJSa6EmOQS02Ek4mvpiIu2DAvA9W0UL0

and assign it to the variable user_pw. This of course has to be done before the user is created. To use it with our new user, we can then modify our little user creation function to something like this:

- name: Create user_name
    user:
      name={{user_name}}
      groups={{user_groups}}
      shell=/bin/bash
      state=present
      append=yes
      update_password=on_create
      password="{{user_pw.stdout}}"

In the last line we are feeding the value of user_pw, user_pw.stdout, to password. But why can't I just feed user_pw? Here's an exercise to you: tell your playbook just print user_pw. Doesn't it look very object-like?

If you run your playbook and all went well, go to the target machine and check if there is a password associated with the user in /etc/shadow. If the user was already created, you will need to delete user and let ansible recreate it.

So we have so far created a way to create a password hash and then create a new user with that password. The last step we need is to make the password random. Here is what I am proposing: how about if we use date since epoch in seconds as our password and then mangle it a bit? Here is a simple mangling example:

raub@desktop:~$ date +%s
1479095572
raub@desktop:~$ date +%s | sha384sum
3d47137f4cd6bfb638deecc661b4e0ae9545ab2454b20eac51b6992807b3f518a49be9ef3b72a5abdc9167e32acc2473  -
raub@desktop:~$ date +%s | sha384sum | md5sum
65db8ecf178a505ea17e9b53cc5adf31  -
raub@desktop:~$ date +%s | sha384sum | md5sum | cut -f 1 -d ' '
b8a49ccaa4721877cf39e510c7ac3622
raub@desktop:~$

Which gives b8a49ccaa4721877cf39e510c7ac3622 as the output, which should be long enough to fulfill our needs. Of course if you run it again, it will spit out a different result, which is what we want? Perfectly random? Not by a long shot, but it is long enough for our needs. Remember: there is nothing saying you have to use the above. Hav efun creating your own function!

So, let's apply that to our little password generating function:

- name: generate random password for new user
    local_action: shell  mkpasswd --method=SHA-512 $(date +%s | sha384sum | md5sum | cut -f 1 -d ' ')
    register: user_pw

And we should be good to go. Here is how the final version should look like in a playbook:

- name: generate random password for new user
    local_action: shell  mkpasswd --method=SHA-512 $(date +%s | sha384sum | md5sum | cut -f 1 -d ' ')
    register: user_pw
# Something might happen here before we create user
- name: Create user_name
    user:
      name={{user_name}}
      groups={{user_groups}}
      shell=/bin/bash
      state=present
      append=yes
      update_password=on_create
      password="{{user_pw.stdout}}"

Now we have an user, we can then create the ssh key pair we talked about in an earlier article. Of course we might edit the ./ssh/authorized_keys file to restrict what that key can do.

Creating and uploading ssh keypair using ansible

As some of you have noticed from previous posts, I do use docker. However, I also use Ansible to install packages on and configure hosts. I am not going to talk about when to use each one. Instead, I will try to make another (hopefully) quick post, this time ansible-related.

Let's say I have an user abd I want to set said user up so I can later on ssh to the target machine as that user using ssh key pair. From the machine I am running ansible from. To do so I need to create a ssh keypair and then put the public key to the user's authorized_keys file. And that might require creating said file and ~/.ssh if not available. Now that is a mouthfull so let's do it in steps.

1. Decide where to put the key pair: This is rather important since we not only need to have it somewhere we can find when we aare ready to copy it to the target server but also we might want not to recreate the key pair every time we run this task.

   For now I will put it in roles/files/, which is not ideal for many reasons. Just to name one, we might want to put it somewhere encrypted or with minimum access by suspicious individuals. But for this discussion it is a great starting place for the key pair. So, let's see how I would verify if the pair already exists or not:

   - name: Check if user_name will need a ssh keypair
       local_action: stat path="roles/files/{{user_name}}.ssh"
       register: sshkeypath
       ignore_errors: True

   The first line (name) writes something to the screen to show where we are in the list of steps. It is rather useful because when we run ansible we then can see which task we are doing. The local_action uses stat to determine if the file roles/files/{{user_name}}.ssh exists. The return/result code is then written to the variable ssheypath. Finally I do not care about any other messages from stat.
   

2. Create key pair: there are probably really clever ways that are purely ansible native to do it, but I am not that bright so I will use openssh instead, specifically ssh-keygen. Ansible has a way to call shell scripts, the shell module. Now, I want to run tt>ssh-keygen on the machine I am running ansible from, the local machine in ansible parlance. And for that reason we have the local_action command. Here is one way to do the deed:

   - name: Create user_name ssh keypair if non-existent
       local_action: shell ssh-keygen -f roles/files/"{{user_name}}".ssh -t rsa -b 4096 -N "" -C "{{user_name}}"
       when: not sshkeypath.stat.exists

   As you guessed, we are using ssh-keygen to create the key pair roles/files/user_name.ssh and roles/files/user_name.ssh.pub. The other ssh-keygen are just because I would like my keys to be a bit more secure than those created by the average bear. So far so good: we are here using local_action and shell like in the previous step.

   The interesting part if the when statement. What it is saying is to only create the key pair if it does not already exist. And that is why we did check if roles/files/user_name.ssh existed in the previous step. Of course that meant we assumed that if roles/files/user_name.ssh existed, roles/files/user_name.ssh.pub must also exist. There are ways to check for both but I will leave that to you.
   

3. Copy public key to remote server: Now we know we have a ssh key pair, we do need to copy it. You can do it using the file module, which will require a few extra tasks (create the ~/.ssh directory with the proper permissions comes to mind). Or, you can use the appropriately named authorized_key module. I am lazy so I will pick the last option.

   - name: Copy the public key to the user's authorized_keys
       # It also creates the dir as needed
       authorized_key:
         user: "{{user_name}}"
         key: '{{ lookup("file", "roles/files/" ~ user_name ~ ".ssh.pub") }}'
         state: present

   What this does is create ~/.ssh and ~/.ssh/authorized_keys and then copy the key to ~/.ssh/authorized_keys. The most important statement here is the key one. Since it expects us to pass the key as a string, we need to read the contents of the public key. And that is why we called upon the lookup command; think of it here as using cat. And, why we have all the crazy escaping in "roles/files/" ~ user_name ~ ".ssh.pub").
   

And that is pretty much it. If you want to be fancy you could use a few local_actions to copy the private key to your (local) ~/.ssh and then create/edit ~/.ssh/config to make your life easier. I did not do it here because this user will be used as is by ansible; I need no access to this user as me outside ansible.

Remotely controlling ESXi with bash and without Ansible or Windows

I would expect the more knowledgeable ESXi admins amongst you will rightfully mention the proper way to run a remote command in a ESXi box is using the vSphere (Web) Client or the vSphere Command-Line Interface. I agree completely, which is why I will ignore the advice and see what else I can use.

Shattering a Dream

Just to let you know, I really really REally wanted to have used Ansible here. In fact, this was supposed to be a post on how to use Ansible to shut down an ESXi server, saving the running vm clients, and then restore those clients after the server was back up. I spent time laying things out but could not get it working. I will not go into details; but at first I thought my issues were due to ESXi not having python-simplejson, but I learned that assumption was wrong; Python in ESXi has json in it:

I will try again some other time. For now, I must put my pride away, accept I cannot be one of the cool kids running Ansible for everything, and see what I can do with what I have.

Back to the Drawing Board

Let's start by getting away from the computer (figuratively speaking unless you printed this) and think on what the goals of this mess are

1. Connect to the ESXi-based vm host, who henceforth shall be named vmhost2.
   
2. Tell vmhost2 to save all the running vm clients, preserving their state.
   
3. Shut vmhost2 down.
   
4. Connect back to vmhost2 after it is back up and running, or when we feel like.
   
5. Tell vmhost2 to resume all those saved vm clients.
   

We already wrote the script to do the suspending and removing business a while ago, so two of those goals have been taken care of.

My code usually starts as a (Linux) shell script unless that is not appropriate for the task (say I need a Windows-only solution), so let me do some thinking aloud in shell before going fancy.

So, we need to connect to vmhost2 in a somewhat secure way. How about ssh? We can create a key pair using ssh-keygen; I would suggest making it at least 4096 bits long. Now we just need to copy the key to vmhost2. I will break some security principles and say we will connect to vmhost2 as root; adjust this as needed. With that in mind, we put the key in /etc/ssh/keys-root/authorized_keys, which probably behaves like your garden-variety authorized_keys file, i.e. command restrictions like we mentioned in an previous article should work. So, let's test it by connecting to vmhost2 and seeing what junk we have in the root dir:

[raub@desktop ~]$ ssh -C -tt -o IdentityFile=keys/vmhost2 -o User=root vmhost2 ls
altbootbank      lib64            sbin             var
bin              local.tgz        scratch          vmfs
bootbank         locker           store            vmimages
bootpart.gz      mbr              tardisks         vmupgrade
dev              opt              tardisks.noauto
etc              proc             tmp
lib              productLocker    usr
Connection to vmhost2 closed.
[raub@desktop ~]$

I think we can use that. Some of you might have noticed we are being a bit explicit with our options, using -o SomeOption when a shortcut would do, but we should make it easier for us to see what is going on, specially if we plan on coding that. And we shall: here is a Bourne (!) shell version of what we just did

#!/bin/sh

user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

ssh -C -tt -o IdentityFile="$KEY"  -o PasswordAuthentication=no \
    -o PreferredAuthentications=publickey \
    -o ConnectTimeout=10 \
    -o User=$user $host \
    $what

Yes, we told it to run a few more commands while it was connected, but the principle is the same: we can send commands to the ESXi box through ssh. Now, those of you who are well-versed with ssh could not be remotely impressed even if you tried really hard. And I agree; I am just laying down the building blocks. If you have been in this blog before, you would know that is how I roll.

Note: I will be honest with you and say sending out command is great, but we need to get replies for this to really be useful. Later on we will revisit this, for now assume it is all rainbows and unicorns.

I think we should stop messing around and switch the Bash so we can use functions. Maybe something like this perhaps?

#!/bin/bash

send_remote_command ()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="/bin/sh -c 'hostname' && /bin/sh -c 'date' && /bin/sh -c 'uptime'"

send_remote_command $host $user $KEY $what

As others have pointed out, passing parameters to a bash function is not fun. No matter, let's try it out:

[raub@desktop esxisave]$ ./esxitest.sh
~ # whoami

Er, that was not supposed to happen. Did I just connect to the vmhost2?

~ # whoami
/bin/sh: whoami: not found
~ # pwd
/
~ # hostname
vmhost2.example.com.
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

What's going on?

[raub@desktop esxisave]$ bash -x esxisave.sh
+ user=root
+ host=vmhost2
+ KEY=keys/vmhost2
+ what='/bin/sh -c '\''hostname'\'' && /bin/sh -c '\''date'\'' && /bin/sh -c '\''uptime'\'''
+ send_remote_command vmhost2 root keys/vmhost2 /bin/sh -c ''\''hostname'\''' '&&' /bin/sh -c ''\''date'\''' '&&' /bin/sh -c ''\''uptime'\'''
+ local hostname=vmhost2
+ local username=root
+ local keyfile=keys/vmhost2
+ local command=/bin/sh
+ ssh -C -tt -o IdentityFile=keys/vmhost2 -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o ConnectTimeout=10 -o User=root vmhost2 /bin/sh
~ # exit
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Duh! Escaping quotes. I think we are overthinking this.

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"

    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

# Let's try our little function out
user=root
host="vmhost2"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

So we run it again:

[raub@desktop esxisave]$ ./esxitest.sh
vmhost2.example.com.
Sat Sept 26 05:05:00 UTC 2015
 05:05:00 up 04:50:20, load average: 0.01, 0.01, 0.01
Connection to vmhost2 closed.
[raub@desktop esxisave]$

Much better. Now we could add other functions to our script; first thing that comes to mind is making the script able to upload a file. You see, a while ago we wrote a script to suspend and resume vm clients, we might as well find a way to upload it. Since we are using ssh, it would make sense to then ask scp for help. And we shall.

Let's put a file over there

We have established we can remotely execute commands in the ESXi box vmhost2 without adding any extra packages to out desktop. On the last paragraph we promised we would show how to upload our old script tho shutdown vmhost2; we better deliver it, and we shall do it using scp if for no other reason than we hinted we were going to use it. Now, based on what we have already tested using ssh, to upload the script we might do something like

scp -i keys/vmhost2.example.com root@vmhost2:/var/tmp/save_runningvms.sh .

For now we will be using /var/tmp/ since it tends to survive a reboot but it is still a temp directory; I am not ready yet to start cluttering vmhost2's OS drive. We could modify esxitest.sh to include uploading support, say

#!/bin/bash

send_remote_command()
{
    # Reading function parameters in bash is not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local command="$4"
 
    ssh -C -tt -o IdentityFile="$keyfile"  -o PasswordAuthentication=no \
        -o PreferredAuthentications=publickey \
        -o ConnectTimeout=10 \
        -o User=$username $hostname \
        $command
}

send_file()
{
    # Reading function parameters in bash is still not fun
    local hostname="$1"
    local username="$2"
    local keyfile="$3"
    local localfile="$4"
    local remotefile="$5" # a bit of a misowner; it is more like the full path

    scp -i "$keyfile" $localfile $username@$hostname:$remotefile
}

# Let's try our little function out
user=root
host="vmhost2.in.kushana.com"
KEY="keys/$host"
what="hostname && date && uptime"

send_remote_command $host $user $KEY "$what"

# Now we send a file
sourcepath="files/save_runningvms.sh" # Yes, relative path because we can
destinationpath="/var/tmp/save_runningvms.sh" # Absolute path when it makes sense

send_file $host $user $KEY $sourcepath $destinationpath

Let's try it out: First we will check if there is nobody on /var/tmp

[root@vmhost2:~] ls /var/tmp/
sfcb_cache.txt
[root@vmhost2:~]

There is somebody there but nobody relevant to this article. So let's run the new script:

[raub@desktop esxisave]$ ./esxisave.sh 
vmhost2
Fri Jan  1 05:04:47 UTC 2016
  5:04:47 up 64 days, 22:37:34, load average: 0.01, 0.01, 0.01
Connection to vmhost2.in.kushana.com closed.
save_runningvms.sh                            100%  918     0.9KB/s   00:00    
[raub@desktop esxisave]$ 

Did you noticed the very last line? It sure looks very scpish to me. Were all of our efforts in vain or we now have a new file in /var/tmp?

[root@vmhost2:~] ls /var/tmp/
save_runningvms.sh  sfcb_cache.txt
[root@vmhost2:~] 

Of course we could have loaded the file and then remotely set its permissions and executed it, all from our little bash script. That hopefully gives you lots of ideas of where to go next.

That is fine, but why can't I run this script by passing the parameters from command line with options so it knows when I want to send a command or a file? And why doesn't it process the return messages? Do you also want me to give you some warm Ovomaltine and fluff your pillow? Seesh! I am just giving a basic example, a building block to solve whatever task we are talking about in a given article; you can embellish it later. I am not here to hold your hand but (hopefully) set you in the right direction.

Save/Suspend and Resume a VMware ESXi vm client command line style

Here is an interesting project: let's say you have one or more UPS (one per power supply) attached to your ESXi vm host (or hosts; this is completely scalable). Yes, it goes without saying providing uninterrupted power to your servers is a good idea. But, unless you are a large company chances are this power will only last so long. You can make it last even longer by having a plan that will decide in which order your physical servers will be shut down based on load and remaining power. That does mean shutting down your vm servers; for the sake of this discussion, we will assume they are ESXi-based.

I have seen interesting articles on shutting down ESXi hosts on case of power failure, but many assume you are monitoring the UPS through the ESXi host. That might be thinking small; what if that is not the case? What if you have a UPS or two on the bottom feeding the entire cabinet? Chances are you will be monitoring it from a host, be it a vm or not, that is running some monitoring program such as Nagios, that is set to do something in case of a power failure. Of course, if you have a monitoring vm you can talk to your UPS using either ethernet or USB passthrough depending on how sophisticated that model is. And it will decide when to tell our ESXi box it is time to shut down.

I do not know about you but I would like to gracefully save/shutdown the vm clients running in that host before that.

The plan is to have the host monitoring the UPS tell the ESXi host to run a shutdown procedure, which would need to first save the vm guests. And, once the vm server is back up and running, it would resume -- by its own accord or by the order of another server -- the saved vm clients. Yes, you will have to worry about how the monitoring and the ESXi hosts will talk to each other and how the client's clock will catch up, but for this article we will focus on creating a tool that only cares about saving and resuming all of the vm guests running in this ESXi box. We can expand later.

If we want to save the running vm clients, we probably should find out which ones are running. In a previous article we wrote a script to see if a given vm client is running, off, or saved. For the script we will be creating, we want to use something else, vmdumper. Here is what the help screen for the program says.

/tmp # vmdumper -h
vmdumper: [options]  
         -f: ignore vsi version check
         -h: print friendly help message
         -l: print information about running VMs
         -g: log specified text to the vmkernel log
/tmp #

Note the -l shows only the running vms, which is what we want to do. So, let's run that and see what it spits back (I will break them a bit so they will kinda fit the screen):

~ # /sbin/vmdumper -l
wid=264397      pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/boot2docker/boot2docker.vmx" uuid="56 4d 11 2b 63 bc 88 fb-d9 e1 
93 fc 69 36 66 45"  displayName="boot2docker"       vmxCartelID=264396
wid=13080       pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/Windows 2012/Windows 2012.vmx"       uuid="56 4d e7 cb 24 11 63 
13-04 0d 9b 41 08 f9 a3 be"  displayName="Windows 2012"      vmxCartelID=13079
wid=527962      pid=-1  cfgFile="/vmfs/volumes/52a08b50-984b4bf0-219f-d067
e51ce7b7/devcentos/devcentos.vmx"     uuid="56 4d d7 e8 25 6c de 91-09 38 
60 ce ab 5d 43 ca"  displayName="devcentos" vmxCartelID=527961
~ #

As you can see, it shows the path for the config file the vm guest is using (cfgFile, its name (displayName) and something called wid. And a few other things I do not feel like caring about. So, how do we save a vm anyway? We know we can start a vm using vim-cmd vmsvc/power.on, so maybe it sounds similar. Some frustrating searching later we find that http://www.vi-toolkit.com/wiki/index.php/Vmsvc/power.hibernate might be a candidate. Thing is it needs wmid as the argument. I will save some time and state (have faith, brother!) it can be obtained by

vim-cmd vmsvc/getallvms | grep "${displayName}" | awk '{ print "vmid=" $1}'

But, does it really work? We shall try with devcentos, which happens to have wmid=3 (again, I cheat because I have spent loads of time testing this):

/tmp # vim-cmd  vmsvc/power.hibernate 3
(vim.fault.ToolsUnavailable) {
   dynamicType = ,
   faultCause = (vmodl.MethodFault) null,
   msg = "Cannot complete operation because VMware Tools is not running in this virtual machine.",
}
/tmp #

And it does not seem to want to work. It needs VMware Tools, and I do not want to worry about it. So let's see what else we can use. After some looking I found vmdumper. To save devcentos we could do

/tmp # vmdumper 527962 suspend_vm
Suspending VM...
/tmp # 

The weird number 527962 is the world id or wid for devcentos, which happens to be the first column in the output of vmdumper -l associated with that vm client.

Pet Peeve: If you remember the output of vmdumper -h, which should be the help page for that command, mentions nothing about suspend-vm. Good job, VMware! That does make me wonder what else you are not documenting...

Now my venting is done, let's see what we need.

1. We need the wid to shut down with vmdumper
   
2. We can resume (I tested already, and so can you!) the vm client using vim-cmd vmsvc/power.on. Thing is it needs wmid as the argument, which we figure out how to get above.
   
3. We then need a way to save wmid so when we can restore the saved vms. Probably saving the names of the vms would also be a nice touch.
   

So, here is the script I wrote to save and restore the running vms. As you can see, it is rather dumb since it is an all or nothing kinda deal. It is also unforgiving: if you run it again to save vms, the old /var/tmp/save_vms file will be overwritten. For what I wrote this script for, that is but a small annoyance.

cat > save_runningvms.sh  << 'EOF'
#!/bin/sh
IFS=$'\n'
USAGE="Usage: $0 {save|resume}"
SAVE_FILE=/var/tmp/save_vms

if [ "$#" == "0" ]; then
        echo "$USAGE"
        exit 1
fi

selection=$1

case $selection in
   # If we want to save them
   save )
      rm -f ${SAVE_FILE}

      # Find which vms are currently running
      for i in $(vmdumper -l \
         | awk ' BEGIN { FS = "\t" }; { print $1 ";" $5 }')
      do
         eval $i
         # Start saving them
         vmid=$(vim-cmd vmsvc/getallvms | grep "${displayName}" \
            | awk '{ print "vmid=" $1}')
         vmdumper $wid suspend_vm

         # Write list of saved guests in $SAVE_FILE
         echo $vmid ";" $i >> ${SAVE_FILE}
      done
      ;;
   # If we want to restore them
   resume )
      # Get list of saved guests
      for i in $(cat ${SAVE_FILE})
      do
         # Wake them up
         eval $i
         vim-cmd vmsvc/power.on $vmid
      done
      ;;
esac
EOF
chmod +x save_runningvms.sh

You will note that I avoid using Bashisms because the shell in busybox is closer to Bourne than Bash.

I think you probably want to see it running. So, let's run it. First we do some saving

/tmp # ./save_runningvms.sh save
Suspending VM...
Suspending VM...
Suspending VM...
/tmp # 

Did it create the /var/tmp/save_vms file? If so, how does it look like?

/tmp # cat /var/tmp/save_vms
vmid=24 ; wid=5718058;displayName="boot2docker"
vmid=23 ; wid=5714001;displayName="Windows 2012"
vmid=3 ; wid=5715871;displayName="devcentos"
/tmp # 

Ok, I am not convinced. You must be lying. Lemme go to the other vmhost, vmhost, and ping devcentos

[raub@vmhost tmp]# ping devcentos
PING devcentos.example.com (10.0.0.112) 56(84) bytes of data.
From vmhost.example.com (10.0.0.19) icmp_seq=2 Destination Host Unreachable
From vmhost.example.com (10.0.0.19) icmp_seq=3 Destination Host Unreachable
From vmhost.example.com (10.0.0.19) icmp_seq=4 Destination Host Unreachable
^C
--- devcentos.example.com ping statistics ---
7 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6125ms
pipe 3
[raub@vmhost tmp]# 

Hmmmm, okay. But maybe it was off and you were lying to me. So, let's see about waking up the sleeping vms.

/tmp # ./save_runningvms.sh resume
Powering on VM:
Powering on VM:
Powering on VM:
/tmp #

And then pinging devcentos

[raub@vmhost tmp]# ping devcentos
PING devcentos.example.com (10.0.0.112) 56(84) bytes of data.
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=1 ttl=64 time=212 ms
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=2 ttl=64 time=0.316 ms
64 bytes from devcentos.example.com (10.0.0.112): icmp_seq=3 ttl=64 time=0.313 ms
^C
--- devcentos.example.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2078ms
rtt min/avg/max/mdev = 0.313/70.992/212.349/99.954 ms
[raub@vmhost tmp]#

I guess the script does work after all. What's the world coming to?