Monday, December 09, 2019

Configuring autofs in a generic way on redhat/debian derived distros in ansible

Let's say you want to install and configure autofs so to have network fileshares mounted on demand. That sounds like a good task for Ansible.

Installing AutoFS

The installing part as you know is fairly easy. In both RedHat and Debian derivatives the package name is autofs. Since we are not doing anything special we can create a rather generic Ansible task using the Ansible package module. What that means is instead of having to worry about having a version for, say, Ubuntu and RedHat, the package module uses whatever package manager is the default for the distro in question:

- name: setup autofs
  package:
    name: autofs
    state: latest
NOTE: I know package works in the redhat/debian derived distros, but I am not sure if it will work in other Linux flavours. I also would check the autofs package name for these other distros.

Configuring AutoFS

Next thing we want to do is ensure we are using the right NFS version. That is done finding the line beginning with mount_nfs_default_protocol and editing its value. In my case I want to make sure it is using NFS v4, which is the default anyway in a modern autofs package. So, why bother? Well, call me paranoid: I want to have exactly what I want. Or call it a simple example of using the Ansible lineinfile module. Or maybe someone is using NFS v3 and want to see how to change it.

- name: Ensure nfs v4
  lineinfile:
    path: /etc/autofs.conf
    regexp: '^mount_nfs_default_protocol '
    line: 'mount_nfs_default_protocol = 4'

Let's use this as an excuse to talk about lineinfile: the regexp here is looking for a line that starts with the string 'mount_nfs_default_protocol '; I wrote it in quotes because it includes the blank space after mount_nfs_default_protocol. Note that the search pattern also means "and anything else to the end of the line," so a line looking like this

mount_nfs_default_protocol could be something you do not want to touch
is fair game. I know usually in regexp you would end the query statement with (.*)$ to include everything to the end of the page, but just nod a lot and move on. Now if the looked like this:
#mount_nfs_default_protocol could be something you do not want to touch
the regexp would not work because it expects the line to begin with m. line defines what we want the line to look like. If it matches that, no changes made.

The next step is to define what we want to use autofs for. In my case, I want to mount user home directories off the fileserver. That means creating a /etc/auto.home file which describes which fileserver we are using. For this I suggest using the template module since we can define the name of the fileserver somewhere earlier in the playbook or in a config file (I am thinking here of a file in host_vars/ or group_vars/ associated with the host in question.). In my task file I use something like

- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs
which
  1. Grabs the template templates/auto.home.j2 and puts it in /etc/auto.home
  2. Sets the permission, ownership, and selinux parameters for /etc/auto.home. The _default means that if there is a default selinux setting for that file/directory we will use it.

And roles/common/templates/auto.home.j2 looks like this:

/etc/auto.home
#
# File: /etc/auto.home
#
*   -fstype=nfs4,hard,intr,rsize=8192,wsize=8192 {{ nfs_server }}:/home/&
where nfs_server is the name of the nfs server defined somewhere else.

Now, there are two ways to deal with it, the old autofs and the new one.

  • Old autofs: In the old days, you would edit the /etc/auto.master file, adding a line underneath +auto.master that would tell us how to mount user home directory. In the following example,
    +auto.master
    /home   /etc/auto.home --timeout=300
    
    the bottom line is saying "if you notice someone trying to access a file/directory in /home, go to /etc/auto.home to see how to mount it. But, if there is no activity after 300 seconds, unmount that." I use timeout of 300 seconds; change it to fit your needs. Now we need to add that to /etc/auto.master, which we will do using the lineinfile module once more:

    - name: Enable auto.home in auto.master
      lineinfile:
        path: /etc/auto.master
        regexp: '^\/home'
        insertafter: '^\+auto.master'
        line: /home   /etc/auto.home --timeout=300

    As you can see, it is a little more complex than the previous task:

    • The regexp statement has to escape the /
    • insertafter is used to look for a line which matches that pattern and then insert/change the line we want below this pattern. This is useful when you have more than one line matching the regexp pattern or you want the line we are creating/replacing to be on a specific location. You see, without that if line does not exist, it is appended on the end of the file.
  • New autofs: The more modern /etc/auto.master file has the following lines in it:
    #
    # Include /etc/auto.master.d/*.autofs
    # The included files must conform to the format of this file.
    #
    +dir:/etc/auto.master.d
    #

    Instead of editing the /etc/auto.master file, which might be overwritten by an upgrade, we simply throw a file inside the /etc/auto.master.d directory which is then loaded into the /etc/auto.master file. This file, say /etc/auto.master.d/home.autofs, looks very much like what we did in the old autofs example, main difference is that it is a file

    raub@desktop:~/dev/ansible$ cat roles/common/files/home.autofs
    /home   /etc/auto.home --timeout=300
    raub@desktop:~/dev/ansible$
    that needs to be uploaded using the Ansible file copy module:

    - name: Enable auto.home in auto.master.d
      copy:
        src: home.autofs
        dest: /etc/auto.master.d/home.autofs
        owner: root
        group: root
        serole: _default
        setype: _default
        seuser: _default
        mode: 0644

Which one to pick? You know your setup, so pick the one that fits your needs.

(Re)Starting autofs

The final step we do need to do is (re)start autofs service after all this configuring. We do that using handlers:

- name: start autofs
  service:
    name: autofs
    state: started
    enabled: yes

- name: restart autofs
  service:
    name: autofs
    state: restarted
    enabled: yes

The way I use them is to start autofs when package is installed (see the notify statement),

- name: setup autofs
  package:
    name: autofs
    state: latest
  notify: start autofs
and then restart it after finishing with auto.home
- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs

After I unleash ansible, I then ssh as the non-root user which is allowed to login to vmhost2 and then see if fileshare was mounted:

[raub@vmhost2 ~]$ df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                               16G     0   16G   0% /dev
tmpfs                                  16G     0   16G   0% /dev/shm
tmpfs                                  16G  8.9M   16G   1% /run
tmpfs                                  16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/vmhost-root               2.0G   71M  2.0G   4% /
/dev/mapper/vmhost-usr                4.0G  1.8G  2.2G  46% /usr
/dev/sda2                             976M  179M  731M  20% /boot
/dev/sda1                             200M  6.8M  194M   4% /boot/efi
/dev/mapper/vmhost-var                4.0G  376M  3.7G  10% /var
/dev/mapper/vmhost-vg_backup           10G  104M  9.9G   2% /var/lib/libvirt/qemu/save
fileserver.example.com:/home/raub     690G  629G   61G  92% /home/raub
tmpfs                                 3.2G     0  3.2G   0% /run/user/1001
[raub@vmhost2 ~]$
[raub@vmhost2 ~]$ systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor pres>
   Active: active (running) since Mon 2019-12-09 14:17:26 EST; 2min 39s ago
 Main PID: 28387 (automount)
    Tasks: 6 (limit: 26213)
   Memory: 3.3M
   CGroup: /system.slice/autofs.service
           └─28387 /usr/sbin/automount --foreground --dont-check-daemon
[raub@vmhost2 ~]$

I do not know about you but it seems we have a winner. I will put a cleaner version of this playbook and supporting files in my github account later on.

Thursday, December 05, 2019

Replacing a VMWare ESXi host with a KVM one

Why?

Ok, I think I did my due diligence. With all the entries I have in this blog as proof, I think I put up long enough with the ESXi box. It was not as bad as Xen but I got tired of not being able to get it to behave as it should. And when I could not do PCI passthrough -- I am not even saying the Netronome card, but every single PCI or PCIe card I had available and had no problems passing to a vm guest using KVM as the hypervisor -- it was time to move on. The writing on the wall came after almost a year I could not get an answer from VMWare.

The Plan

  1. While the ESXi server, vmhost2, is still running, export the guests in .ovf format to a safe location. Just to be on the safe side, write down somewhere the specs for each guest (memory, cpu, OS, which network it is using, etc).
  2. Build the new vmhost2 using Debian or CentOS as the base OS and kvm as the hypervisor. Some things to watch out for:
    • Setup Network trunk and bridges to replicate old setup.
    • Use the same IP as before since we are keeping the same hostname.
    • Setup the logical volume manager so I can move things around later on.
    • Configure ntp to use our internal server
    • Configure DNS to use our internal server
    • Accounts of users who need to access the vm host itself will be mounted through autofs. If that fails, can login to root using ssh keypair authentication. If that is down (say, network issues, switch to console).
    • Like in the old vmhost2, ISOs for the install images are available through NSF.
    • Add whatever kernel options we might need. Remember we are building this from scratch, not just dropping a prebuilt system like xenserver. Or ESXi.
  3. Import enough vm guests to validate the system. Might take the opportunity to do some guest housecleaning.
  4. Add any PCI/PCIe cards we want to passthrough.
  5. Import the rest of the vm guests.
  6. (Future:) set it up so it can move/load balance vm guests with vmhost, the other KVM host.
Note: I did not wipe the original hard drive; instead I just bought a 111.8GB (128GB in fake numbers) SSD to run the OS in. I did not get a second drive and make it a RAID for now since I plan on running the OS in that disk, which will be configured using Ansible so I can rebuilt it quickly. Any vm guest running in this vm host will either run from the fileserver (iSCSI) or in a local RAID setup of sorts. Or I might simply deploy ZFS and be done with it. With that said, I might run a few vm gusts from that drive to validate the system.
Note: This project will be broken down into many articles otherwise it will be long and boring to read on a single sitting (some of the steps are applicable to other applications besides going from ESXi to KVM). I will try to come back and add the links of those articles, treating this post as the index.