Showing posts with label debian. Show all posts
Showing posts with label debian. Show all posts

Sunday, December 13, 2020

Yet another post on the demise of CentOS and the ascension of CentOS Stream

A friend of mine has been told me that 2020 is the year you expect one catastophe each month: we had Australian wildfires, pandemic (which still is going on), murder hornets, and so on. And now it is December!

Excluding those who might have been inside a cave, or have been otherwise distracted with real life events, it is fair to assume you know that CentOS as it is was (i.e. from before its 2014 aquisition by RedHat until now) will cease to exist in 2021, it will be replaced with CentOS Stream. I will not comment on the reason and whether that was to be expected after the IBM aquisition of RedHat; there is already a lot of people doing that, the comment session of that centos.org blog entry included. Instead, let's talk about the few things I know. From what I have gathered, its relationship with RHES has changed a bit. Before, changes were made to RHES and then applied to CentOS, but things have changed. For instance

What to do?

I think that depends on how you feel about this and how much you have tied into CentOS.

  1. Stick with CentOS Stream. This may or not work; it probably would be wise to wait until, say, the middle of 2021 to see how this works out.
  2. Upgrade to a RHES subscription. In their defense, my experience with their paid technical support was very good. I do not know if my experience was unique.
  3. If you have a single private computer running CentOS, you can get a dev license for RHES.
  4. Switch to another RHES port. There are a few options here, which in a certain way behave like CentOS of old.
  5. Switch to another distro. If you do not want to have any further business with a Red Hat distro, there are options for server (Arch linux: please step away from the line even though you have excellents docs.) duties:
    Linux
    • Debian, a very stable operating system. Its project leader, Jonathan Carter, wrote a blog about the CentOS demise.
    • Ubuntu, which it seems to be the platform of choice of many new developments. Just check where code for GPUs and FPGAs is first written on.
    • Open SuSe. It may not be as popular as Debian and RedHat based distros but it is a good contender.
    NOT Linux
    • FreeBSD. Very stable UNIX operating system. Package list is smaller than Linux though, but this is a proper server operating system. If you like ZFS, you may want to investigate it, or at least FreeNAS (Now called TrueNAS).
    • OpenBSD. From the same people who brought us openssl and openssh. When was the last time you heard of people breaking into an OpenBSD box?

And that is all for now. If you expected a nice closing argument, there is none. This is just a change. Think of CentOS as a cheese: it moved; now you have to choose if you are going to follow it or look for another cheese.

Monday, December 09, 2019

Configuring autofs in a generic way on redhat/debian derived distros in ansible

Let's say you want to install and configure autofs so to have network fileshares mounted on demand. That sounds like a good task for Ansible.

Installing AutoFS

The installing part as you know is fairly easy. In both RedHat and Debian derivatives the package name is autofs. Since we are not doing anything special we can create a rather generic Ansible task using the Ansible package module. What that means is instead of having to worry about having a version for, say, Ubuntu and RedHat, the package module uses whatever package manager is the default for the distro in question:

- name: setup autofs
  package:
    name: autofs
    state: latest
NOTE: I know package works in the redhat/debian derived distros, but I am not sure if it will work in other Linux flavours. I also would check the autofs package name for these other distros.

Configuring AutoFS

Next thing we want to do is ensure we are using the right NFS version. That is done finding the line beginning with mount_nfs_default_protocol and editing its value. In my case I want to make sure it is using NFS v4, which is the default anyway in a modern autofs package. So, why bother? Well, call me paranoid: I want to have exactly what I want. Or call it a simple example of using the Ansible lineinfile module. Or maybe someone is using NFS v3 and want to see how to change it.

- name: Ensure nfs v4
  lineinfile:
    path: /etc/autofs.conf
    regexp: '^mount_nfs_default_protocol '
    line: 'mount_nfs_default_protocol = 4'

Let's use this as an excuse to talk about lineinfile: the regexp here is looking for a line that starts with the string 'mount_nfs_default_protocol '; I wrote it in quotes because it includes the blank space after mount_nfs_default_protocol. Note that the search pattern also means "and anything else to the end of the line," so a line looking like this

mount_nfs_default_protocol could be something you do not want to touch
is fair game. I know usually in regexp you would end the query statement with (.*)$ to include everything to the end of the page, but just nod a lot and move on. Now if the looked like this:
#mount_nfs_default_protocol could be something you do not want to touch
the regexp would not work because it expects the line to begin with m. line defines what we want the line to look like. If it matches that, no changes made.

The next step is to define what we want to use autofs for. In my case, I want to mount user home directories off the fileserver. That means creating a /etc/auto.home file which describes which fileserver we are using. For this I suggest using the template module since we can define the name of the fileserver somewhere earlier in the playbook or in a config file (I am thinking here of a file in host_vars/ or group_vars/ associated with the host in question.). In my task file I use something like

- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs
which
  1. Grabs the template templates/auto.home.j2 and puts it in /etc/auto.home
  2. Sets the permission, ownership, and selinux parameters for /etc/auto.home. The _default means that if there is a default selinux setting for that file/directory we will use it.

And roles/common/templates/auto.home.j2 looks like this:

/etc/auto.home
#
# File: /etc/auto.home
#
*   -fstype=nfs4,hard,intr,rsize=8192,wsize=8192 {{ nfs_server }}:/home/&
where nfs_server is the name of the nfs server defined somewhere else.

Now, there are two ways to deal with it, the old autofs and the new one.

  • Old autofs: In the old days, you would edit the /etc/auto.master file, adding a line underneath +auto.master that would tell us how to mount user home directory. In the following example,
    +auto.master
    /home   /etc/auto.home --timeout=300
    
    the bottom line is saying "if you notice someone trying to access a file/directory in /home, go to /etc/auto.home to see how to mount it. But, if there is no activity after 300 seconds, unmount that." I use timeout of 300 seconds; change it to fit your needs. Now we need to add that to /etc/auto.master, which we will do using the lineinfile module once more:

    - name: Enable auto.home in auto.master
      lineinfile:
        path: /etc/auto.master
        regexp: '^\/home'
        insertafter: '^\+auto.master'
        line: /home   /etc/auto.home --timeout=300

    As you can see, it is a little more complex than the previous task:

    • The regexp statement has to escape the /
    • insertafter is used to look for a line which matches that pattern and then insert/change the line we want below this pattern. This is useful when you have more than one line matching the regexp pattern or you want the line we are creating/replacing to be on a specific location. You see, without that if line does not exist, it is appended on the end of the file.
  • New autofs: The more modern /etc/auto.master file has the following lines in it:
    #
    # Include /etc/auto.master.d/*.autofs
    # The included files must conform to the format of this file.
    #
    +dir:/etc/auto.master.d
    #

    Instead of editing the /etc/auto.master file, which might be overwritten by an upgrade, we simply throw a file inside the /etc/auto.master.d directory which is then loaded into the /etc/auto.master file. This file, say /etc/auto.master.d/home.autofs, looks very much like what we did in the old autofs example, main difference is that it is a file

    raub@desktop:~/dev/ansible$ cat roles/common/files/home.autofs
    /home   /etc/auto.home --timeout=300
    raub@desktop:~/dev/ansible$
    that needs to be uploaded using the Ansible file copy module:

    - name: Enable auto.home in auto.master.d
      copy:
        src: home.autofs
        dest: /etc/auto.master.d/home.autofs
        owner: root
        group: root
        serole: _default
        setype: _default
        seuser: _default
        mode: 0644

Which one to pick? You know your setup, so pick the one that fits your needs.

(Re)Starting autofs

The final step we do need to do is (re)start autofs service after all this configuring. We do that using handlers:

- name: start autofs
  service:
    name: autofs
    state: started
    enabled: yes

- name: restart autofs
  service:
    name: autofs
    state: restarted
    enabled: yes

The way I use them is to start autofs when package is installed (see the notify statement),

- name: setup autofs
  package:
    name: autofs
    state: latest
  notify: start autofs
and then restart it after finishing with auto.home
- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs

After I unleash ansible, I then ssh as the non-root user which is allowed to login to vmhost2 and then see if fileshare was mounted:

[raub@vmhost2 ~]$ df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                               16G     0   16G   0% /dev
tmpfs                                  16G     0   16G   0% /dev/shm
tmpfs                                  16G  8.9M   16G   1% /run
tmpfs                                  16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/vmhost-root               2.0G   71M  2.0G   4% /
/dev/mapper/vmhost-usr                4.0G  1.8G  2.2G  46% /usr
/dev/sda2                             976M  179M  731M  20% /boot
/dev/sda1                             200M  6.8M  194M   4% /boot/efi
/dev/mapper/vmhost-var                4.0G  376M  3.7G  10% /var
/dev/mapper/vmhost-vg_backup           10G  104M  9.9G   2% /var/lib/libvirt/qemu/save
fileserver.example.com:/home/raub     690G  629G   61G  92% /home/raub
tmpfs                                 3.2G     0  3.2G   0% /run/user/1001
[raub@vmhost2 ~]$
[raub@vmhost2 ~]$ systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor pres>
   Active: active (running) since Mon 2019-12-09 14:17:26 EST; 2min 39s ago
 Main PID: 28387 (automount)
    Tasks: 6 (limit: 26213)
   Memory: 3.3M
   CGroup: /system.slice/autofs.service
           └─28387 /usr/sbin/automount --foreground --dont-check-daemon
[raub@vmhost2 ~]$

I do not know about you but it seems we have a winner. I will put a cleaner version of this playbook and supporting files in my github account later on.

Friday, July 29, 2016

Checking if RedHat/CentOS has new updates

If you use Debian Linux derivatives, specially Ubuntu, you probably noticed when you login it will tell you (using the MOTD) that there are new updates waiting for you. And, you can use that if, say, you are writing a script to let you know about that; someone I know use that to monitor his Ubunu boxes in Nagios. But, what about in RedHat and derviatives the lazy way?

Let's do some thinking aloud and see if we can come up with something. We know that if we run yum check-update, it should reply with the list of packages needing to be upgraded if any

[root@vmhost ~][raub@duckwitch ~]$ yum check-update
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.supremebytes.com
 * extras: mirror.hostduplex.com
 * updates: mirror.scalabledns.com

chkconfig.x86_64                        1.3.61-5.el7_2.1                 updates
device-mapper.x86_64                    7:1.02.107-5.el7_2.5             updates
device-mapper-libs.x86_64               7:1.02.107-5.el7_2.5             updates
dracut.x86_64                           033-360.el7_2.1                  updates
glibc.x86_64                            2.17-106.el7_2.6                 updates
glibc-common.x86_64                     2.17-106.el7_2.6                 updates
iproute.x86_64                          3.10.0-54.el7_2.1                updates
kernel.x86_64                           3.10.0-327.22.2.el7              updates
kpartx.x86_64                           0.4.9-85.el7_2.5                 updates
libxml2.x86_64                          2.9.1-6.el7_2.3                  updates
ntpdate.x86_64                          4.2.6p5-22.el7.centos.2          updates
pcre.x86_64                             8.32-15.el7_2.1                  updates
selinux-policy.noarch                   3.13.1-60.el7_2.7                updates
selinux-policy-targeted.noarch          3.13.1-60.el7_2.7                updates
systemd.x86_64                          219-19.el7_2.11                  updates
systemd-libs.x86_64                     219-19.el7_2.11                  updates
systemd-python.x86_64                   219-19.el7_2.11                  updates
systemd-sysv.x86_64                     219-19.el7_2.11                  updates
tzdata.noarch                           2016f-1.el7                      updates
[raub@duckwitch ~]$

As you can see, it does not require you to run that command as root. And you can even check if a repo you use but configured to be normally disabled, like epel in the following example:

raub@duckwitch ~]$ yum check-update --enablerepo=epel
Loaded plugins: fastestmirror
epel/x86_64/metalink                                     |  11 kB     00:00
epel                                                     | 4.3 kB     00:00
(1/3): epel/x86_64/group_gz                                | 170 kB   00:00
(2/3): epel/x86_64/updateinfo                              | 584 kB   00:00
(3/3): epel/x86_64/primary_db                              | 4.2 MB   00:00
Loading mirror speeds from cached hostfile
 * base: mirror.supremebytes.com
 * epel: mirror.chpc.utah.edu
 * extras: mirror.hostduplex.com
 * updates: mirror.scalabledns.com

chkconfig.x86_64                        1.3.61-5.el7_2.1                 updates
device-mapper.x86_64                    7:1.02.107-5.el7_2.5             updates
device-mapper-libs.x86_64               7:1.02.107-5.el7_2.5             updates
dracut.x86_64                           033-360.el7_2.1                  updates
epel-release.noarch                     7-7                              epel
glibc.x86_64                            2.17-106.el7_2.6                 updates
glibc-common.x86_64                     2.17-106.el7_2.6                 updates
iproute.x86_64                          3.10.0-54.el7_2.1                updates
kernel.x86_64                           3.10.0-327.22.2.el7              updates
kpartx.x86_64                           0.4.9-85.el7_2.5                 updates
libxml2.x86_64                          2.9.1-6.el7_2.3                  updates
ntpdate.x86_64                          4.2.6p5-22.el7.centos.2          updates
pcre.x86_64                             8.32-15.el7_2.1                  updates
selinux-policy.noarch                   3.13.1-60.el7_2.7                updates
selinux-policy-targeted.noarch          3.13.1-60.el7_2.7                updates
systemd.x86_64                          219-19.el7_2.11                  updates
systemd-libs.x86_64                     219-19.el7_2.11                  updates
systemd-python.x86_64                   219-19.el7_2.11                  updates
systemd-sysv.x86_64                     219-19.el7_2.11                  updates
tzdata.noarch                           2016f-1.el7                      updates
[raub@duckwitch ~]$

What if there are no upgrades?

[root@server1 ~]# yum check-update
Loaded plugins: fastestmirror
base                                                     | 3.6 kB     00:00
extras                                                   | 3.4 kB     00:00
updates                                                  | 3.4 kB     00:00
Loading mirror speeds from cached hostfile
 * base: mirror.us.leaseweb.net
 * extras: mirror.us.leaseweb.net
 * updates: reflector.westga.edu
[root@server1 ~]#

Sounds like we need to check for the first blank line. We can do that using sed. We can find the blank line by using the /^\s*$/ search pattern in sed. So we could start with something like (the -n is there because we only care when we find said blank line)

yum check-update | `sed -n '/^\s*$/p'`

which if you run does not seem to do much. Reason is that what we really want is to know if the match was successful or not. And, sed actually has the answer: look at these entries I stole from its man pange:

q [exit-code]
              Immediately  quit  the  sed  script  without processing any more
              input, except that if auto-print is  not  disabled  the  current
              pattern  space will be printed.  The exit code argument is a GNU
              extension.

       Q [exit-code]
              Immediately quit the sed  script  without  processing  any  more
              input.  This is a GNU extension.

Let's try it then. First we will find a machine that has a package that needs to be updated. In this case it will be my KVM server, which you have met before when we talked about USB passthrough:

[raub@vmhost ~]$ yum check-update 
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: mirror.vcu.edu
 * extras: centos.mirror.constant.com
 * updates: mirror.vcu.edu

samba4-libs.x86_64                    4.2.10-7.el6_8                     updates
[raub@vmhost ~]$

So as of this writing it has one update. We then try our one-liner, which we want to return 0 if it did not find any pending upgrades and 1 if it did. And that is done by adding q1 past the search string, as in /^\s*$/q1, which means write 1 if search successful. Of course we now need to print the result, which can be done with echo $?. So, let's try it:

[raub@vmhost ~]$ yum check-update | `sed -n '/^\s*$/q1'` ; echo $? 1 [raub@vmhost ~]$

It thinks that it found it. I am not completely sure it works, so we need also to verify it works as it should when we find no updates:

[root@server1 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.netdepot.com
 * extras: mirror.solarvps.com
 * updates: ftpmirror.your.org
[root@server1 ~]# yum check-update | `sed -n '/^\s*$/q1'` ; echo $?
0
[root@server1 ~]# 

Sounds like we have a winner. Now all that is left is to wrap something around it to something with that. I will leave that to you. Note that it does not differentiate between normal and security updates though.

Sunday, March 29, 2015

Building a single/simple-minded website in Python

A lot of scripts I end up writing begin as solutions to a problem or a way for me to be lazy and maybe learn something in the process; today's topic is no different. As some of you know, when you are deploying new Linux boxes from scratch, you can be lazy and use stuff like kickstart (centos/RedHat) and preseed (ubuntu, Debian) to do the initial install before handing over to something like Puppet, Ansible, or Salt (to name a few). Thing is you need to feed the preseed/kickstart file somehow.

One way to do it is to build a .iso or setup network install (you know the drill: PXE boot, DHCP, and so on). A very nice example (which I myself have used before) is shown in the CentOS docs. Ubuntu/Debian have something very similar. Now, the step that is relevant to this blog entry is the one in which the preseed/kickstart file is passed to the new host you are building. We can make it available in a web server, and tell the new machine where it is. Just to let you know, if you are using docker, the concept is the same. I know I am going really quickly through this because all I am doing right now is explaining the need that caused me to write this.

So, we established we need a web server to feed the preseed/kickstart file. But, there are times we do not need a full fledged website, all we want it to do is to offer one single file. And once the file is provided and the host is created, the website can go away. I imagine you are smelling some kind of automated host building script that automagically creates the web server it needs in the process. And you are right, which is why I wanted something with as little footprint as I can get away with. In other words, I would love to have a web server that completely runs off a single script.

To do the deed, I chose to use Python. Besides the fact I suck at ruby, I bumped into an example of a simple python-based webserver using something called BaseHttpServer. I modified it a bit and came up with the following script to serve a preseed.cfg file:

#! /usr/bin/env python
'''
Simple dumb webserver to serve a file.
It will try to serve a file called preseed.cfg, located in the directory
program was called, on localhost:8000

The idea is you can ask for any file you want, and will get what we
give to you.

Shamelessly based on https://wiki.python.org/moin/BaseHttpServer
'''
import time
import BaseHTTPServer

HOST_NAME = '' # Accept requests on all interfaces
PORT_NUMBER = 8000
FILE = 'preseed.cfg'

def read_file(filename):
    '''
    Read in (text) file and return it as a string
    '''
    file = open(filename, "r")
    return file.read()

class MyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_GET(self):
        file = read_file(FILE)
        self.send_response(200)
        self.send_header()
        self.end_headers()
        self.wfile.write(file)

if __name__ == '__main__':
    server_class = BaseHTTPServer.HTTPServer
    httpd = server_class((HOST_NAME, PORT_NUMBER), MyHandler)
    print time.asctime(), "Server Starts - %s:%s" % (HOST_NAME, PORT_NUMBER)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    print time.asctime(), "Server Stops - %s:%s" % (HOST_NAME, PORT_NUMBER)

Here's a quick rundown on what it does:

  1. Import the two libraries we need. Note they are default libraries every python install should have. This is not supposed to be a fancy or remotely clever script.
  2. Define some constants.
    HOST_NAME = '' # Accept requests on all interfaces
    PORT_NUMBER = 8000
    FILE = 'preseed.cfg'
    In the script I showed how to take the lazy way and make the service listen on all interfaces. In fact, when you run the above script, it should show in netstat as
    raub@desktop:~$ netstat -apn|grep 8000
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    tcp   0   0 0.0.0.0:8000    0.0.0.0:*     LISTEN     11539/python
    raub@desktop:~$
    As you know, setting the IP to 0.0.0.0 means everyone + the cat, which is why you can see it is listening on every interface in this machine, localhost included, on port 8000.
  3. The MyHandler class, which handles all the http requests, only cares about processing GET events. And, when it sees one, all it does is spits out the file preseed.cfg as a Content-type: text/plain.
  4. When you run the script, it should show something like
    Mon Mar 23 09:17:50 2015 Server Starts - :8000
    when it starts. And then when someone actually does hit the server, it would show a message like
    192.168.5.10 - - [23/Mar/2015 09:28:13] "GET / HTTP/1.0" 200 -
    which would indicate that 192.168.5.10 connected to our little webserver and, as a result, got the preseed file. If you use wget,
    wget the-server:8000
    it will create a index.html file with the contents of preseed.cfg

I will be the first to say this Python script is very small and dumbed down from the script shown in the wiki. I did that for a reason: since it ignores any real request from user (no matter what you ask, it only sends the config file), it is very simple minded in a good way. Asking it to show the list of files somewhere or upload something might be a bit challenging. Now, you might want instead of offering this config file to serve some kind of simple webpage that is created on the fly, like some status page. You could use a script like the above to do the deed.

I guess where I am really getting to is that if you need, say, a webserver to only server one simple stupid page changes are you do not need a full Apache install. In my own case, why I would even want to have a full fledged webserver running 24/7 just to server a page (or many pages) that only need to be available for a few minutes? I know this concept is not hip anymore, but there is something to be said about having a simple tool that does one single thing well and can cooperate with the other tools to build a complex task.

As I mentioned above, the script is pretty hopelessly dumb. And I bet you can improve on it. I mean, even I decided to improve on it a bit. Specifically, I wanted to be able to provide the filename, IP address (so it is only running on the network interface using that IP), and port from the command line. That would make it easier to use the script without having to modify its code.

And I found that BaseHttpServer really did not want to do that. In fact, what it really want is to read the request from a client and do something based on that. Since that is not what I wanted, I had to learn how to, well, hack my way around that by overriding the __init__ constructor. I am not going to waste time here posting the modified code; I placed the script on github where I hope one day to prettify/improve it.

Notes

  • If you do not want to use python, you can run a webserver in one line of bash or Powershell