Updating only the latest/default Linux kernel boot arguments in CentOS/RedHat/Fedora

I think from the title you know what I have in mind. My direct application is adding intel_iommu=on to the kernel in the KVM server I built to replace my VMWare ESXi one:

[root@vmhost2 ~]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU 
appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
[root@vmhost2 ~]#

The official docs would state the right way to do it is to use grub-mkconfig (might require grub-install first):

echo 'GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"' >> /etc/default/grub
grub-mkconfig -o "$(readlink -f /etc/grub2.cfg)"

And then reboot. Problem with that is it applies intel_iommu=on to every single kernel listed in the grub menu. What if I just want to do that to one of the kernels (in my case the latest)? This way if something goes boink I can boot to the grub menu, select the last one, and continue booting.

Well, in previous CentOS version like 6 and 7, I would

1. Open the grub.cfg
   
2. Find the latest kernel menu entry (the top one)
   
3. Find the line that tells which kernel to load for that version
   
4. Append the option I wanted to add, say the intel_iommu=on from above:

   linux16 /boot/vmlinuz-3.10.0-957.12.2.el7.x86_64
   root=UUID=1a4cb560-eade-47cd-b1a5-57f8e0f53b8f ro console=tty0 crashkernel=auto
   console=ttyS0,115200 intel_iommu=on

5. Reboot.

But I can no longer do that since I cannot find the lines identifying the path to their respective kernels.

Enter the Grubby

Yes, this is a Today I Learned (TIL) event, and as you might have guessed, we are talking about grubby (if you click on the link you will go to the Red Hat official github repo for it), a command line tool to edit the boot config. I usually try to avoid commands that are but wrappers hiding what is really going on, but this one does seem to be useful (given I am no longer able to just edit the config file) and does not seem to require a lot of extra packages. It also came with both the Fedora31 and CentOS8 installs I have done. However when I saw which distros have prebuilt packages, none of the Debian-derived (ubuntu, mint, etc) are listed. It seems Ubuntu prefers update-grub, which is a wrapper around grub2-mkconfig, meaning it updates all the listed kernels.

Let's see what we can break:

1. The man page says that to append arguments to a given kernel, we should run

   grubby --update-kernel=the_kernel --args="kernel_args"

   where the_kernel is the path to the kernel we want to edit. So, where are the kernel hiding and how to find out which one is the latest? For the first question, the kernel files are the ones starting with vmlinuz in the /boot directory:

   [root@vmhost2 ~]# ls /boot/
   config-4.18.0-80.11.2.el8_0.x86_64
   config-4.18.0-80.el8.x86_64
   efi
   grub2
   initramfs-0-rescue-133a53b45d2b47168497d47a34dd932f.img
   initramfs-4.18.0-80.11.2.el8_0.x86_64.img
   initramfs-4.18.0-80.11.2.el8_0.x86_64kdump.img
   initramfs-4.18.0-80.el8.x86_64.img
   initramfs-4.18.0-80.el8.x86_64kdump.img
   loader
   lost+found
   System.map-4.18.0-80.11.2.el8_0.x86_64
   System.map-4.18.0-80.el8.x86_64
   vmlinuz-0-rescue-133a53b45d2b47168497d47a34dd932f
   vmlinuz-4.18.0-80.11.2.el8_0.x86_64
   vmlinuz-4.18.0-80.el8.x86_64
   [root@vmhost2 ~]#

2. Get the path of the latest kernel. From the previous step we know where they are, but now we need a way to identify the lastest one. We could write a script... or find in the man page that grubby has an option, --default-kernel,

   [root@vmhost2 ~]# grubby --default-kernel
   /boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64
   [root@vmhost2 ~]#

   but is this default kernel the same as the latest one (currently in use)? Let's ask the host what is the current kernel (I did boot up with the latest one installed in vmhost2):

   [root@vmhost2 ~]# uname -a
   Linux vmhost2 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
   [root@vmhost2 ~]#

   Looks to be the case.
   
3. Apply the above to update the kernel. Now we know how to identify the latest kernel, we can run

   grubby --update-kernel=$(grubby --default-kernel) --args="kernel_args"

   to add new arguments to the latest kernel. kernel_args is the space-separated list of the arguments we want to feed to it. The format should be the same you would feed to the kernel if booting msnuslly. For my vmhost2 kvm server, that would be intel_iommu=on. So,

   grubby --update-kernel=$(grubby --default-kernel) --args="intel_iommu=on"

   followed by a reboot should be just fine.
   
4. We can do it better. If we are only updating the latest/default kernel, passing DEFAULT as the_kernel does exactly what we want but with less effort from out part:

   grubby --update-kernel DEFAULT --args="intel_iommu=on"

After we reboot, we can log back in and and see if it intel_iommu=on has been added:

[root@vmhost2 ~]# grubby --info DEFAULT
index=0
kernel="/boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64"
args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params "intel_iommu=on""
root="/dev/mapper/vmhost-root"
initrd="/boot/initramfs-4.18.0-80.11.2.el8_0.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-80.11.2.el8_0.x86_64) 8 (Core)"
id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.11.2.el8_0.x86_64"
[root@vmhost2 ~]#

Fine, but did it only change the latest kernel? Let's find out by picking another kernel and asking what's up (I have only two kernels listed so we pick the other one):

[root@vmhost2 ~]# grubby --info vmlinuz-4.18.0-80.el8.x86_64
index=1
kernel="/boot/vmlinuz-4.18.0-80.el8.x86_64"
args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params"
root="/dev/mapper/vmhost-root"
initrd="/boot/initramfs-4.18.0-80.el8.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-80.el8.x86_64) 8 (Core)"
id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.el8.x86_64"
[root@vmhost2 ~]#

And now kvm is happy since IOMMU is enabled:

[root@vmhost2 ~]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
[root@vmhost2 ~]#

Final thoughts

It seems they (don't you always wonder who "they" are?) are deprecating/phasing grubby out. And it is not available in Debian/Ubuntu/derivatives. So next time I play with kernel boot options, which will be soon, I will see about using a more generic solution.

Mounting user fileshare on boot2docker boot

For docker container development, and light using, I found boot2docker to be quite convenient. I have it to boot off its ISO and then mount a permanent drive for the containers (and to store config files).

/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs

Whenever there is a new version, I shut the vm down, swap the ISOs (really just point the alias to the new file), and reboot. Nice and brainless.

I do not like to use the default account, docker, to do container development and running. Also, in a nice production environment you want to have other users running their containers. So, I created a user called ducker, which is a quick play on the default username. I also would prefer not having the user homedir in the drive where the containers are, which has been suggested before. You see, the way I see containers they are by design not important; blow them up if you feel like or wonder if they have been compromised. What matters is the data and the dockerfile required to rebuild the container. As a result, ducker has an account in the fileserver, which does its RAID and backup thingie as any good fileserver should. Now, if we want to have containers created and running from ducker's account when the server boots up, we need to have said account available.

So, the plan is that whenever the boot2docker server reboots, ducker will be there. And we then have a few issues to deal with. First, docker2file's ISO can't do automount. And second, usually when we shut down any user is lost because we are running of an ISO.

boot2docker's ISO is built so that we can provide some permanent stuff, which is why it mounts /dev/sda. But, that is not the only place we can mount things, nor the only way. In /opt/bootscript.sh we have the following interesting lines:

# Allow local bootsync.sh customisation
if [ -e /var/lib/boot2docker/bootsync.sh ]; then
    /bin/sh /var/lib/boot2docker/bootsync.sh
    echo "------------------- ran /var/lib/boot2docker/bootsync.sh"
fi

# Launch Docker
/etc/rc.d/docker

# Allow local HD customisation
if [ -e /var/lib/boot2docker/bootlocal.sh ]; then
    /bin/sh /var/lib/boot2docker/bootlocal.sh > /var/log/bootlocal.log 2>&1 &
    echo "------------------- ran /var/lib/boot2docker/bootlocal.sh"
fi

The files in /var/lib/boot2docker are in /dev/sda. Don't know which one to pick, but the second one does have a comforting Allow local HD customization message. So I will pick /var/lib/boot2docker/bootlocal.sh and add the following lines:

# Create local user, also creating the homedir
adduser -D -u 1003 ducker
# add user to docker group
adduser ducker docker
# Mount homedir
mount.nfs  fileserver.example.com:/export/home/ducker /home/ducker

After we create the file and reboot, we get

docker@boot2docker:~$ id ducker
uid=1003(ducker) gid=1003(ducker) groups=1003(ducker),100(docker)
docker@boot2docker:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    123.8M    772.8M  14% /
tmpfs                   498.1M         0    498.1M   0% /dev/shm
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda
cgroup                  498.1M         0    498.1M   0% /sys/fs/cgroup
df: /mnt/hgfs: Protocol error
fileserver.example.com:/home/ducker
                        295.3G    285.1G     10.0G  97% /home/ducker
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
docker@boot2docker:~$

User ducker can login because there is a public ssh key already in place. However, we did not do anything for that user's password, but there are a few ways to take care of that such as using /var/lib/boot2docker/bootlocal.sh to copy the hash into /etc/passwd. And, you really want to take care of that otherwise you will not be able to login.

Juniper SRX router booted from backup image (or, orange LED light again!)

So, uranus is in pain once more. Power went out and it was not properly shut down before the UPS gave its last gasp (I do need to do something about that). When I rebooted it, it seemed to have come back without an issue -- it was working fine -- but it had The Light on again just like it was in last time. I ssh'd into it and this is what the motd looked like:

raub@desktop:~$ ssh janitor@uranus.example.com
janitor@uranus.example.com's password:
--- JUNOS 12.1X45-D25.1 built 2014-04-23 20:45:48 UTC

***********************************************************************
**                                                                   **
**  WARNING: THIS DEVICE HAS BOOTED FROM THE BACKUP JUNOS IMAGE      **
**                                                                   **
**  It is possible that the primary copy of JUNOS failed to boot up  **
**  properly, and so this device has booted from the backup copy.    **
**                                                                   **
**  Please re-install JUNOS to recover the primary copy in case      **
**  it has been corrupted.                                           **
**                                                                   **
***********************************************************************

janitor@uranus>

Hmmm, so the normal image (does that mean the entire OS partition or just the boot stuff?) got corrupted when power went bye-bye. Good thing it keeps a backup. Out of curiosity, let's see if that is related to the LED being on:

janitor@uranus> show chassis alarms
1 alarms currently active
Alarm time               Class  Description
2015-02-26 05:38:12 EST  Minor  Host 0 Boot from backup root

janitor@uranus>

It seems to be the case. Funny that it is labelled minor but important enough to become the motd, but I digress. Personally, we should take care of that as soon as we have a scheduled maintenance window.

Ok, maintenance window time. Before rebooting, let's prepare a few things. If we know the backup copy is good (configuration files are also backed up, and you can push them back with ansible or whatnot if you feel uncomfortable), you could be lazy like me and copy the backup into the primary partition.

janitor@uranus> request system snapshot slice alternate
Formatting alternate root (/dev/da0s1a)...
Copying '/dev/da0s2a' to '/dev/da0s1a' .. (this may take a few minutes)
The following filesystems were archived: /

janitor@uranus>

If all went well, we should see the primary snapshot having the creation date of when we ran the ,tt>request system snapshot slice alternate command.

janitor@uranus> show system snapshot media internal
Information for snapshot on       internal (/dev/da0s1a) (primary)
Creation date: Apr 19 22:34:04 2015
JUNOS version on snapshot:
  junos  : 12.1X45-D25.1-domestic
Information for snapshot on       internal (/dev/da0s2a) (backup)
Creation date: Feb 26 05:33:18 2015
JUNOS version on snapshot:
  junos  : 12.1X45-D25.1-domestic

janitor@uranus>

As you noticed, I did this a while ago but never got around to making an article about it, but there it is. However, I still had the evil LED staring at me. Time to turn it off

request system reboot media internal

So I rebooted and did not get that motd anymore not the LED:

janitor@uranus> show chassis alarms
No alarms currently active

janitor@uranus>

Commands I need to remember

This will be a bit of a work-in-progress post. The idea is that I sometimes find neat commands I really should know because, let's be quite honest, they can save a lot of time and aggravation. I am going to make this a series of articles -- one specifically for linux, another for OSX, another for Solaris, another for Windows, one for generic unix stuff, and whatever else I can think of -- to keep things organized in my demented mind. So, let's start with linux:

Boot> linux dd

This is entered when you boot the machine up, whether by a CD or its own hard drive or whatever. You know, when the machine asks you if you want to change the boot options. I used that during a centos install and booting a (once again) centos liveCD distro. What it does is ask you during the boot process to insert a disk/cd/floppy/whatever with extra drivers it needs to know about. Made setting up a certain supermicro's onboard SATA raid controller possible right at boot time, because centos 4.4 was not aware of that raid controller.

Finding OS Version

I do not know about you but every so often I need to find out which version of OS I am running. Originally I thought all Linuces were the same, but I was wrong. In fact, that is the reason I started writing that down. Then, I decided to expand for the different unices and other operating systems I mess with. So, we have.

• Redhat (RHES4 box in this example):

  $ cat /etc/redhat-release
  Red Hat Enterprise Linux ES release 4 (Nahant)
  $

• ubuntu

  raub@keg:~$ ls /etc/issue*
  /etc/issue  /etc/issue.net
  raub@keg:~$ cat /etc/issue
  Ubuntu 7.04 \n \l
  
  raub@keg:~$ cat /etc/issue.net 
  Ubuntu 7.04
  raub@keg:~$ cat /etc/issue.net 
  Ubuntu 7.04
  raub@keg:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 7.04
  Release:        7.04
  Codename:       feisty
  raub@keg:~$ cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=7.04
  DISTRIB_CODENAME=feisty
  DISTRIB_DESCRIPTION="Ubuntu 7.04"
  raub@keg:~$ 

• AIX

  $ oslevel
  5.3.0.0
  $ oslevel -r
  5300-05
  $ 

• Solaris

boot {disk, cdrom, net} {-r}

open boot command to start the boot process. Normally you have the machine configured such that when it turns on it knows where to boot from, but if you are doing things to the machine and want to specify which device to boot from, you would use this command. I am not going to go over all its details but it suffice to say if you type disk it would boot off its default disk. The same goes for cdrom. Now, the -r option is used to tell it to scan for new hardware devices, which is important if you added/removed devices and drives and little cards from the machine.

Where does the scripts to strt files during the startup hide?

• Redhat

  /etc/rc.d/init.d/ssh

• BSD, Debian

  /etc/init.d/ssh

pkgadd

pkgadd is one of the most common ways to install packages in a Solaris box. It is similar in certain ways to rpm in Linux. Incidentally, if you want something more on the lines of apt-get, you want to look into pkg-get, which I shall talk about later on. But, for now, we spend a bit of time with pkgadd.

Let's say, for the sake of argument, that you decided to get a sunpci 2 pro card for your trusty Solaris box. So, after you install the card and reboot the machine (we will talk about the entire process in another blog entry), you go to the sun website and get the file SUNWspci2.3.2.tar.Z.

raub@nassau-25>ls
SUNWspci2.3.2.tar.Z
raub@nassau-26>

Then you untar/uncompress the file:

raub@nassau-26>zcat SUNWspci2.3.2.tar.Z | tar xf -
raub@nassau-27>ls
SUNWspci2  SUNWspci2.3.2.tar.Z
raub@nassau-28>

As you can see, you ended up with a new directory, SUNWspci2. Now the next step is to run pkgadd itself:

root@nassau-19>pkgadd -d .

The following packages are available:
 1  SUNWspci2     SunPCi II
                  (sparc) 2.3.2

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:
[...]

You will notice that I entered a dot (.) as the name of the package. What that means is every package in the current directory. I could specify which package but I am lazy. Also, I am given the option of which packages to install anyway. As you probably guesses, -i stands for install package.

After careful consideration, I decided not to include AIX in this entry; since it is so different from the other Unix flavours, I've decided to make its own entry. Keep an eye for it!