I think from the title you know what I have in mind. My direct application is adding intel_iommu=on to the kernel in the KVM server I built to replace my VMWare ESXi one:
[root@vmhost2 ~]# virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'memory' controller mount-point : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpu' controller mount-point : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller mount-point : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller mount-point : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'devices' controller mount-point : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for cgroup 'blkio' controller mount-point : PASS QEMU: Checking for device assignment IOMMU support : PASS QEMU: Checking if IOMMU is enabled by kernel : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments) [root@vmhost2 ~]#The official docs would state the right way to do it is to use grub-mkconfig (might require grub-install first):
echo 'GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"' >> /etc/default/grub grub-mkconfig -o "$(readlink -f /etc/grub2.cfg)"
And then reboot. Problem with that is it applies intel_iommu=on to every single kernel listed in the grub menu. What if I just want to do that to one of the kernels (in my case the latest)? This way if something goes boink I can boot to the grub menu, select the last one, and continue booting.
Well, in previous CentOS version like 6 and 7, I would
- Open the grub.cfg
- Find the latest kernel menu entry (the top one)
- Find the line that tells which kernel to load for that version
- Append the option I wanted to add, say the intel_iommu=on from above:
linux16 /boot/vmlinuz-3.10.0-957.12.2.el7.x86_64 root=UUID=1a4cb560-eade-47cd-b1a5-57f8e0f53b8f ro console=tty0 crashkernel=auto console=ttyS0,115200 intel_iommu=on
- Reboot.
Enter the Grubby
Yes, this is a Today I Learned (TIL) event, and as you might have guessed, we are talking about grubby (if you click on the link you will go to the Red Hat official github repo for it), a command line tool to edit the boot config. I usually try to avoid commands that are but wrappers hiding what is really going on, but this one does seem to be useful (given I am no longer able to just edit the config file) and does not seem to require a lot of extra packages. It also came with both the Fedora31 and CentOS8 installs I have done. However when I saw which distros have prebuilt packages, none of the Debian-derived (ubuntu, mint, etc) are listed. It seems Ubuntu prefers update-grub, which is a wrapper around grub2-mkconfig, meaning it updates all the listed kernels.
Let's see what we can break:
- The man page says that to append arguments to a given kernel, we should run
grubby --update-kernel=the_kernel --args="kernel_args"
where the_kernel is the path to the kernel we want to edit. So, where are the kernel hiding and how to find out which one is the latest? For the first question, the kernel files are the ones starting with vmlinuz in the /boot directory:[root@vmhost2 ~]# ls /boot/ config-4.18.0-80.11.2.el8_0.x86_64 config-4.18.0-80.el8.x86_64 efi grub2 initramfs-0-rescue-133a53b45d2b47168497d47a34dd932f.img initramfs-4.18.0-80.11.2.el8_0.x86_64.img initramfs-4.18.0-80.11.2.el8_0.x86_64kdump.img initramfs-4.18.0-80.el8.x86_64.img initramfs-4.18.0-80.el8.x86_64kdump.img loader lost+found System.map-4.18.0-80.11.2.el8_0.x86_64 System.map-4.18.0-80.el8.x86_64 vmlinuz-0-rescue-133a53b45d2b47168497d47a34dd932f vmlinuz-4.18.0-80.11.2.el8_0.x86_64 vmlinuz-4.18.0-80.el8.x86_64 [root@vmhost2 ~]#
- Get the path of the latest kernel. From the previous step we know where they are, but now we need a way to identify the lastest one. We could write a script... or find in the man page that grubby has an option, --default-kernel,
[root@vmhost2 ~]# grubby --default-kernel /boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64 [root@vmhost2 ~]#
but is this default kernel the same as the latest one (currently in use)? Let's ask the host what is the current kernel (I did boot up with the latest one installed in vmhost2):[root@vmhost2 ~]# uname -a Linux vmhost2 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux [root@vmhost2 ~]#
Looks to be the case.
- Apply the above to update the kernel. Now we know how to identify the latest kernel, we can run
grubby --update-kernel=$(grubby --default-kernel) --args="kernel_args"
to add new arguments to the latest kernel. kernel_args is the space-separated list of the arguments we want to feed to it. The format should be the same you would feed to the kernel if booting msnuslly. For my vmhost2 kvm server, that would be intel_iommu=on. So,grubby --update-kernel=$(grubby --default-kernel) --args="intel_iommu=on"
followed by a reboot should be just fine.
- We can do it better. If we are only updating the latest/default kernel, passing DEFAULT as the_kernel does exactly what we want but with less effort from out part:
grubby --update-kernel DEFAULT --args="intel_iommu=on"
After we reboot, we can log back in and and see if it intel_iommu=on has been added:
[root@vmhost2 ~]# grubby --info DEFAULT index=0 kernel="/boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64" args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params "intel_iommu=on"" root="/dev/mapper/vmhost-root" initrd="/boot/initramfs-4.18.0-80.11.2.el8_0.x86_64.img $tuned_initrd" title="CentOS Linux (4.18.0-80.11.2.el8_0.x86_64) 8 (Core)" id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.11.2.el8_0.x86_64" [root@vmhost2 ~]#
Fine, but did it only change the latest kernel? Let's find out by picking another kernel and asking what's up (I have only two kernels listed so we pick the other one):
[root@vmhost2 ~]# grubby --info vmlinuz-4.18.0-80.el8.x86_64 index=1 kernel="/boot/vmlinuz-4.18.0-80.el8.x86_64" args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params" root="/dev/mapper/vmhost-root" initrd="/boot/initramfs-4.18.0-80.el8.x86_64.img $tuned_initrd" title="CentOS Linux (4.18.0-80.el8.x86_64) 8 (Core)" id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.el8.x86_64" [root@vmhost2 ~]#
And now kvm is happy since IOMMU is enabled:
[root@vmhost2 ~]# virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'memory' controller mount-point : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpu' controller mount-point : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller mount-point : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller mount-point : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'devices' controller mount-point : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for cgroup 'blkio' controller mount-point : PASS QEMU: Checking for device assignment IOMMU support : PASS QEMU: Checking if IOMMU is enabled by kernel : PASS [root@vmhost2 ~]#
Final thoughts
It seems they (don't you always wonder who "they" are?) are deprecating/phasing grubby out. And it is not available in Debian/Ubuntu/derivatives. So next time I play with kernel boot options, which will be soon, I will see about using a more generic solution.
