Showing posts with label centos. Show all posts
Showing posts with label centos. Show all posts

Saturday, May 18, 2024

Configuring an interface in Linux without(!) nmcli

In ages past, when you wanted to configure a network interface eth0 with MAC CC:00:FF:EE:12:34 with static IP 128.227.3.20 and using uranus (128.227.3.1) as the gateway in Red Hat Linux or its derivatives, you could have something like this

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << 'EOF'
DEVICE="eth0"
BOOTPROTO="static"
HWADDR="CC:00:FF:EE:12:34"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
DHCP_HOSTNAME=vmhost
IPADDR=128.227.3.20
NETMASK=255.255.255.0
GATEWAY=128.227.3.1
EOF

in your notes and cut-n-paste it as needed. The same would work for a vlan trunk (think 802.1q), where the interface using 128.227.3.20 is now associated to tagged vlan 3; all we needed is 3 files.

  • Define the base interface (we could have used uuid instead of MAC address)
    cat /etc/sysconfig/network-scripts/ifcfg-eno1 << 'EOF'
    TYPE=Ethernet
    NAME="eno1"
    DEVICE="eno1"
    HWADDR="CC:00:FF:EE:12:34"
    BOOTPROTO="none"
    ONBOOT=yes
  • Define the tagged vlan 3, and which bridge it is associated with
    cat /etc/sysconfig/network-scripts/ifcfg-eno1.3 << 'EOF'
    DEVICE="eno1.3"
    BOOTPROTO="none"
    NM_CONTROLLED="no"
    ONBOOT="yes"
    VLAN=yes
    BRIDGE=dmzbr
    EOF
  • Define the bridge with static IP
    cat /etc/sysconfig/network-scripts/ifcfg-dmzbr << 'EOF'
    DEVICE=dmzbr
    TYPE=Bridge
    BOOTPROTO="static"
    NM_CONTROLLED="no"
    ONBOOT="yes"
    TYPE="Ethernet"
    DHCP_HOSTNAME=vmhost
    IPADDR=128.227.3.20
    NETMASK=255.255.255.0
    GATEWAY=128.227.3.1
    EOF

We could get away with two but the 3rd one is there because I like to use bridges. The awake reader will have noticed I switched from eth0 to eno1; I will leave that to the reader but the point here is the above applies to whatever networking naming convention you have to deal with.

You shalll count to three (Monty Python)

As some (those who do not fall asleep reading these posts) know, I like automation and one of my tools of choice is ansible. Creating the above files for a given host from network declarations in its host_vars/host is very convenient using ansible.

But, there is Network Manager. Until recently -- CentOS 8, Rocky 8, Alma 8 -- you could still tell Network Manager to leave these interfaces alone (the NM_CONTROLLED entry), but now I am building a rocky 9 host, I am being forced to use nmcli instead. And what would it take to nmcli all of that? Let's find out (I think I missed a step here, so don't trust this):

nmcli con add ifname dmzbr type bridge con-name dmzbr
nmcli con modify dmzbr ipv6.method disabled
nmcli connection modify dmzbr ipv4.address 192.168.3.20/24
nmcli connection modify dmzbr ipv4.gateway 192.168.3.1
nmcli connection modify dmzbr ipv4.dns 192.168.3.1
nmcli connection add type vlan con-name eno1.3 ifname eno1.3 dev eno1 id 3
nmcli connection modify eno1.3 master dmzbr slave-type bridge
nmcli connection up eno1.3
nmcli connection up dmzbr

And this should end up with something like this

[root@testbox ~]# nmcli con show
NAME       UUID                                  TYPE      DEVICE
DMZ        6a97eddf-ac72-45d9-ba29-d12c7d59b511  vlan      eno1.3
dmzBridge  ccffabb8-0c8a-47d8-a2d6-15cab0e9b53b  bridge    dmzbr
eno1       9b1b155f-8197-3a7a-a9cc-79cab8b92da1  ethernet  eno1
lo         2dd22561-0fd3-436a-9da0-a53c61d63848  loopback  lo
[root@testbox ~]#

Before you get excited, that UUID is not set in stone. If you are going to do this in Ansible, take a look at the official docs on the nmcli_module. Short version is you are doing all those nmcli commands I did before, which I know I need to check since I know I missed something (I changed the bridge name later). And that is the proper official way to do the deed.

And then there is you

Yep, there is me. I know I will make a mistake. So, let's take a look on this network manager thing. No matter what, the configuration of all of these network interfaces have to go somewhere? If I am unlucky, some kind of binary-only database like Microsoft. If I am lucky, a text file. Well, it turned out my luck still holds: the files are hidden in /etc/NetworkManager/system-connections/:

[root@testbox ~]# ls /etc/NetworkManager/system-connections/
dmzbr.nmconnection  eno1.nmconnection   DMZ.nmconnection
[root@testbox ~]# 

Let's take a look at eno1:

[root@testbox ~]# cat /etc/NetworkManager/system-connections/eno1.nmconnection 
[connection]
id=eno1
uuid=9b1b155f-8197-3a7a-a9cc-79cab8b92da1
type=ethernet
interface-name=eno1
timestamp=1711852227

[ethernet]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=eui64
method=disabled

[proxy]
[root@testbox ~]# 

admit it, it may have a different, more grandiose format than /etc/sysconfig/network-scripts/ifcfg-eno1 but it describes the same thing. What would it take to make my own files? Long story short. not much:

[root@testbox ~]# more /etc/NetworkManager/system-connections/dmzbr.nmconnection /etc/NetworkManager/system-connections/DMZ.nmconnection 
::::::::::::::
/etc/NetworkManager/system-connections/dmzbr.nmconnection
::::::::::::::
[connection]
id=dmzBridge
type=bridge
interface-name=dmzbr

[ethernet]

[bridge]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=default
method=disabled

[proxy]
::::::::::::::
/etc/NetworkManager/system-connections/DMZ.nmconnection
::::::::::::::
[connection]
id=DMZ
type=vlan
interface-name=eno1.3
master=dmzbr
slave-type=bridge

[ethernet]

[vlan]
flags=1
id=3
parent=eno1

[bridge-port]
[root@testbox ~]# 

Note I did not even bother to declare the timestamp or uuid; the later can be created on the fly as the interface comes online. What does that mean? I can keep a copy of these files in a safe place, In fact, I have the following file (that is the filename, I swear)

[root@testbox ~]# cat NICs/eno1-oh_shit.nmconnection 
[connection]
id=eno1
uuid=9b1b155f-8197-3a7a-a9cc-79cab8b92da1
type=ethernet
autoconnect-priority=-999
interface-name=eno1

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]
[root@testbox ~]# 

The idea of this file is if I screw the network up and need to get it back quickly, I copy that to /etc/NetworkManager/system-connections/eno1.nmconnection, make sure it is connected to an untagged vlan switchport, and then restart network manager. That will then get a dhcp IP address and off it goes.

What about Ansible?

The same way I created /etc/sysconfig/network-scripts/ifcfg-eno1.3 I can create these files; the template looks a bit different but everything else is similar. Heretic? Surely, but that is how I roll.

Sunday, December 13, 2020

Yet another post on the demise of CentOS and the ascension of CentOS Stream

A friend of mine has been told me that 2020 is the year you expect one catastophe each month: we had Australian wildfires, pandemic (which still is going on), murder hornets, and so on. And now it is December!

Excluding those who might have been inside a cave, or have been otherwise distracted with real life events, it is fair to assume you know that CentOS as it is was (i.e. from before its 2014 aquisition by RedHat until now) will cease to exist in 2021, it will be replaced with CentOS Stream. I will not comment on the reason and whether that was to be expected after the IBM aquisition of RedHat; there is already a lot of people doing that, the comment session of that centos.org blog entry included. Instead, let's talk about the few things I know. From what I have gathered, its relationship with RHES has changed a bit. Before, changes were made to RHES and then applied to CentOS, but things have changed. For instance

What to do?

I think that depends on how you feel about this and how much you have tied into CentOS.

  1. Stick with CentOS Stream. This may or not work; it probably would be wise to wait until, say, the middle of 2021 to see how this works out.
  2. Upgrade to a RHES subscription. In their defense, my experience with their paid technical support was very good. I do not know if my experience was unique.
  3. If you have a single private computer running CentOS, you can get a dev license for RHES.
  4. Switch to another RHES port. There are a few options here, which in a certain way behave like CentOS of old.
  5. Switch to another distro. If you do not want to have any further business with a Red Hat distro, there are options for server (Arch linux: please step away from the line even though you have excellents docs.) duties:
    Linux
    • Debian, a very stable operating system. Its project leader, Jonathan Carter, wrote a blog about the CentOS demise.
    • Ubuntu, which it seems to be the platform of choice of many new developments. Just check where code for GPUs and FPGAs is first written on.
    • Open SuSe. It may not be as popular as Debian and RedHat based distros but it is a good contender.
    NOT Linux
    • FreeBSD. Very stable UNIX operating system. Package list is smaller than Linux though, but this is a proper server operating system. If you like ZFS, you may want to investigate it, or at least FreeNAS (Now called TrueNAS).
    • OpenBSD. From the same people who brought us openssl and openssh. When was the last time you heard of people breaking into an OpenBSD box?

And that is all for now. If you expected a nice closing argument, there is none. This is just a change. Think of CentOS as a cheese: it moved; now you have to choose if you are going to follow it or look for another cheese.

Thursday, July 30, 2020

Variable expansion and searching for packages that contain a file using yum/dnf

I know some articles in this blog are rather clever, but this one is here to remind me (learn from my mistakes!) that understanding how a command thinks is important. I was having some issues with cryptsetup and was told (by Matthew Heon: let me make sure to recognize him for throwing a searchlight at my problem. Thanks!) I the file /usr/share/cracklib/pw_dict.pwd.gz was missing. Fine, this is a CentOS 8 docker container. I can use yum (until they remove it completely) or its replacement, dnf, to look for it. I will be using yum in this discussion knowing that they are interchangeable within the limits of this article.

If the file in in the directory /usr/share/cracklib, chances are it belongs to the cracklib package, so let's begin by seeing what we have matching that:

[root@moe /]# yum search cracklib
Failed to set locale, defaulting to C.UTF-8
========================== Name Exactly Matched: cracklib ==========================
cracklib.x86_64 : A password-checking library
cracklib.i686 : A password-checking library
========================= Name & Summary Matched: cracklib =========================
cracklib-dicts.x86_64 : The standard CrackLib dictionaries
[root@moe /]#

Oh, there are more than one, so we need to be more specific. That is a great job for the whatprovides option; it allows us to find all the packages that contain a given file.

[root@moe /]# yum whatprovides pw_dict.pwd.gz
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:27 ago on Tue Jul 28 20:32:27 2020.
Error: No Matches found
[root@moe /]#
I did learn that sometimes looking for a package by just providing the filename of a file that belongs to it does not work well, but if you make it look like you are giving a path will work. And this path can begin with a * so it can expand the path to any path in the system. So, let's try that and hope for the best:
[root@moe /]# yum whatprovides */pw_dict.pwd.gz
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 17:43:12 ago on Tue Jul 28 20:32:27 2020.
Error: No Matches found
[root@moe /]#
What is going on? Well, let's up on a limb: of the three cracklib-related files, cracklib-dicts seems to be the one with the most potential because the file we want is a dictionary. And then see what lurks in /usr/share/cracklib/:
[root@moe /]# yum install cracklib-dicts
[...]
[root@moe /]# ls /usr/share/cracklib/
cracklib-small.hwm  cracklib-small.pwi  pw_dict.hwm  pw_dict.pwi
cracklib-small.pwd  cracklib.magic      pw_dict.pwd
[root@moe /]#

A candle lights over my heard, indicating I was enlightened: it is called pw_dict.pwd, not pw_dict.pwd.gz! I did not account for it to be in a different format (gzipped in this case)! Well duh!

With that in mind, we should see if we could have saved some aggravation. We expanded the search path by entering */pw_dict.pwd.gz before; would that work for the filename? Let's find out:

[root@moe /]# yum whatprovides */pw_dict.pwd
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 17:43:45 ago on Tue Jul 28 20:32:27 2020.
cracklib-dicts-2.9.6-15.el8.x86_64 : The standard CrackLib dictionaries
Repo        : @System
Matched from:
Filename    : /usr/share/cracklib/pw_dict.pwd

cracklib-dicts-2.9.6-15.el8.x86_64 : The standard CrackLib dictionaries
Repo        : BaseOS
Matched from:
Filename    : /usr/share/cracklib/pw_dict.pwd

[root@moe /]# yum whatprovides pw_dict.*
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 17:46:32 ago on Tue Jul 28 20:32:27 2020.
Error: No Matches found
[root@moe /]#
[root@moe /]# yum whatprovides */pw_dict.*
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 17:44:42 ago on Tue Jul 28 20:32:27 2020.
cracklib-dicts-2.9.6-15.el8.x86_64 : The standard CrackLib dictionaries
Repo        : @System
Matched from:
Filename    : /usr/share/cracklib/pw_dict.hwm
Filename    : /usr/share/cracklib/pw_dict.pwd
Filename    : /usr/share/cracklib/pw_dict.pwi

cracklib-dicts-2.9.6-15.el8.x86_64 : The standard CrackLib dictionaries
Repo        : BaseOS
Matched from:
Filename    : /usr/share/cracklib/pw_dict.hwm
Filename    : /usr/share/cracklib/pw_dict.pwd
Filename    : /usr/share/cracklib/pw_dict.pwi

[root@moe /]# yum whatprovides */*_dict.pwd
Failed to set locale, defaulting to C.UTF-8
cracklib-dicts-2.9.6-15.el8.x86_64 : The standard CrackLib dictionaries
Repo        : BaseOS
Matched from:
Filename    : /usr/lib64/cracklib_dict.pwd
Filename    : /usr/share/cracklib/pw_dict.pwd

[root@moe /]#

Interesting that we really do not need to tack a * to the end of the search pattern. So, what we learned from this article is that if searching for a package a given file belongs to does not work, we can broaden the search by replacing part of the filename in question with a *. And that we do not need that if the bit of the filename we are taking is at the end.

Learning something useful in this blog: who would've thought?

Tuesday, January 14, 2020

Updating only the latest/default Linux kernel boot arguments in CentOS/RedHat/Fedora

I think from the title you know what I have in mind. My direct application is adding intel_iommu=on to the kernel in the KVM server I built to replace my VMWare ESXi one:

[root@vmhost2 ~]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU 
appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
[root@vmhost2 ~]#
The official docs would state the right way to do it is to use grub-mkconfig (might require grub-install first):

echo 'GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"' >> /etc/default/grub
grub-mkconfig -o "$(readlink -f /etc/grub2.cfg)"

And then reboot. Problem with that is it applies intel_iommu=on to every single kernel listed in the grub menu. What if I just want to do that to one of the kernels (in my case the latest)? This way if something goes boink I can boot to the grub menu, select the last one, and continue booting.

Well, in previous CentOS version like 6 and 7, I would

  1. Open the grub.cfg
  2. Find the latest kernel menu entry (the top one)
  3. Find the line that tells which kernel to load for that version
  4. Append the option I wanted to add, say the intel_iommu=on from above:
    linux16 /boot/vmlinuz-3.10.0-957.12.2.el7.x86_64
    root=UUID=1a4cb560-eade-47cd-b1a5-57f8e0f53b8f ro console=tty0 crashkernel=auto
    console=ttyS0,115200 intel_iommu=on
  5. Reboot.
But I can no longer do that since I cannot find the lines identifying the path to their respective kernels.

Enter the Grubby

Yes, this is a Today I Learned (TIL) event, and as you might have guessed, we are talking about grubby (if you click on the link you will go to the Red Hat official github repo for it), a command line tool to edit the boot config. I usually try to avoid commands that are but wrappers hiding what is really going on, but this one does seem to be useful (given I am no longer able to just edit the config file) and does not seem to require a lot of extra packages. It also came with both the Fedora31 and CentOS8 installs I have done. However when I saw which distros have prebuilt packages, none of the Debian-derived (ubuntu, mint, etc) are listed. It seems Ubuntu prefers update-grub, which is a wrapper around grub2-mkconfig, meaning it updates all the listed kernels.

Let's see what we can break:

  1. The man page says that to append arguments to a given kernel, we should run
    grubby --update-kernel=the_kernel --args="kernel_args"
    where the_kernel is the path to the kernel we want to edit. So, where are the kernel hiding and how to find out which one is the latest? For the first question, the kernel files are the ones starting with vmlinuz in the /boot directory:
    [root@vmhost2 ~]# ls /boot/
    config-4.18.0-80.11.2.el8_0.x86_64
    config-4.18.0-80.el8.x86_64
    efi
    grub2
    initramfs-0-rescue-133a53b45d2b47168497d47a34dd932f.img
    initramfs-4.18.0-80.11.2.el8_0.x86_64.img
    initramfs-4.18.0-80.11.2.el8_0.x86_64kdump.img
    initramfs-4.18.0-80.el8.x86_64.img
    initramfs-4.18.0-80.el8.x86_64kdump.img
    loader
    lost+found
    System.map-4.18.0-80.11.2.el8_0.x86_64
    System.map-4.18.0-80.el8.x86_64
    vmlinuz-0-rescue-133a53b45d2b47168497d47a34dd932f
    vmlinuz-4.18.0-80.11.2.el8_0.x86_64
    vmlinuz-4.18.0-80.el8.x86_64
    [root@vmhost2 ~]#
  2. Get the path of the latest kernel. From the previous step we know where they are, but now we need a way to identify the lastest one. We could write a script... or find in the man page that grubby has an option, --default-kernel,
    [root@vmhost2 ~]# grubby --default-kernel
    /boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64
    [root@vmhost2 ~]#
    but is this default kernel the same as the latest one (currently in use)? Let's ask the host what is the current kernel (I did boot up with the latest one installed in vmhost2):
    [root@vmhost2 ~]# uname -a
    Linux vmhost2 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
    [root@vmhost2 ~]#
    Looks to be the case.
  3. Apply the above to update the kernel. Now we know how to identify the latest kernel, we can run
    grubby --update-kernel=$(grubby --default-kernel) --args="kernel_args"
    to add new arguments to the latest kernel. kernel_args is the space-separated list of the arguments we want to feed to it. The format should be the same you would feed to the kernel if booting msnuslly. For my vmhost2 kvm server, that would be intel_iommu=on. So,
    grubby --update-kernel=$(grubby --default-kernel) --args="intel_iommu=on"
    followed by a reboot should be just fine.
  4. We can do it better. If we are only updating the latest/default kernel, passing DEFAULT as the_kernel does exactly what we want but with less effort from out part:
    grubby --update-kernel DEFAULT --args="intel_iommu=on"

After we reboot, we can log back in and and see if it intel_iommu=on has been added:

[root@vmhost2 ~]# grubby --info DEFAULT
index=0
kernel="/boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64"
args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params "intel_iommu=on""
root="/dev/mapper/vmhost-root"
initrd="/boot/initramfs-4.18.0-80.11.2.el8_0.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-80.11.2.el8_0.x86_64) 8 (Core)"
id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.11.2.el8_0.x86_64"
[root@vmhost2 ~]#

Fine, but did it only change the latest kernel? Let's find out by picking another kernel and asking what's up (I have only two kernels listed so we pick the other one):

[root@vmhost2 ~]# grubby --info vmlinuz-4.18.0-80.el8.x86_64
index=1
kernel="/boot/vmlinuz-4.18.0-80.el8.x86_64"
args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params"
root="/dev/mapper/vmhost-root"
initrd="/boot/initramfs-4.18.0-80.el8.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-80.el8.x86_64) 8 (Core)"
id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.el8.x86_64"
[root@vmhost2 ~]#

And now kvm is happy since IOMMU is enabled:

[root@vmhost2 ~]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
[root@vmhost2 ~]#

Final thoughts

It seems they (don't you always wonder who "they" are?) are deprecating/phasing grubby out. And it is not available in Debian/Ubuntu/derivatives. So next time I play with kernel boot options, which will be soon, I will see about using a more generic solution.

Monday, December 09, 2019

Configuring autofs in a generic way on redhat/debian derived distros in ansible

Let's say you want to install and configure autofs so to have network fileshares mounted on demand. That sounds like a good task for Ansible.

Installing AutoFS

The installing part as you know is fairly easy. In both RedHat and Debian derivatives the package name is autofs. Since we are not doing anything special we can create a rather generic Ansible task using the Ansible package module. What that means is instead of having to worry about having a version for, say, Ubuntu and RedHat, the package module uses whatever package manager is the default for the distro in question:

- name: setup autofs
  package:
    name: autofs
    state: latest
NOTE: I know package works in the redhat/debian derived distros, but I am not sure if it will work in other Linux flavours. I also would check the autofs package name for these other distros.

Configuring AutoFS

Next thing we want to do is ensure we are using the right NFS version. That is done finding the line beginning with mount_nfs_default_protocol and editing its value. In my case I want to make sure it is using NFS v4, which is the default anyway in a modern autofs package. So, why bother? Well, call me paranoid: I want to have exactly what I want. Or call it a simple example of using the Ansible lineinfile module. Or maybe someone is using NFS v3 and want to see how to change it.

- name: Ensure nfs v4
  lineinfile:
    path: /etc/autofs.conf
    regexp: '^mount_nfs_default_protocol '
    line: 'mount_nfs_default_protocol = 4'

Let's use this as an excuse to talk about lineinfile: the regexp here is looking for a line that starts with the string 'mount_nfs_default_protocol '; I wrote it in quotes because it includes the blank space after mount_nfs_default_protocol. Note that the search pattern also means "and anything else to the end of the line," so a line looking like this

mount_nfs_default_protocol could be something you do not want to touch
is fair game. I know usually in regexp you would end the query statement with (.*)$ to include everything to the end of the page, but just nod a lot and move on. Now if the looked like this:
#mount_nfs_default_protocol could be something you do not want to touch
the regexp would not work because it expects the line to begin with m. line defines what we want the line to look like. If it matches that, no changes made.

The next step is to define what we want to use autofs for. In my case, I want to mount user home directories off the fileserver. That means creating a /etc/auto.home file which describes which fileserver we are using. For this I suggest using the template module since we can define the name of the fileserver somewhere earlier in the playbook or in a config file (I am thinking here of a file in host_vars/ or group_vars/ associated with the host in question.). In my task file I use something like

- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs
which
  1. Grabs the template templates/auto.home.j2 and puts it in /etc/auto.home
  2. Sets the permission, ownership, and selinux parameters for /etc/auto.home. The _default means that if there is a default selinux setting for that file/directory we will use it.

And roles/common/templates/auto.home.j2 looks like this:

/etc/auto.home
#
# File: /etc/auto.home
#
*   -fstype=nfs4,hard,intr,rsize=8192,wsize=8192 {{ nfs_server }}:/home/&
where nfs_server is the name of the nfs server defined somewhere else.

Now, there are two ways to deal with it, the old autofs and the new one.

  • Old autofs: In the old days, you would edit the /etc/auto.master file, adding a line underneath +auto.master that would tell us how to mount user home directory. In the following example,
    +auto.master
    /home   /etc/auto.home --timeout=300
    
    the bottom line is saying "if you notice someone trying to access a file/directory in /home, go to /etc/auto.home to see how to mount it. But, if there is no activity after 300 seconds, unmount that." I use timeout of 300 seconds; change it to fit your needs. Now we need to add that to /etc/auto.master, which we will do using the lineinfile module once more:

    - name: Enable auto.home in auto.master
      lineinfile:
        path: /etc/auto.master
        regexp: '^\/home'
        insertafter: '^\+auto.master'
        line: /home   /etc/auto.home --timeout=300

    As you can see, it is a little more complex than the previous task:

    • The regexp statement has to escape the /
    • insertafter is used to look for a line which matches that pattern and then insert/change the line we want below this pattern. This is useful when you have more than one line matching the regexp pattern or you want the line we are creating/replacing to be on a specific location. You see, without that if line does not exist, it is appended on the end of the file.
  • New autofs: The more modern /etc/auto.master file has the following lines in it:
    #
    # Include /etc/auto.master.d/*.autofs
    # The included files must conform to the format of this file.
    #
    +dir:/etc/auto.master.d
    #

    Instead of editing the /etc/auto.master file, which might be overwritten by an upgrade, we simply throw a file inside the /etc/auto.master.d directory which is then loaded into the /etc/auto.master file. This file, say /etc/auto.master.d/home.autofs, looks very much like what we did in the old autofs example, main difference is that it is a file

    raub@desktop:~/dev/ansible$ cat roles/common/files/home.autofs
    /home   /etc/auto.home --timeout=300
    raub@desktop:~/dev/ansible$
    that needs to be uploaded using the Ansible file copy module:

    - name: Enable auto.home in auto.master.d
      copy:
        src: home.autofs
        dest: /etc/auto.master.d/home.autofs
        owner: root
        group: root
        serole: _default
        setype: _default
        seuser: _default
        mode: 0644

Which one to pick? You know your setup, so pick the one that fits your needs.

(Re)Starting autofs

The final step we do need to do is (re)start autofs service after all this configuring. We do that using handlers:

- name: start autofs
  service:
    name: autofs
    state: started
    enabled: yes

- name: restart autofs
  service:
    name: autofs
    state: restarted
    enabled: yes

The way I use them is to start autofs when package is installed (see the notify statement),

- name: setup autofs
  package:
    name: autofs
    state: latest
  notify: start autofs
and then restart it after finishing with auto.home
- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs

After I unleash ansible, I then ssh as the non-root user which is allowed to login to vmhost2 and then see if fileshare was mounted:

[raub@vmhost2 ~]$ df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                               16G     0   16G   0% /dev
tmpfs                                  16G     0   16G   0% /dev/shm
tmpfs                                  16G  8.9M   16G   1% /run
tmpfs                                  16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/vmhost-root               2.0G   71M  2.0G   4% /
/dev/mapper/vmhost-usr                4.0G  1.8G  2.2G  46% /usr
/dev/sda2                             976M  179M  731M  20% /boot
/dev/sda1                             200M  6.8M  194M   4% /boot/efi
/dev/mapper/vmhost-var                4.0G  376M  3.7G  10% /var
/dev/mapper/vmhost-vg_backup           10G  104M  9.9G   2% /var/lib/libvirt/qemu/save
fileserver.example.com:/home/raub     690G  629G   61G  92% /home/raub
tmpfs                                 3.2G     0  3.2G   0% /run/user/1001
[raub@vmhost2 ~]$
[raub@vmhost2 ~]$ systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor pres>
   Active: active (running) since Mon 2019-12-09 14:17:26 EST; 2min 39s ago
 Main PID: 28387 (automount)
    Tasks: 6 (limit: 26213)
   Memory: 3.3M
   CGroup: /system.slice/autofs.service
           └─28387 /usr/sbin/automount --foreground --dont-check-daemon
[raub@vmhost2 ~]$

I do not know about you but it seems we have a winner. I will put a cleaner version of this playbook and supporting files in my github account later on.

Thursday, May 23, 2019

Programming a Netronome network card (inside a VM) from command line

This is the same card(s) we got to work inside a vm guest using the magic of PCI passthrough. Netronome wants us to use a Windows-only IDE to do development work in it while the card is placed in a Linux box we can reach; some of its features remind me of the IDE Google has for Androids, which allows you to run an emulator and do some real time debugging. The difference is the Google one works in Linux, Windows, and Mac, and it only requires one computer (which could be remotely accessed).

Do we really need to use the Netronome SDK? I guess it depends on what we want to do. For now, let's see if we can get something running using command line only to the point we can compile in Micro-C and run something in the card.


Get the packages

  1. First we need a few packages available for either CentOS or Ubuntu.
    Note: The Netronome Linux SDK officially only support CentOS and Ubuntu. So we will only be covering those distros.
    • Ubuntu:
      apt-get install libftdi1 libjansson4 build-essential \
       linux-headers-`uname -r` dkms git
    • CentOS: (Still using yum; will do a dnf version when I feel like.
      yum -y install epel-release && yum update -y
      yum -y install libftdi jansson pciutils kernel-devel dkms wget git

    Netronome does require you to have an account to get the SDK packages. I can't help with that; what I can tell you is that once I got the account I downloaded everything which was available at the time I wrote this article:

    raub@desktop:~$ ls Downloads/netronome/
    SDK
    agilio-nfp-driver-dkms-2018.01.11.2333.f40482a-1.el7.noarch.rpm
    agilio-nfp-driver-dkms_2018.01.11.2333.f40482a_all.deb
    firmware
    readme
    raub@desktop:~$ ls Downloads/netronome/SDK/
    6.0.4.1 6.1.0.1
    raub@desktop:~$ ls Downloads/netronome/SDK/6.1.0.1/
    nfp-sdk-6.1.0.1-preview-3286-setup.exe
    nfp-sdk-6.1.0.1_preview-0-3243.x86_64.rpm
    nfp-sdk-p4-rte-6.1.0.1-preview-3202.centos.x86_64.tar
    nfp-sdk-p4-rte-6.1.0.1-preview-3214.ubuntu.x86_64.tar
    nfp-sdk-sim-6.1.0.0-preview-3179.x86_64.tar
    nfp-sdk_6.1.0.1-preview-3243-2_amd64.deb
    nfp-toolchain-6.1.0.1-preview-3243.x86_64.tar
    raub@desktop:~$

    and then copied them all to the development vm guest we created in the previous article, desktop1

    What are those files? I put what I have gathered about their function in the readme file (it covers the old file version but it should get an idea):
    Programmer Studio IDE
    nfp-sdk-6.0.4.1-3276-setup.exe - Windows
    
    Run Time Environment (RTE)
    nfp-sdk-p4-rte-6.0.4.1-3195.ubuntu.x86_64.tgz
    nfp-sdk-p4-rte-6.0.4.1-3191.centos.x86_64.tgz
    
    Hosted Toolchain (to be used with BSP and SmartNIC)
    nfp-sdk_6.0.4.1-3227-2_amd64.deb
    nfp-sdk-6.0.4.1-0-3227.x86_64.rpm
    
    NFP Simulator
    nfp-sdk-sim-6.0.4.1-3177.x86_64.tgz
    
    Hosted Toolchain (to be used with NFP Simulator)
    nfp-toolchain-6.0.4.1-3227.x86_64.tgz
    Note: There are two versions of the SDK. Just pick the latest.
  2. Then install the basic SDK
    • Ubuntu:
      sudo dpkg -i nfp-sdk_6.1.0.1-preview-3243-2_amd64.deb
    • CentOS:
      sudo rpm -ivh nfp-sdk-6.1.0.1_preview-0-3243.x86_64.rpm

    This creates a /opt/netronome directory.

  3. And add to the path where the binaries will be installed.
    cat >> ~/.bash_profile << 'EOF'
    
    # Netronome SDK
    PATH=$PATH:/opt/netronome/bin
    export PATH
    EOF
    source ~/.bash_profile
    Note: If the user you are building your code on does not have rights to write to the card, you should edit the root's .bash_profile file as well.
  4. We do need the Netronome modified but open source nfp driver which has the development features (specificially, it has a nfp_dev_cpp option we will need to expose the low-level user space access ABIs of non-netdev mode) we need. So, we install it, which requires installing the Netronome repo:

    • Ubuntu:
      wget https://deb.netronome.com/gpg/NetronomePublic.key
      apt-key add NetronomePublic.key
      add-apt-repository "deb https://deb.netronome.com/apt stable main"
      apt-get update
      apt-get install agilio-nfp-driver-dkms
    • CentOS:
      wget https://rpm.netronome.com/gpg/NetronomePublic.key
      rpm --import NetronomePublic.key
      cat << 'EOF' > /etc/yum.repos.d/netronome.repo
      [netronome]
      name=netronome
      baseurl=https://rpm.netronome.com/repos/centos/
      gpgcheck=0
      enabled=1
      EOF
      yum makecache
      yum install -y agilio-nfp-driver-dkms --nogpgcheck
    and then reboot.
  5. Now we can install the RTE
    • Ubuntu:
      tar xvf nfp-sdk-p4-rte-6.1.0.1-preview-3214.ubuntu.x86_64.tar
      cd nfp-sdk-6-rte-v6.1.0.1-preview-Ubuntu-Release-r2750-2018-10-10-ubuntu.binary/
      sudo ./sdk6_rte_install.sh install
    • CentOS:
      tar xvf nfp-sdk-p4-rte-6.1.0.1-preview-3202.centos.x86_64.tar
      cd nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary/
      sudo ./sdk6_rte_install.sh install
    This should cause /opt/netronome/bin/ to fill with many more files; this is a good way to check progress.
    NOTE:Chances are It will get pissed:
    [...]
    Loaded plugins: fastestmirror
    Examining /home/centos/netronome/SDK/6.1.0.1/nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary/dependencies/nfp-bsp/rpm//nfp-bsp-dkms_2018.08.17.1104_all.rpm: nfp-bsp-dkms-2018.08.17.1104-1dkms.noarch
    Marking /home/centos/netronome/SDK/6.1.0.1/nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary/dependencies/nfp-bsp/rpm//nfp-bsp-dkms_2018.08.17.1104_all.rpm to be installed
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfp-bsp-dkms.noarch 0:2018.08.17.1104-1dkms will be installed
    --> Processing Conflict: agilio-nfp-driver-dkms-2019.04.02.0225.bf81349-1.el7.noarch conflicts nfp-bsp-dkms
    Loading mirror speeds from cached hostfile
     * base: packages.oit.ncsu.edu
     * epel: mirror.umd.edu
     * extras: packages.oit.ncsu.edu
     * updates: packages.oit.ncsu.edu
    No package matched to upgrade: nfp-bsp-dkms
    --> Finished Dependency Resolution
    Error: agilio-nfp-driver-dkms conflicts with nfp-bsp-dkms-2018.08.17.1104-1dkms.noarch
     You could try using --skip-broken to work around the problem
     You could try running: rpm -Va --nofiles --nodigest
    Error! There are no instances of module: nfp-bsp-dkms
    located in the DKMS tree.
    [centos@desktop1 nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary]$
    but it will get over and will work fine.
  6. Ensure that nfp_dev_cpp = 1
    theuser@desktop1:~$ cat /sys/module/nfp/parameters/nfp_dev_cpp
    1
    theuser@desktop1:~$ 

    If not, say, yet get an error message like this

    [theuser@desktop1 ~]# cat /sys/module/nfp/parameters/nfp_dev_cpp
    cat: /sys/module/nfp/parameters/nfp_dev_cpp: No such file or directory
    [theuser@desktop1 ~]#

    uninstall nfp and install it back with the option set. There are ways to load said option at boot time; I will leave that as an exercise to the reader.

    theuser@desktop1:~$ sudo modprobe -r -v nfp && sudo modprobe nfp nfp_dev_cpp=1
    theuser@desktop1:~$
  7. Ensure nfp-hwinfo is talking to the card. The expected outcome should look like this:
    theuser@desktop1:~$ sudo /opt/netronome/bin/nfp-hwinfo
    nfp.interface=pci.0.0
    nfp.model=0x40010010
    nfp.serial=00:15:4d:13:5d:2b
    board.exec=bootloader.bin
    uart.baud=115200
    preinit.setup.version=nfp-bsp-6000-b0 (4ef1e19ba176)
    pcie0.type=ep
    assembly.revision=11
    assembly.model=lithium
    assembly.partno=AMDA0096-0001
    assembly.serial=17290647
    assembly.vendor=SMC
    ddr0.spd=spi:1:0:0x3F0F00
    ddr1.spd=spi:1:0:0x3F0F00
    ddr2.spd=none
    ddr3.spd=none
    ddr4.spd=none
    ddr5.spd=none
    emu1.type=cache
    emu2.type=cache
    ethm.mac=00:15:4d:13:5d:2b
    eth.mac=00:15:4d:13:5d:2c
    eth.macs=2
    vpd=fis:1:0:vpd.bin
    board.setup.version=nfp-bsp-6000-b0 (4ef1e19ba176)
    chip.model=NFP4001
    chip.revision=B0
    core.speed=633
    me.speed=633
    arm.speed=475
    chip.model.device=0x62006c20
    chip.identifier=0x219b8546c
    chip.model.hard=0x5
    chip.model.soft=0x40010096
    chip.route=0xc96f1e8e
    chip.island=0x1001f13000112
    mem.setup.version=nfp-bsp-6000-b0 (4ef1e19ba176)
    ddr0.mem.size=1024
    ddr1.mem.size=1024
    ddr0.mem.speed=1600
    ddr1.mem.speed=1600
    emu0.mem.size=2048
    emu0.mem.base=0x2000000000
    emu1.mem.size=3
    [...]
    theuser@desktop1:~$

    If it looks like this:

    theuser@desktop1:~$ sudo /opt/netronome/bin/nfp-hwinfo
    /opt/netronome/bin/nfp-hwinfo: Failed to open NFP device 0 (No such device)
    Please check that:
     -lspci -d 19ee: shows atleast one Netronome device
     -the nfp device number is correct
     -the user has read and write permissions to the Netronome device
     -the nfp.ko module is loaded
     -the nfp_dev_cpp option is enabled (please try modinfo nfp to see all params)
    theuser@desktop1:~$ 
    stop, do not continue. Go back and check if nfp_dev_cpp = 1 and also if the vm was configured to support PCIe cards. Do not continue until you have checked and addressed these two items.

Coding, at last!

This Hello World was stolen from the Netronome appropriately named Hello World example. I will be rushing through it, concentrating on getting it to compile and showing some common issues. Lookup on the example docs for what each line does.

  1. So we create our hello world project using lab_template as the, well, template.
    mkdir dev
    cd dev
    git clone https://github.com/open-nfpsw/c_packetprocessing.git
    cd c_packetprocessing/apps/
    cp -r lab_template lab_hello_world
    cd lab_hello_world
    NOTE: This creates a ~/dev/c_packetprocessing/apps/lab_hello_world directory. If you want to move it to a different location, edit the line
    ROOT_SRC_DIR  ?= $(realpath $(app_src_dir)/../..)
    in the Makefile.
  2. So far the hello world directory looks rather bare:

    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ ls
    Makefile  README
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

    so let's start populating it.

    cat > hello_world.c << 'EOF'
    #include <nfp.h>
    __declspec(ctm) int old[] = {1,2,3,4,5,6,7,8,9,10};
    __declspec(ctm) int new[sizeof(old)/sizeof(int)];
    
    int main(void)
    {
            if (__ctx() == 0)
            {
                    int i, size;
                    size = sizeof(old)/sizeof(int);
                    for (i=0; i < size; i++)
                    {
                            new[i] = old[size - i - 1];
                    }
            }
            return 0;
    }
    EOF
  3. We add a few lines to the makefile. Their explanation is listed in.

    sed -i -e '/^# Application definition starts here/ a\
    $(eval $(call micro_c.compile_with_rtl,hello_world_obj,hello_world.c)) \
    $(eval $(call fw.add_obj,hello_world,hello_world_obj,i32.me0 i32.me1)) \
    $(eval $(call fw.link_with_rtsyms,hello_world))' Makefile
  4. Time for some compiling!

    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ make
    /opt/netronome/bin/nfcc -Fo/home/theuser/dev/c_packetprocessing/apps/lab_hello_world/ -Fe/home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world_obj.list -W3 -chip nfp-4xxx-b0 -Qspill=7 -Qnn_mode=1 -Qno_decl_volatile -single_dram_signal -Qnctx_mode=8 -I. -I/home/theuser/dev/c_packetprocessing/microc/include -I/home/theuser/dev/c_packetprocessing/microc/lib   /opt/netronome/components/standardlibrary/microc/src/rtl.c /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.c
    /opt/netronome/bin/nfld -chip nfp-4xxx-b0 -mip -rtsyms -o /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw -map /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.map -u i32.me0 /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world_obj.list -u i32.me1 /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world_obj.list
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

    which creates a few intermediate files:

    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ ls
    hello_world.c   hello_world.map  hello_world_obj.list  README
    hello_world.fw  hello_world.obj  Makefile              rtl.obj
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ cat hello_world.map
    Memory Map file: /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.map
    Date: Tue May  7 10:39:30 2019
    
    nfld version: 6.0.4.1,  NFFW: /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw
    
    Address       Region     ByteSize        Symbol
    ===================================================
    0x0000000000800000    i24.emem      108                 .mip
    0x0000000000000000    i32.ctm       704                 i32.me0.ctm_40$tls
    0x00000000000002c0    i32.ctm       704                 i32.me1.ctm_40$tls
    
    ImportVar                       Uninitialized Value
    ===================================================
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$
  5. Next upload the firmware we created into the card. This needs to be run either as root or as an user who can write to card.
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# make load_hello_world
    nfp-nffw load --no-start /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#
    NOTE: If you see the following error message
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ make load_hello_world
    nfp-nffw load --no-start /home/centos/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw
    nfp-nffw: Failed to open NFP device 0 (No such device)
    Please check that:
     -lspci -d 19ee: shows atleast one Netronome device
     -the nfp device number is correct
     -the user has read and write permissions to the Netronome device
     -the nfp.ko module is loaded
     -the nfp_dev_cpp option is enabled (please try modinfo nfp to see all params)
    nfp-nffw: Command 'load' failed
    make: *** [load_hello_world] Error 1
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$
    ou should check if
    • If you are running make load_hello_world as user who can write to the card.
    • nfp_dev_cpp = 1
    • the vm was configured to support PCIe cards.
    Go back in this document for instructions on how to do so.

    Now, if you see this error message

    [F] nfp6000_nffw.c:4643: Firmware already loaded. Unload first.
    Failed to load firmware: Operation not permitted
    nfp-nffw: Command 'load' failed
    Makefile:43: recipe for target 'load_hello_world' failed
    either you or someone else had already loaded firmware into the card. All you have to do is unload it
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ nfp-nffw unload
    theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$
    and then run make load_hello_world again.

  6. In the hello world instructions, the next step is to see the card memory since later on we will be writing to it. So, here is it.
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# nfp-rtsym --len 176 i32.me0.ctm_40\$tls:0
    0x0000000000:  0x00000001 0x00000002 0x00000003 0x00000004
    0x0000000010:  0x00000005 0x00000006 0x00000007 0x00000008
    0x0000000020:  0x00000009 0x0000000a 0x00000000 0x00000000
    0x0000000030:  0x00000000 0x00000000 0x00000000 0x00000000
    *
    0x0000000050:  0x00000000 0x00000000 0x00000001 0x00000002
    0x0000000060:  0x00000003 0x00000004 0x00000005 0x00000006
    0x0000000070:  0x00000007 0x00000008 0x00000009 0x0000000a
    0x0000000080:  0x00000000 0x00000000 0x00000000 0x00000000
    *
    
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#
  7. Unleash the code so it does things:
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# make fw_start
    nfp-nffw start
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# 
  8. If things were successfully done, we now can see the memory contents have changed:
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# nfp-rtsym --len 176 i32.me0.ctm_40\$tls:0
    0x0000000000:  0x00000001 0x00000002 0x00000003 0x00000004
    0x0000000010:  0x00000005 0x00000006 0x00000007 0x00000008
    0x0000000020:  0x00000009 0x0000000a 0x00000000 0x00000000
    0x0000000030:  0x0000000a 0x00000009 0x00000008 0x00000007
    0x0000000040:  0x00000006 0x00000005 0x00000004 0x00000003
    0x0000000050:  0x00000002 0x00000001 0x00000001 0x00000002
    0x0000000060:  0x00000003 0x00000004 0x00000005 0x00000006
    0x0000000070:  0x00000007 0x00000008 0x00000009 0x0000000a
    0x0000000080:  0x00000000 0x00000000 0x00000000 0x00000000
    *
    
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#
  9. Don't forget to unload the firmware by typing nfp-nffw unload!
  10. Checking that we are done
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# nfp-rtsym --len 176 i32.me0.ctm_40\$tls:0
    No runtime symbol named 'i32.me0.ctm_40$tls'
    root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#

So congratulations! You not only installed the SDK and wrote and ran your first Netronome program! You may want to look into the Network Flow C Compiler User's Guide for further info on what you can do; I would put the link but right on the front page it states it is Proprietary and Confidential.

Next time we will do some openflow or P4 coding. Don't ask me to tell which one will be because I have not decided yet. Brain hurts!

What about the simulator? Maybe one day.

Friday, May 03, 2019

Passing a Network card to a KVM vm guest because we are too lazy to configure SR-IOV

This can be taken as a generic how-to about passing PCIe cards to a mv guest. I will

Why

I can come up with a lot of excuses. The bottom line is you want the vm guest to do something with the card the vm host can't or shouldn't. For instance, what if we want to give a wireless card for a given vm guest? And the card is not supported by the vm host (I am looking at you, VMWare ESXi) or the vm host does not know how to virtualize it in a meaningful way?

Note: What we are talking about here should work with any PCI/PCIe card, but we said we will be talking about network cards, so there.

The Card

The card is a PCIe network card; for this article it should be seen as a garden-variety network card. You probably will not let me leave at that so, here is the info on the specific card I will be using in this article: it is a Netronome Agilio CX 2x10GbE (the one in the picture is a CX 1x40GbE, which I happen to own hence the crappy picture), which is built around their NFP-4000 flow processor. Basic informercial on it can be found at https://www.netronome.com/m/documents/PB_Agilio_CX_2x10GbE.pdf (it used to be https://www.netronome.com/media/documents/PB_Agilio_CX_2x10GbE.pdf, but I guess they thought media was too long a word. It also means that sometime after this article is posted the link will change again; no point on making them orange links). It is supposed to do things like KVM hypervisor support (SR-IOV comes to mind) right out of the box, so why we would want to passthrough the entire card to a vm guest? Here are some reasons:

  • What if the card can do something and the VM abstraction layers do not expose that?
  • What if we want to program the card to do our bidding?
  • What if we want to change the firmware of the card? Some cards allow you to upgrade the firmware, or change it completely to use it for other thingies (the Netronome card in question fits this second option, details about that might be discussed in a future article).
  • Why did you pick this card? Hey, this is not a reason to pass the entire card, but I will answer it anyway: because I have a box with 3 of them I was going to use for something else (we may talk about that in a future article). With that said, I avoided going over any of the special sauce this card has. For the purpose of this article, it is just a PCIe card I want to give to a vm guest.

How

Finding the card

Ok, card is inserted into the vm host, which booted properly. Now what? Well, we need to find where the card is so we can tell our guests. Most Linux distros come with lspci, which probulates the PCI bus. The trick is to search for the right pattern. Let's for instance look for network devices in one of my ESXi nodes:

[root@vmhost2:~] lspci | grep 'Network'
0000:00:19.0 Network controller: Intel Corporation 82579LM Gigabit Network Connection [vmnic0]
0000:04:00.0 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic1]
0000:04:00.1 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic2]
0000:05:00.0 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic3]
0000:05:00.1 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic4]
[root@vmhost2:~]

Notes

  1. ESXi is really not Linux but freebsd with gnu packages sprinkled over
  2. I just mentioned ESXi here because I needed another system I could run lscpi on.
  3. The lscpi options in ESXi are not as extensive as in garden-variety Linux. But, it is good enough to show it in action.
  4. If we had searched for Intel Corporation we would get much much more replies including the CPU itself. So, taking the time to get the right search string pays off.

If we were going to probulate in a Linux host, Ethernet works better than Network as the search pattern. We can even look at virtual interfaces KVM is feeding to a vm guest:

theuser@desktop1:~$ lspci |grep Ethernet
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:06.0 Ethernet controller: Red Hat, Inc. Virtio network device
theuser@desktop1:~$

Note that the 0000: is assumed. A very useful option available in the Linux version of lspci but not the ESXi one is -nn:

theuser@desktop1:~$ lspci -nn |grep Ethernet
00:03.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
00:06.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
theuser@desktop1:~$

The [1af4:1000] means [vendor_id:product_id]; remember it well.

For the Netronome cards we can just look for netronome since there should be no other devices matching that name besides the cards made by them:

raub@vmhost ~$ sudo lspci -nn|grep -i netronome
11:00.0 Ethernet controller [0200]: Netronome Systems, Inc. Device [19ee:4000]
raub@vmhost ~$

The card's PCI address is 11:00.0

Handing out the card to the guest

Two things we need to do when passing a PCI device to a vm guest (a.k.a. desktop1 in this example):

  1. Tell the vm host to keep its hands off it. The reason is that, in the case of a network card, it might want to configure it, creating interfaces (in the /dev/ directory tree) which either the host server (vmhost) can use for its own nefarious uses or so KVM can then virtualize (as a Virtio network device or some other emulation) to hand out to the guests. Since we want to use said card for our personal private personal nefarious purposes within a specific vm guest (desktop1), we are not going to be nice and share it.

    So we need to tell vmhost to leave it alone.

    • KVM knows it exists because it can look in the PCI chain by itself:
      [root@vmhost ~]# virsh nodedev-list | grep pci_0000_11
      pci_0000_11_00_0
      [root@vmhost ~]#
    • So now we can tell vmhost to leave pci-0000:11:00.0 alone:

      [root@vmhost ~]$ sudo virsh nodedev-dettach pci_0000_11_00_0
      Device pci_0000_11_00_0 detached
      
      [root@vmhost ~]$
  2. Tell the vm guest there is this shiny card it can lay its noodly appendages on.
    1. Shut the vm guest down.
    2. Edit the desktop.
      virsh edit desktop
    3. Add something like
      <hostdev mode='subsystem' type='pci' managed='yes'>
            <source>
                <address domain='0x0000' bus='0x11' slot='0x00' function='0x0'/>
            </source>
          </hostdev>
      to the end of the devices session. When you save it, it will properly place
      and configure the entry.
    4. Restart vm guest check if it can see the card using dmesg (Ubuntu 19.04 example. Note it is being listed as pci-0000:04:00.0 inside the vm guest). I expect to see something like

      [    7.348276] Netronome NFP CPP API
      [    7.352347] nfp-net-vnic: NFP vNIC driver, Copyright (C) 2010-2015 Netronome Systems
      [    7.361865] nfp 0000:04:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe
      [    7.372133] nfp 0000:04:00.0: RESERVED BARs: 0.0: General/MSI-X SRAM, 0.1: PCIe XPB/MSI-X PBA, 0.4: Explicit0, 0.5: Explicit1, free: 20/24
      [    7.396094] nfp 0000:11:00.0: Model: 0x40010010, SN: 00:15:4d:13:5d:58, Ifc: 0x10ff

      But what I am getting is something more like this:

      [    1.768683] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
      [    1.773014] nfp 0000:00:07.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe
      [    1.774066] nfp 0000:00:07.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
      [    1.775212] nfp 0000:00:07.0: can't find PCIe Serial Number Capability
      [    1.776252] nfp 0000:00:07.0: Interface type 15 is not the expected 1
      [    1.777285] nfp 0000:00:07.0: NFP6000 PCI setup failed

      What is going on? The answer to that is the next topic. You see,

PCIe is more demanding

Do you remember the can't find PCIe Serial Number Capability message? This is a PCIe card, meaning we need to setup the vm guest machine type to q35, which supports the ICH9 chipset which can handle a PCIe bus. The default (I440FX) can only do PCI bus. QEMU has a nice description on the difference. So, let's give it a try by recreating the KVM guest:

virt-install \
   --name desktop1 \
   --disk path=/home/raub/desktop1.qcow2,format=qcow2,size=10 \
   --ram 4098 --vcpus 2 \
   --cdrom /export/public/ISOs/Linux/ubuntu/ubuntu-16.04.5-server-amd64.iso  \
   --os-type linux --os-variant ubuntu19.04 \
   --network network=default \
   --graphics vnc --noautoconsole \
   --machine=q35 \
   --arch x86_64

When we try to build that vm guest, we get an error message stating that

ERROR    No domains available for virt type 'hvm', arch 'x86_64', machine type 'q35'

What now? You see, at the time I wrote this, the CentOS KVM package does not support q35 out of the box. We need more packages!

yum install centos-release-qemu-ev
yum update
reboot

And we try again, this time when we login to the guest, desktop1, it looks more promising (note the PCI address changed to 0000:01:00.0; this is a new vm guest):

theuser@desktop1:~$ dmesg |grep -i netro
[    1.922051] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[    1.954196] nfp 0000:01:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe
[    2.239018] nfp 0000:01:00.0: nfp:   netronome/serial-00-15-4d-13-5d-46-10-ff.nffw: not found
[    2.239059] nfp 0000:01:00.0: nfp:   netronome/pci-0000:01:00.0.nffw: not found
[    2.239913] nfp 0000:01:00.0: nfp:   netronome/nic_AMDA0096-0001_2x10.nffw: found, loading...
[   11.954477] nfp 0000:01:00.0 eth0: Netronome NFP-6xxx Netdev: TxQs=2/32 RxQs=2/32
[   11.971175] nfp 0000:01:00.0 eth1: Netronome NFP-6xxx Netdev: TxQs=2/31 RxQs=2/31
theuser@desktop1:~$

Which then becomes

theuser@desktop1:~$ ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d4:9e:50 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.105/24 brd 192.168.122.255 scope global dynamic enp0s3
       valid_lft 3489sec preferred_lft 3489sec
    inet6 fe80::5054:ff:fed4:9e50/64 scope link
       valid_lft forever preferred_lft forever
3: enp1s0np0:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:15:4d:13:5d:47 brd ff:ff:ff:ff:ff:ff
4: enp1s0np1:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:15:4d:13:5d:48 brd ff:ff:ff:ff:ff:ff
theuser@desktop1:~$

And now we can do something useful with it.

References

  • https://stackoverflow.com/questions/14061840/kvm-and-libvirt-wrong-cpu-type-in-virtual-host
  • https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-kvm_guest_virtual_machine_compatibility-supported_cpu_models
  • https://github.com/libvirt/libvirt/blob/v4.0.0/src/util/virarch.c#L37

Tuesday, March 07, 2017

Setting up zabbix using official instructions and repo: Step 1 we ain't there yet

So I am installing Zabbix. Why, well, you probably know. If not, we can talk about that in a different article. Yes I am testing my ansible playbook in a docker container, but right now that too is not important. The How I did It will be in a different article. This article is the Everything That Went Wrong and How I Got Around That one. Think of it as an insight of how I deal with me being clueless; laughing at my expense is acceptable and maybe even recommended.

I want to install latest version of Zabbix in a CentOS 7 host, as a result I will be using the official zabbix 3.2 install docs, which are the most current when I wrote this article. For now I will be lazy and use the mysql version since it is faster to setup; we can revisit that later.

Dependencies

  1. Need the repo. Per the official Zabbix instructions, I am using the official Zabbix repo, which as of the time of this writing can be obtained by

    rpm -ivh http://repo.zabbix.com/zabbix/3.2/rhel/7/x86_64/zabbix-release-3.2-1.el7.noarch.rpm

    I did write a script to get the latest rpm, but it is not important right now. Now, if you are curious, here is the repo config file:

    [root@zabbix ~]# cat /etc/yum.repos.d/zabbix.repo 
    [zabbix]
    name=Zabbix Official Repository - $basearch
    baseurl=http://repo.zabbix.com/zabbix/3.2/rhel/7/$basearch/
    enabled=1
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
    
    [zabbix-non-supported]
    name=Zabbix Official Repository non-supported - $basearch 
    baseurl=http://repo.zabbix.com/non-supported/rhel/7/$basearch/
    enabled=0
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX
    gpgcheck=1
    [root@zabbix ~]# 

    Before you ask, I am making a point to accidentally post it here for a reason, which will become clearer later.

  2. The database server. Thanks to irc user leManu enlightening me, we are not supposed to install mysql (or whatever db) server on the machine that will run zabbix server. With that said, the line

    mysql> grant all privileges on zabbix.* to zabbix@localhost identified by '';

    in the official docs has a very localhost feel to it.

    We better build the db server first, and then go there and create the zabbix user, tying it to the IP for the zabbix server. I used mariadb and then grabbed the required -- FQDN, port, zabbix password -- info and came back to the zabbix server.

  3. Packages. We have the repo setup, and database server info on standby. We might as well start installing zabbix itself, right?

    [root@zabbix ~]# yum install zabbix-server-mysql zabbix-web-mysql 
    [...]
    --> Finished Dependency Resolution
    Error: Package: zabbix-server-mysql-3.2.4-2.el7.x86_64 (zabbix)
               Requires: fping
    Error: Package: zabbix-server-mysql-3.2.4-2.el7.x86_64 (zabbix)
               Requires: libiksemel.so.3()(64bit)
     You could try using --skip-broken to work around the problem
     You could try running: rpm -Va --nofiles --nodigest
    [root@zabbix ~]# 

    Bummer. Why didn't it grab them from the normal centos repo? I guess maybe it does not have them and we will need to fetch them from another repo. But, before we add another repo, do you remember the file /etc/yum.repos.d/zabbix.repo, whose contents we pasted earlier? It has a zabbix-non-supported; how about if we take a quick look there?

    [root@zabbix ~]# yum whatprovides */fping --enablerepo=zabbix-non-supported
    Loaded plugins: fastestmirror, ovl
    Loading mirror speeds from cached hostfile
     * base: mirrors.gigenet.com
     * extras: mirror.keystealth.org
     * updates: mirror.umd.edu
    fping-3.10-1.el7.x86_64 : Scriptable, parallelized ping-like utility
    Repo        : zabbix-non-supported
    Matched from:
    Filename    : /usr/sbin/fping
    
    
    
    fping-3.10-1.el7.x86_64 : Scriptable, parallelized ping-like utility
    Repo        : @zabbix-non-supported
    Matched from:
    Filename    : /usr/sbin/fping
    
    
    
    [root@zabbix ~]# 

    Short version: grab the two packages we need from it already:

    yum install fping iksemel --enablerepo=zabbix-non-supported
  4. Missing setup file (thanks Yum!). Per the docs, we are now supposed to grab a file called /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz and use it to initially populate the zabbix database. Thing is, I can't find that (/usr/share/doc/zabbix-server-mysql-3.2.4) directory, much less the file:

    [root@zabbix ~]# ls  /usr/share/doc/
    coreutils-8.22    pam-1.1.8             python-pycurl-7.19.0
    gnupg2-2.0.22     pygpgme-0.3           unixODBC-2.3.1
    krb5-libs-1.13.2  python-kitchen-1.1.1  zabbix-release-3.2
    [root@zabbix ~]# 

    Maybe /usr/share/doc/zabbix-release-3.2/ is the directory and the docs were off? I will have to expertly crush your hopes:

    [root@zabbix ~]# ls  /usr/share/doc/zabbix-release-3.2/
    GPL
    [root@zabbix ~]# ls -l  /usr/share/doc/zabbix-release-3.2/
    total 20
    -rw-r--r-- 1 root root 18385 Feb 15  2016 GPL
    [root@zabbix ~]# head -10  /usr/share/doc/zabbix-release-3.2/GPL 
    *****************************************************************************
    The following copyright applies to the Red Hat Linux compilation and any 
    portions of Red Hat Linux it does not conflict with. Whenever this
    policy does conflict with the copyright of any individual portion of Red Hat 
    Linux, it does not apply.
    
    *****************************************************************************
    
                        GNU GENERAL PUBLIC LICENSE
                           Version 2, June 1991
    [root@zabbix ~]# 

    Maybe it is somewhere else? Nope.

    [root@zabbix ~]# find / -name create.sql.gz -print
    [root@zabbix ~]# 

    So, where's it? Hey, don't look at me like that. I too have no idea. Let's grab the rpm and then take a look at it:

    root@zabbix:/tmp$ rpm -qlp zabbix-server-mysql-3.2.4-2.el7.x86_64.rpm | grep cre
    ate.sql.gz 
    warning: zabbix-server-mysql-3.2.4-2.el7.x86_64.rpm: Header V4 DSA/SHA1 Signatur
    e, key ID 79ea5ed4: NOKEY
    /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz
    root@zabbix:/tmp$ 

    That is the version we installed, right?

    [root@zabbix ~]# rpm -q zabbix-server-mysql
    zabbix-server-mysql-3.2.4-2.el7.x86_64
    [root@zabbix ~]#

    Looks like it. And yum's log, /var/log/yum.log file think so too:

    Mar 07 14:30:20 Installed: zabbix-web-mysql-3.2.4-2.el7.noarch
    Mar 07 14:30:21 Installed: zabbix-web-3.2.4-2.el7.noarch
    Mar 07 14:30:22 Installed: zabbix-server-mysql-3.2.4-2.el7.x86_64

    This really does not make sense. Let me look again at the contents of the installed package, not at the rpm:

    [root@zabbix ~]# rpm -qlv zabbix-server-mysql
    -rw-r--r--    1 root    root                      132 Mar  2 14:55 /etc/logrotate.d/zabbix-server
    -rw-r-----    1 root    zabbix                  14876 Mar  2 14:55 /etc/zabbix/zabbix_server.conf
    -rw-r--r--    1 root    root                      415 Mar  2 14:29 /usr/lib/systemd/system/zabbix-server.service
    -rw-r--r--    1 root    root                       35 Mar  2 14:29 /usr/lib/tmpfiles.d/zabbix-server.conf
    drwxr-xr-x    2 root    root                        0 Mar  2 14:55 /usr/lib/zabbix/alertscripts
    drwxr-xr-x    2 root    root                        0 Mar  2 14:55 /usr/lib/zabbix/externalscripts
    -rwxr-xr-x    1 root    root                  2220064 Mar  2 14:55 /usr/sbin/zabbix_server_mysql
    drwxr-xr-x    2 root    root                        0 Mar  2 14:55 /usr/share/doc/zabbix-server-mysql-3.2.4
    -rw-r--r--    1 root    root                       98 Feb 27 09:22 /usr/share/doc/zabbix-server-mysql-3.2.4/AUTHORS
    -rw-r--r--    1 root    root                    17990 Feb 27 09:23 /usr/share/doc/zabbix-server-mysql-3.2.4/COPYING
    -rw-r--r--    1 root    root                   742520 Feb 27 09:22 /usr/share/doc/zabbix-server-mysql-3.2.4/ChangeLog
    -rw-r--r--    1 root    root                       52 Feb 27 09:24 /usr/share/doc/zabbix-server-mysql-3.2.4/NEWS
    -rw-r--r--    1 root    root                      188 Feb 27 09:22 /usr/share/doc/zabbix-server-mysql-3.2.4/README
    -rw-r--r--    1 root    root                  1161488 Mar  2 14:49 /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz
    -rw-r--r--    1 root    root                      881 Mar  2 14:55 /usr/share/man/man8/zabbix_server.8.gz
    drwxr-xr-x    2 zabbix  zabbix                      0 Mar  2 14:55 /var/log/zabbix
    drwxr-xr-x    2 zabbix  zabbix                      0 Mar  2 14:55 /var/run/zabbix
    [root@zabbix ~]# ls /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz
    ls: cannot access /usr/share/doc/zabbix-server-mysql-3.2.4/create.sql.gz: No such file or directory
    [root@zabbix ~]# 

    It turns out (kudos to irc user TrevorH for pointing that out) that yum is configured not to install docs

    [root@zabbix ~]# grep -ir tsflags /etc/yum.*
    /etc/yum.conf:tsflags=nodocs
    [root@zabbix ~]# 

    Let's comment it out then and try again

    [root@zabbix ~]# sed -i -e 's/^tsflags=nodocs/#tsflags=nodocs/' /etc/yum.conf
    [root@zabbix ~]# yum reinstall zabbix-server-mysql zabbix-web-mysql --enablerepo=zabbix
    Loaded plugins: fastestmirror, ovl
    Loading mirror speeds from cached hostfile
     * base: dist1.800hosting.com
     * extras: mirror.eboundhost.com
     * updates: mirror.es.its.nyu.edu
    Resolving Dependencies
    --> Running transaction check
    ---> Package zabbix-server-mysql.x86_64 0:3.2.4-2.el7 will be reinstalled
    ---> Package zabbix-web-mysql.noarch 0:3.2.4-2.el7 will be reinstalled
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================
     Package                   Arch         Version              Repository    Size
    ================================================================================
    Reinstalling:
     zabbix-server-mysql       x86_64       3.2.4-2.el7          zabbix       1.8 M
     zabbix-web-mysql          noarch       3.2.4-2.el7          zabbix       5.1 k
    
    Transaction Summary
    ================================================================================
    Reinstall  2 Packages
    
    Total download size: 1.8 M
    Installed size: 4.0 M
    Is this ok [y/d/N]: y
    Downloading packages:
    (1/2): zabbix-web-mysql-3.2.4-2.el7.noarch.rpm             | 5.1 kB   00:00     
    (2/2): zabbix-server-mysql-3.2.4-2.el7.x86_64.rpm          | 1.8 MB   00:01     
    --------------------------------------------------------------------------------
    Total                                              1.7 MB/s | 1.8 MB  00:01     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : zabbix-web-mysql-3.2.4-2.el7.noarch                          1/2 
      Installing : zabbix-server-mysql-3.2.4-2.el7.x86_64                       2/2 
      Verifying  : zabbix-server-mysql-3.2.4-2.el7.x86_64                       1/2 
      Verifying  : zabbix-web-mysql-3.2.4-2.el7.noarch                          2/2 
    
    Installed:
      zabbix-server-mysql.x86_64 0:3.2.4-2.el7                                      
      zabbix-web-mysql.noarch 0:3.2.4-2.el7                                         
    
    Complete!
    [root@zabbix ~]# ls /usr/share/doc/
    coreutils-8.22    pygpgme-0.3           zabbix-release-3.2
    gnupg2-2.0.22     python-kitchen-1.1.1  zabbix-server-mysql-3.2.4
    krb5-libs-1.13.2  python-pycurl-7.19.0
    pam-1.1.8         unixODBC-2.3.1
    [root@zabbix ~]# ls /usr/share/doc/zabbix-server-mysql-3.2.4/
    AUTHORS  ChangeLog  COPYING  create.sql.gz  NEWS  README
    [root@zabbix ~]# 

    Success at last!

I think that is enough for one article. If you expect this to have any closure or redeeming message, I have news for you sunshine. Just hope that the next zabbix article will talk about actually getting it installed and configured and running. But, I make no guarantees.

Sunday, March 05, 2017

Checking if you are running redhat or centos or ubuntu or neither

So I wanted to make a script that would behave differently if we are running RedHat, CentOS, or Ubuntu. The findings here probably can be applied to other distros, but we need to start somewhere.

  1. Using lsb_release. I have been told before that the proper way to detect the OS/distro version is to use lsb_release. So, something like

    distro=$(lsb_release -i | awk '{ print $3}' | tr 'A-Z' 'a-z')

    Should do the trick. Of course that would imply it is installed, which might not be the case depending on how barebones is your install (less is more in my book). So, for our next trick, let's assume we do not have it installed.

  2. Without lsb_release. It might come as a shock to some but it is possible to find a linux install without it... and also without word processor and games and even web browsers. Like in servers. How would we find out which distro we have?

    1. RedHat and derivatives have the /etc/redhat-release file. It is easy to say if it is redhat or centos because it is written in the file itself.

      distro=$([ -f /etc/redhat-release ] && echo rhel )
      distro=$(grep -qi "redhat" /etc/redhat-release && echo rhel || echo centos )

      But Ubuntu does not have that file. Back to the drawing board.

    2. uname -v works on ubuntu

      raub@desktop:/tmp$ uname -v
      #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017
      raub@desktop:/tmp$

      But not on centos or redhat

      [raub@vmguest ~]$ uname -v
      #1 SMP Tue Aug 23 19:58:13 UTC 2016
      [raub@vmguest ~]$

      Come on! We can do better than that!

    3. /etc/issue seems to have the most potential

      [raub@vmguest ~]$ cat /etc/issue
      CentOS release 6.8 (Final)
      Kernel \r on an \m
      
      [raub@vmguest ~]$

      and on ubuntu

      raub@desktop:/tmp$ cat /etc/issue
      Ubuntu 16.04.1 LTS \n \l
      
      raub@desktop:/tmp$

      I think we got our winner

Thursday, November 03, 2016

Yum Manually and multiple repos

Quick post (I hope) about something I learned today that has a bit of a Captain Obvious taste to it. But I thought some people might find that amusing... even if it is at my expense.

Like many who use Red Hat products (CentOS comes to mind) and derived distros, I use repos outside the official ones. Because those repos tend to have newer versions of some packages, I try to be careful about only upgrading the packages that are required by the package I added said repo to my list. Short answer is the Law of Unintended Consequences. Long version is that I expect that people building the official packages being quite careful about compatibility and security. So, I should only reach out to the different repos after I found out the official packages do not do what I need.

What I have been doing, and I do not claim it is the best solution, is to install and then disable the non-official repos so if I want something from them I have to specifically ask for it. So, if I want to use the remi repo, I would first disable it

sed -i -e 's/^enabled=1/enabled=0/' /etc/yum.repos.d/remi.repo

and then specifically ask for it to install, say, php

yum install php --enablerepo=remi

which would install the latest PHP version that remi has. On a side note, if you had to install php 5.6 from remi, you would use remi-php56. But, what about upgrading them? After all, yum check-update and yum update by default will not check the disabled repos even if you have packages installed from them. So, you have to use --enablerepo. Now, what I learned today is that you can just list all the repos you are using by separating them with commas.

Let me show you in action: at first we think there are no updates:

raub@pickles ~]$ sudo yum check-update
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-
              : manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
[raub@pickles ~]$

Now, let's list all the repos we have been using with this machine and see what it tells us:

raub@pickles ~]$ sudo yum check-update --enablerepo=remi-php56,epel,secu
rity_shibboleth
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-
              : manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
security_shibboleth                                      | 1.2 kB     00:00
security_shibboleth/primary                                |  15 kB   00:00
security_shibboleth                                                       96/96

libcurl-openssl.x86_64                7.51.0-2.1             security_shibboleth
opensaml-schemas.x86_64               2.6.0-1.1              security_shibboleth
shibboleth.x86_64                     2.6.0-2.1              security_shibboleth
xmltooling-schemas.x86_64             1.6.0-1.1              security_shibboleth
[raub@pickles ~]$

As you can see, we do need to update shibboleth and its dependencies! And update we shall:

[raub@pickles ~]$ sudo yum update --enablerepo=remi-php56,epel,security_shibboleth
[...]
[raub@pickles ~]$ 

Kinda neat, eh?

Friday, July 29, 2016

Checking if RedHat/CentOS has new updates

If you use Debian Linux derivatives, specially Ubuntu, you probably noticed when you login it will tell you (using the MOTD) that there are new updates waiting for you. And, you can use that if, say, you are writing a script to let you know about that; someone I know use that to monitor his Ubunu boxes in Nagios. But, what about in RedHat and derviatives the lazy way?

Let's do some thinking aloud and see if we can come up with something. We know that if we run yum check-update, it should reply with the list of packages needing to be upgraded if any

[root@vmhost ~][raub@duckwitch ~]$ yum check-update
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.supremebytes.com
 * extras: mirror.hostduplex.com
 * updates: mirror.scalabledns.com

chkconfig.x86_64                        1.3.61-5.el7_2.1                 updates
device-mapper.x86_64                    7:1.02.107-5.el7_2.5             updates
device-mapper-libs.x86_64               7:1.02.107-5.el7_2.5             updates
dracut.x86_64                           033-360.el7_2.1                  updates
glibc.x86_64                            2.17-106.el7_2.6                 updates
glibc-common.x86_64                     2.17-106.el7_2.6                 updates
iproute.x86_64                          3.10.0-54.el7_2.1                updates
kernel.x86_64                           3.10.0-327.22.2.el7              updates
kpartx.x86_64                           0.4.9-85.el7_2.5                 updates
libxml2.x86_64                          2.9.1-6.el7_2.3                  updates
ntpdate.x86_64                          4.2.6p5-22.el7.centos.2          updates
pcre.x86_64                             8.32-15.el7_2.1                  updates
selinux-policy.noarch                   3.13.1-60.el7_2.7                updates
selinux-policy-targeted.noarch          3.13.1-60.el7_2.7                updates
systemd.x86_64                          219-19.el7_2.11                  updates
systemd-libs.x86_64                     219-19.el7_2.11                  updates
systemd-python.x86_64                   219-19.el7_2.11                  updates
systemd-sysv.x86_64                     219-19.el7_2.11                  updates
tzdata.noarch                           2016f-1.el7                      updates
[raub@duckwitch ~]$

As you can see, it does not require you to run that command as root. And you can even check if a repo you use but configured to be normally disabled, like epel in the following example:

raub@duckwitch ~]$ yum check-update --enablerepo=epel
Loaded plugins: fastestmirror
epel/x86_64/metalink                                     |  11 kB     00:00
epel                                                     | 4.3 kB     00:00
(1/3): epel/x86_64/group_gz                                | 170 kB   00:00
(2/3): epel/x86_64/updateinfo                              | 584 kB   00:00
(3/3): epel/x86_64/primary_db                              | 4.2 MB   00:00
Loading mirror speeds from cached hostfile
 * base: mirror.supremebytes.com
 * epel: mirror.chpc.utah.edu
 * extras: mirror.hostduplex.com
 * updates: mirror.scalabledns.com

chkconfig.x86_64                        1.3.61-5.el7_2.1                 updates
device-mapper.x86_64                    7:1.02.107-5.el7_2.5             updates
device-mapper-libs.x86_64               7:1.02.107-5.el7_2.5             updates
dracut.x86_64                           033-360.el7_2.1                  updates
epel-release.noarch                     7-7                              epel
glibc.x86_64                            2.17-106.el7_2.6                 updates
glibc-common.x86_64                     2.17-106.el7_2.6                 updates
iproute.x86_64                          3.10.0-54.el7_2.1                updates
kernel.x86_64                           3.10.0-327.22.2.el7              updates
kpartx.x86_64                           0.4.9-85.el7_2.5                 updates
libxml2.x86_64                          2.9.1-6.el7_2.3                  updates
ntpdate.x86_64                          4.2.6p5-22.el7.centos.2          updates
pcre.x86_64                             8.32-15.el7_2.1                  updates
selinux-policy.noarch                   3.13.1-60.el7_2.7                updates
selinux-policy-targeted.noarch          3.13.1-60.el7_2.7                updates
systemd.x86_64                          219-19.el7_2.11                  updates
systemd-libs.x86_64                     219-19.el7_2.11                  updates
systemd-python.x86_64                   219-19.el7_2.11                  updates
systemd-sysv.x86_64                     219-19.el7_2.11                  updates
tzdata.noarch                           2016f-1.el7                      updates
[raub@duckwitch ~]$

What if there are no upgrades?

[root@server1 ~]# yum check-update
Loaded plugins: fastestmirror
base                                                     | 3.6 kB     00:00
extras                                                   | 3.4 kB     00:00
updates                                                  | 3.4 kB     00:00
Loading mirror speeds from cached hostfile
 * base: mirror.us.leaseweb.net
 * extras: mirror.us.leaseweb.net
 * updates: reflector.westga.edu
[root@server1 ~]#

Sounds like we need to check for the first blank line. We can do that using sed. We can find the blank line by using the /^\s*$/ search pattern in sed. So we could start with something like (the -n is there because we only care when we find said blank line)

yum check-update | `sed -n '/^\s*$/p'`

which if you run does not seem to do much. Reason is that what we really want is to know if the match was successful or not. And, sed actually has the answer: look at these entries I stole from its man pange:

q [exit-code]
              Immediately  quit  the  sed  script  without processing any more
              input, except that if auto-print is  not  disabled  the  current
              pattern  space will be printed.  The exit code argument is a GNU
              extension.

       Q [exit-code]
              Immediately quit the sed  script  without  processing  any  more
              input.  This is a GNU extension.

Let's try it then. First we will find a machine that has a package that needs to be updated. In this case it will be my KVM server, which you have met before when we talked about USB passthrough:

[raub@vmhost ~]$ yum check-update 
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: mirror.vcu.edu
 * extras: centos.mirror.constant.com
 * updates: mirror.vcu.edu

samba4-libs.x86_64                    4.2.10-7.el6_8                     updates
[raub@vmhost ~]$

So as of this writing it has one update. We then try our one-liner, which we want to return 0 if it did not find any pending upgrades and 1 if it did. And that is done by adding q1 past the search string, as in /^\s*$/q1, which means write 1 if search successful. Of course we now need to print the result, which can be done with echo $?. So, let's try it:

[raub@vmhost ~]$ yum check-update | `sed -n '/^\s*$/q1'` ; echo $? 1 [raub@vmhost ~]$

It thinks that it found it. I am not completely sure it works, so we need also to verify it works as it should when we find no updates:

[root@server1 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.netdepot.com
 * extras: mirror.solarvps.com
 * updates: ftpmirror.your.org
[root@server1 ~]# yum check-update | `sed -n '/^\s*$/q1'` ; echo $?
0
[root@server1 ~]# 

Sounds like we have a winner. Now all that is left is to wrap something around it to something with that. I will leave that to you. Note that it does not differentiate between normal and security updates though.

Friday, April 29, 2016

My Introduction to firewallD

WARNING:

  1. When I say My Introduction, I do not mean I am trying to introduce you to firewallD. I really mean this is the first time I had to properly deal with it. I wish this story would involve robots, but it is not that exciting

  2. You should not follow this blindly as a guide. I screwed it up and documented it here. Read it first and see what I did wrong and why.

So I am building an apache webserver on a Linux box (Red Hat Enterprise Server 7.2 if you are curious, but there is nothing stopping the gist of my story working on any Linux distro using systemd and, more specifically, firewallD) and want to limit the number of hosts that can access it to one, which happens to have 192.168.79.13 as its IP.

In the golden days...

If we were to use dear ol' pedestrian iptables, one way to create the rules would be

iptables -I INPUT -s '192.168.79.13/32' -m state --state NEW \
   -m tcp -p tcp --dport 80 -j ACCEPT
iptables -I INPUT -s '192.168.79.13/32' -m state --state NEW \
   -m tcp -p tcp --dport 443 -j ACCEPT

and then check if it has been loaded using

iptables -L -n

At this point we might have decided we really should have also told it to apply the rule to only a specific interface or IP (we might be listening to many IPs on the same interface). So we would adjust it as needed until we are satisfied. And, once satisfied it works, save it using

service iptables save

Modern Times

So far so good. But in Red Hat 7 and CentOS 7 as a result, iptables save is no longer an available command. firewallD should be used instead, which is amusing since it is but a front end to iptables. I guess the point is that if iptables is replaced you would still be able to use firewallD to control whatever comes next; in other words, it is an abstraction layer.

But, I digress. Bottom line is we need to figure out how to use firewallD. First thing is that we will be using firewall-cmd to talk to it. Next is that it allows to create separate firewall zones (same idea as the Windows firewall zones for the Microsoft crowd amongst you) which define the interface and possibly IPs it is associated with and which services and rules are supposed to do what there. And, yes, that last sentence was a mouthful, so we shall begin by seeing which zone we are using.

[user@webtest httpd]$ sudo firewall-cmd --get-default-zone
public
[user@webtest httpd]$

I guess that makes sense since this is a pretty much out-of-the-box install; we are still customizing it. But, which zones are in use?

[user@webtest httpd]$ sudo firewall-cmd --get-active-zones
public
  interfaces: eth0
[user@webtest httpd]$

So, the only active zone is the default one, which is being applied to eth0. What can you tell me about the public zone?

[user@webtest httpd]$ sudo firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

[user@webtest httpd]$

I knew about ssh service (sshd) -- that is how I am connected to it -- but I did not realize it actually has a dhcpv6 client. How about ports in use?

[user@webtest httpd]$ sudo firewall-cmd --zone=public --list-ports
[user@webtest httpd]$

Hmmm, nothing? But sshd uses port 22 by default (which is how it is set right now); shouldn't it show up? Well, like Windows' firewall you can specify a service or just a port. If you do the former, its port is not listed. I wonder if it is just satisfied the ssh service is on and relies on it to know which port it is using; this sounds like something worthwhile to test later. Talking about services, which ones firewallD knows of, be them running or not?

[user@webtest httpd]$ sudo firewall-cmd --get-services
RH-Satellite-6 amanda-client bacula bacula-client dhcp dhcpv6 dhcpv6-client 
dns freeipa-ldap freeipa-ldaps freeipa-replication ftp high-availability http 
https imaps ipp ipp-client ipsec iscsi-target kerberos kpasswd ldap ldaps 
libvirt libvirt-tls mdns mountd ms-wbt mysql nfs ntp openvpn pmcd pmproxy 
pmwebapi pmwebapis pop3s postgresql proxy-dhcp radius rpc-bind rsyncd samba 
samba-client smtp ssh telnet tftp tftp-client transmission-client vdsm 
vnc-server wbem-https
[user@webtest httpd]$

It even knows about telnet! Not that it is even installed here, which means firewallD has some kind of configuration file somewhere listing all the services it should be aware of. Which means you can add/subtract services to this list... or it ends up overwritten during an upgrade. One of the two, right?

So we created the firewall zone. How does it look like? It is actually a xml file; here's the one for the public zone as it stands right now in this conversation:

[user@webtest httpd]$ sudo cat /etc/firewalld/zones/public.xml
<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <description>For use in public areas. You do not trust the other 
computers on networks to not harm your computer. Only selected incoming 
connections are accepted.</description>
  <service name="dhcpv6-client"/>
  <service name="ssh"/>
</zone>
[user@webtest httpd]$

I think we have an idea of the beast.

Houston, we have a problem

So, let's say we want to allow access to our webserver on port 80. We can do something like

firewall-cmd --zone=public --add-port=80/tcp --permanent

or

firewall-cmd --zone=public --add-service=http --permanent

right? Er, not quite. Let me show what I mean: after we run one of those commands (I picked the service one) we should always check if the zone was updated.

[user@webtest httpd]$ sudo firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

[user@webtest httpd]$

But yet, it is in the file!

[user@webtest httpd]$ sudo cat /etc/firewalld/zones/public.xml
<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <description>For use in public areas. You do not trust the other 
computers on networks to not harm your computer. Only selected incoming 
connections are accepted.</description>
  <service name="dhcpv6-client"/>
  <service name="ssh"/>
  <service name="http"/>
</zone>
[user@webtest httpd]$

What is going on? Short version: user error. Long version: --permanent means save to file only. It does not affect the running version. If you want to turn it on, you need to restart firewalld.

[user@webtest httpd]$ sudo systemctl restart firewalld
[user@webtest httpd]$

Another alternative is not to use --permanent, as in

firewall-cmd --zone=public --add-service=http

That will commit the change immediately. If you reboot it is lost, so it is a good way to test it. And then once you are ready to commit, you can then do

firewall-cmd --runtime-to-permanent

which as the name implies saves the running firewall to the appropriate zone files. In any case, here is the outcome:

[user@webtest httpd]$ sudo firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client httpd ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

[user@webtest httpd]$

If you want to remove that very rule, you can do it by

firewall-cmd --zone=public --remove-service=httpd

Back to the iptables example

Do you remember it? No? Let me put it here

iptables -I INPUT -s '192.168.79.13/32' -m state --state NEW \
   -m tcp -p tcp --dport 80 -j ACCEPT
iptables -I INPUT -s '192.168.79.13/32' -m state --state NEW \
   -m tcp -p tcp --dport 443 -j ACCEPT

The key here is not we are specifying the ports, but that we want to specify the IP traffic is coming from. In the firewalld universe that requires a rich rule. Here is what I am using:

firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
    source address="192.168.79.13/32" service name="http" accept"
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
    source address="192.168.79.13/32" service name="https" accept"

it should behave the very same way as the iptables entries. But, let's see how it looks like in runtime

[user@webtest httpd]$ sudo firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:
        rule family="ipv4" source address="192.168.79.13/32" service 
name="https" accept
        rule family="ipv4" source address="192.168.79.13/32" service 
name="http" accept
[user@webtest httpd]$

And, after we are satisfied and committed it, in the zone file

[user@webtest httpd]$ sudo cat /etc/firewalld/zones/public.xml
<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <description>For use in public areas. You do not trust the other 
computers on networks to not harm your computer. Only selected incoming 
connections are accepted.</description>
  <service name="dhcpv6-client"/>
  <service name="ssh"/>
  <rule family="ipv4">
    <source address="192.168.79.13/32"/>
    <service name="https"/>
    <accept/>
  </rule>
  <rule family="ipv4">
    <source address="192.168.79.13/32"/>
    <service name="http"/>
    <accept/>
  </rule>
</zone>
[user@webtest httpd]$

References