Record of the UNIX Wars: shell

Showing posts with label shell. Show all posts

Tuesday, January 14, 2020

Updating only the latest/default Linux kernel boot arguments in CentOS/RedHat/Fedora

I think from the title you know what I have in QEMU: Checking for hardware virtualization QEMU: Checking if device /dev/kvm exists QEMU: Checking if device /dev/kvm is accessible QEMU: Checking if device /dev/vhost-net exists QEMU: Checking if device /dev/net/tun exists QEMU: Checking for cgroup 'memory' controller support QEMU: Checking for cgroup 'memory' controller mount-point QEMU: Checking for cgroup 'cpu' controller support QEMU: Checking for cgroup 'cpu' controller mount-point QEMU: Checking for cgroup 'cpuacct' controller support QEMU: Checking for cgroup 'cpuacct' controller mount-point QEMU: Checking for cgroup 'cpuset' controller support QEMU: Checking for cgroup 'cpuset' controller mount-point QEMU: Checking for cgroup 'devices' controller support QEMU: Checking for cgroup 'devices' controller mount-point QEMU: Checking for cgroup 'blkio' controller support QEMU: Checking for cgroup 'blkio' controller mount-point QEMU: Checking for device assignment IOMMU support QEMU: Checking if IOMMU is enabled by kernel appears to be disabled in kernel. Add intel_iommu=on [root@vmhost2 ~]#The official docs would mind. My direct application is adding intel_iommu=on to the kernel in the KVM server I built to replace my VMWare ESXi one:

[root@vmhost2 ~]# virt-host-validate : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : PASS : WARN (IOMMU to kernel cmdline arguments) state the right way to do it is to use grub-mkconfig (might require grub-install first):

echo 'GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"' >> /etc/default/grub
grub-mkconfig -o "$(readlink -f /etc/grub2.cfg)"

And then reboot. Problem with that is it applies intel_iommu=on to every single kernel listed in the grub menu. What if I just want to do that to one of the kernels (in my case the latest)? This way if something goes boink I can boot to the grub menu, select the last one, and continue booting.

Well, in previous CentOS version like 6 and 7, I would

Open the grub.cfg
Find the latest kernel menu entry (the top one)
Find the line that tells which kernel to load for that version

Append the option I wanted to add, say the intel_iommu=on from above:

linux16 /boot/vmlinuz-3.10.0-957.12.2.el7.x86_64
root=UUID=1a4cb560-eade-47cd-b1a5-57f8e0f53b8f ro console=tty0 crashkernel=auto
console=ttyS0,115200 intel_iommu=on

Reboot.

But I can no longer do that since I cannot find the lines identifying the path to their respective kernels.

Enter the Grubby

Yes, this is a Today I Learned (TIL) event, and as you might have guessed, we are talking about grubby (if you click on the link you will go to the Red Hat official github repo for it), a command line tool to edit the boot config. I usually try to avoid commands that are but wrappers hiding what is really going on, but this one does seem to be useful (given I am no longer able to just edit the config file) and does not seem to require a lot of extra packages. It also came with both the Fedora31 and CentOS8 installs I have done. However when I saw which distros have prebuilt packages, none of the Debian-derived (ubuntu, mint, etc) are listed. It seems Ubuntu prefers update-grub, which is a wrapper around grub2-mkconfig, meaning it updates all the listed kernels.

Let's see what we can break:

The man page says that to append arguments to a given kernel, we should run

grubby --update-kernel=the_kernel --args="kernel_args"

where the_kernel is the path to the kernel we want to edit. So, where are the kernel hiding and how to find out which one is the latest? For the first question, the kernel files are the ones starting with vmlinuz in the /boot directory:

[root@vmhost2 ~]# ls /boot/
config-4.18.0-80.11.2.el8_0.x86_64
config-4.18.0-80.el8.x86_64
efi
grub2
initramfs-0-rescue-133a53b45d2b47168497d47a34dd932f.img
initramfs-4.18.0-80.11.2.el8_0.x86_64.img
initramfs-4.18.0-80.11.2.el8_0.x86_64kdump.img
initramfs-4.18.0-80.el8.x86_64.img
initramfs-4.18.0-80.el8.x86_64kdump.img
loader
lost+found
System.map-4.18.0-80.11.2.el8_0.x86_64
System.map-4.18.0-80.el8.x86_64
vmlinuz-0-rescue-133a53b45d2b47168497d47a34dd932f
vmlinuz-4.18.0-80.11.2.el8_0.x86_64
vmlinuz-4.18.0-80.el8.x86_64
[root@vmhost2 ~]#

Get the path of the latest kernel. From the previous step we know where they are, but now we need a way to identify the lastest one. We could write a script... or find in the man page that grubby has an option, --default-kernel,
```
[root@vmhost2 ~]# grubby --default-kernel
/boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64
[root@vmhost2 ~]#
```
but is this default kernel the same as the latest one (currently in use)? Let's ask the host what is the current kernel (I did boot up with the latest one installed in vmhost2):
```
[root@vmhost2 ~]# uname -a
Linux vmhost2 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@vmhost2 ~]#
```
Looks to be the case.
Apply the above to update the kernel. Now we know how to identify the latest kernel, we can run
```
grubby --update-kernel=$(grubby --default-kernel) --args="kernel_args"
```
to add new arguments to the latest kernel. kernel_args is the space-separated list of the arguments we want to feed to it. The format should be the same you would feed to the kernel if booting msnuslly. For my vmhost2 kvm server, that would be intel_iommu=on. So,
```
grubby --update-kernel=$(grubby --default-kernel) --args="intel_iommu=on"
```
followed by a reboot should be just fine.
We can do it better. If we are only updating the latest/default kernel, passing DEFAULT as the_kernel does exactly what we want but with less effort from out part:
```
grubby --update-kernel DEFAULT --args="intel_iommu=on"
```

After we reboot, we can log back in and and see if it intel_iommu=on has been added:

[root@vmhost2 ~]# grubby --info DEFAULT
index=0
kernel="/boot/vmlinuz-4.18.0-80.11.2.el8_0.x86_64"
args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params "intel_iommu=on""
root="/dev/mapper/vmhost-root"
initrd="/boot/initramfs-4.18.0-80.11.2.el8_0.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-80.11.2.el8_0.x86_64) 8 (Core)"
id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.11.2.el8_0.x86_64"
[root@vmhost2 ~]#

Fine, but did it only change the latest kernel? Let's find out by picking another kernel and asking what's up (I have only two kernels listed so we pick the other one):

[root@vmhost2 ~]# grubby --info vmlinuz-4.18.0-80.el8.x86_64
index=1
kernel="/boot/vmlinuz-4.18.0-80.el8.x86_64"
args="ro crashkernel=auto rd.lvm.lv=vmhost/root rd.lvm.lv=vmhost/usr rhgb quiet $tuned_params"
root="/dev/mapper/vmhost-root"
initrd="/boot/initramfs-4.18.0-80.el8.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-80.el8.x86_64) 8 (Core)"
id="133a53b45d2b47168497d47a34dd932f-4.18.0-80.el8.x86_64"
[root@vmhost2 ~]#

And now kvm is happy since IOMMU is enabled:

[root@vmhost2 ~]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
[root@vmhost2 ~]#

Final thoughts

It seems they (don't you always wonder who "they" are?) are deprecating/phasing grubby out. And it is not available in Debian/Ubuntu/derivatives. So next time I play with kernel boot options, which will be soon, I will see about using a more generic solution.

Tuesday, November 12, 2019

Capturing the output of a command sent through ssh within script without it asking to verify host key

You have noticed when you connect to a new server, if its key is not in ~/.ssh/known_hosts ssh will ask you to verify the key:

raub@desktop:~$ ssh -i $SSHKEY_FILE cc@"$server_ip"
The authenticity of host 'headless.example.com (10.0.1.160)' can't be established.
ECDSA key fingerprint is SHA256:AgwYevnTsG2m9hQLu/ROp+Rjj5At2HU0HVoGZ+5Ug58.
Are you sure you want to continue connecting (yes/no)?

That is a bit of a drag if you want to run a script to connect to said server. Fortunately ssh has an option just for that, StrictHostKeyChecking, as mentioned in the man page:

ssh automatically maintains and checks a database containing identification for all hosts it has ever been used with. Host keys are stored in ~/.ssh/known_hosts in the user's home directory. Additionally, the file /etc/ssh/ssh_known_hosts is automatically checked for known hosts. Any new hosts are automatically added to the user's file. If a host's identification ever changes, ssh warns about this and disables password authentication to prevent server spoofing or man-in-the-middle attacks, which could otherwise be used to circumvent the encryption. The StrictHostKeyChecking option can be used to control logins to machines whose host key is not known or has changed.

Now we can rewrite the ssh command as

ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip"

and it will login without waiting for us to verify the key. Of course this should only be done when the balance between security and automation meets your comfort level. However, if we run the above from a script, it will connect and just stay there with the session open, not accept any commands. i.e. if we try to send a command, pwd in this case,

ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip"
pwd

it will never see the pwd. Only way to get to the pwd is to kill the ssh session, when it will then go to the next command but running it in the local host, not the remote one. If we want to run pwd in the host we ssh into, we need to pass it as an argument, i.e.

raub@desktop:/tmp$ pwd
/tmpraub@desktop:/tmp$ ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip" pwd
/home/raub
raub@desktop:/tmp$

which means it will connect, run said command, and then exit. But, how to do that from a script? The answer is to use eval or bash -c (or other available shell):

raub@desktop:/tmp$ moo="ssh -o \"StrictHostKeyChecking no\" -i $SSHKEY_FILE cc@$server_ip"
raub@desktop:/tmp$ dirlist=$(eval "$moo pwd")
raub@desktop:/tmp$ echo $dirlist
/home/raub
raub@desktop:/tmp$ dirlist2=$(sh -c "$moo pwd")
raub@desktop:/tmp$ echo $dirlist2
/home/raub
raub@desktop:/tmp$

Now, there are subtle differences between eval and bash -c. There is a good thread in stackexchange which explains them better than I can do.

Now why someone would want to do that?

How about doing some poor-man ansible, i.e. when host on the other side does not have python or has some other idiosyncrasy. Like my home switch, but that is another story...

Saturday, November 09, 2019

Creating a RSA key for rsync for Android using docker

I use Rsync4Android to backup my phone. It is convenient and you can even run a cronjob so it does its thing without your intervention (think of it backing up while you are having dinner somewhere) and it backs up to wherever you want, so you are in control. Now, one day it stopped working. Not knowing what was going on, I did what one usually does when dealing with ssh issues: run sshd in debug mode:

raub@desktop:~$ sudo /usr/sbin/sshd -D
[...]
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: inetd sockets after dupping: 3, 3
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user raub service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "raub"
debug1: PAM: setting PAM_RHOST to "10.0.0.129"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user raub service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]
Connection closed by 10.0.0.129 port 39739 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 10239
debug1: audit_event: unhandled event 12
raub@desktop:~$

The line that tells what is going on is

userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]

When creating a key pair, Rsync4Android uses DSA algorithm. As we know, DSA has been considered insecure for a while and the current releases of openssh do not support it by default. So, if I want to keep on using rsync4android, I either configure my ssh server to accept DSA keys or find a way to convince it to use a RSA key. I chose the RSA route, but how to do it?

Create Key

Nothing special here.

ssh-keygen -t rsa -b 4096 -C "Phone_backup" -f ~/.ssh/phonebackup

Don't like 4096 bits? Double it; Rsync4Android does not care.

Convert private key to something Rsync4Android can use

The public key will go to the Linux host, which runs openssh and can handle those keys just fine. Rsync4Android, on the other hand, needs dropbear style keys. As mentioned in Rsync4Android docs, the best way to convert is to use the dropbearconvert command, which for ubuntu comes in the dropbear package. As I did not want to install it in desktop, I quickly created a docker container, copied the private key, and then installed the package. And then ran it (note the path) telling I am feeding it an openssh format key and want a dropbear style key:

root@b3f7ed8c4f24:/home# apt-get install dropbear
[...]
root@b3f7ed8c4f24:/home# /usr/lib/dropbear/dropbearconvert openssh dropbear phonebackup phonebackup.dropbear
Key is a ssh-rsa key
Wrote key to 'phonebackup.dropbear'
root@b3f7ed8c4f24:/home#

Now I have the dropbear-formated private key, phonebackup.dropbear, I can finally set the circus up.

Copy public and private keys to the proper locations

Public key goes to the computer we are backing the phone to (in my case desktop): added to the account's ~/.ssh/athorized_keys file.

The private key to the android. I put it in the same directory Rsync4Android placed the original (DSA) key it created, /sdcard. Then it was a matter of renaming the key in the Rsync4Android config and running it again.

Test it

Rsync4Android has a dry run mode so you can see if it works. When testing, I also ran sshd in debug mode. Then I ran the backup in "production mode"; to the left is a screen capture of my phone. The reason you see a lot of output is because I have --partial --progress as rsync options; you should configure it to fit your needs. I suggest to also check on the --exclude option; I think you will find it quite useful.

Now we know it works, we can worry about running it as a cron job (the Rsync4Android docs have a link for how to do that) and then make it work from an external network. But that is for another article.

Saturday, June 22, 2019

Bitten by double quotes while passing arguments to a bash function

So I wrote a script to setup some networks. The script is not that important for this article; in fact I do not like said script anymore and will replace it with some wholesome ansible playbooks. What matters in it is this bit of code:

So the script is called nething.sh and kinda looks like this:

#! /bin/bash
# Populate openstack networks

HEAD_NODE_IPADDRESS="10.0.0.30"
COMPUTE_NOTE=$HEAD_NODE_IP_ADDRESS
SEC_GROUP="custom-1"
NET_TYPE="flat"
PHYSICAL_NETWORK_DEFAULT="extnet"
NET_NAME="public"
SUBNET_NAME="subnet-${NET_NAME}"
CIDR="10.0.0.0/24"
DNS_NAMESERVER1="10.0.0.10"
GATEWAY_IP="10.0.0.2"
DHCP_ALLOCATION_START="10.0.0.249"
DHCP_ALLOCATION_END="10.0.0.254"

# Boring things here
function CreateNetwork
{
   net_name=$1
   net_id=$2          # vlan tag, segmentation ID
   net_type=$3
   subnet_name=$4
   subnet_range=$5    # Subnet is not necessarily as wide as its parent network
   gateway=$6
   subnet_dhcp_start=$7
   subnet_dhcp_end=$8
   nameserver=$9
   physical_net=$10

   if [[ ${net_id} == "EXT" ]]; then
      openstack network create --provider-network-type ${net_type} \
[...]
}

# Public Network
   CreateNetwork \
      $NET_NAME \
      "EXT" \
      $NET_TYPE \
      $SUBNET_NAME \
      $CIDR \
      $GATEWAY_IP \
      $DHCP_ALLOCATION_START \
      $DHCP_ALLOCATION_END \
      $DNS_NAMESERVER1 \
      $PHYSICAL_NETWORK_DEFAULT

which defines a few constants on the top and then pass them to a function in the end of the script.

Every time I ran the script, it kept crashing and burning, and I did not know why. So I ran the script as bash -x nething.sh so it would spit out each line/variable as it passed through them. Here is how the output looks like:

+ HEAD_NODE_IPADDRESS=10.0.0.30
+ COMPUTE_NOTE=
+ SEC_GROUP=custom-1
+ NET_TYPE=flat
+ PHYSICAL_NETWORK_DEFAULT=extnet
+ NET_NAME=public
+ SUBNET_NAME=public_subnet
+ CIDR=10.0.0.0/24
+ DNS_NAMESERVER1=10.0.0.10
+ GATEWAY_IP=10.0.0.2
+ DHCP_ALLOCATION_START=10.0.0.249
+ DHCP_ALLOCATION_END=10.0.0.254
+ CreateNetwork public BLANK flat public_subnet 10.0.0.0/24 ' '
+ net_name=public
+ net_id=BLANK
+ net_type=flat
+ subnet_name=public_subnet
+ subnet_range=10.0.0.0/24
+ gateway=' '
+ subnet_dhcp_start=
+ subnet_dhcp_end=
+ nameserver=
+ physical_net=public0
+ [[ BLANK == \E\X\T ]]
+ [[ BLANK == \B\L\A\N\K ]]

The line starting with CreateNetwork public BLANK means we are going to the function CreateNetwork(); what is come after is a list of the local variables inside the function which were populated by the function call. The [[ BLANK == \E\X\T ]] is how an if test looks like under the -x option, specifically

if [[ ${net_id} == "EXT" ]]; then

To save time I will spoil the fun: take a look at the gateway variable. Why is it blank? And why every other variable after it is blank? Note that the line

+ CreateNetwork public BLANK flat public_subnet 10.0.0.0/24 ' '

tells us that

gateway is being passed as ' ' right from the function call.
No other argument after gateway is being passed.

Why?

The problem is CIDR="10.0.0.0/24", specifically the double quotes. They are expanding the string 10.0.0.0/24 and interpreting the /24 as a character. The solution is to use single quotes, as in

HEAD_NODE_IPADDRESS="10.0.0.30"
COMPUTE_NOTE=$HEAD_NODE_IP_ADDRESS
SEC_GROUP="custom-1"
NET_TYPE="flat"
PHYSICAL_NETWORK_DEFAULT="extnet"
NET_NAME="public"
SUBNET_NAME="${NET_NAME}_subnet"
CIDR='10.0.0.0/24'
DNS_NAMESERVER1="10.0.0.10"
GATEWAY_IP="10.0.0.2"
DHCP_ALLOCATION_START="10.0.0.249"
DHCP_ALLOCATION_END="10.0.0.254"

If we then run that, the strings now behave as expected:

[root@stakola ~(keystone_admin)]# bash -x nething.sh
+ HEAD_NODE_IPADDRESS=10.0.0.30
+ COMPUTE_NOTE=
+ SEC_GROUP=custom-1
+ NET_TYPE=flat
+ PHYSICAL_NETWORK_DEFAULT=extnet
+ NET_NAME=public
+ SUBNET_NAME=public_subnet
+ CIDR=10.0.0.0/24
+ DNS_NAMESERVER1=10.0.0.10
+ GATEWAY_IP=10.0.0.2
+ DHCP_ALLOCATION_START=10.0.0.249
+ DHCP_ALLOCATION_END=10.0.0.254
+ CreateNetwork private 6855 vlan private_subnet 192.168.55.0/24 192.168.55.1 192.
168.55.100 192.168.55.200 10.0.0.10 physnet1
+ net_name=private
+ net_id=6855
+ net_type=vlan
+ subnet_name=private_subnet
+ subnet_range=192.168.55.0/24
+ gateway=192.168.55.1
+ subnet_dhcp_start=192.168.55.100
+ subnet_dhcp_end=192.168.55.200
+ nameserver=10.0.0.10
+ physical_net=private0
+ [[ 6855 == \E\X\T ]]
+ [[ 6855 == \B\L\A\N\K ]]
[...]

Moral of the story: know when to use single and double quotes!

Thursday, May 23, 2019

Programming a Netronome network card (inside a VM) from command line

This is the same card(s) we got to work inside a vm guest using the magic of PCI passthrough. Netronome wants us to use a Windows-only IDE to do development work in it while the card is placed in a Linux box we can reach; some of its features remind me of the IDE Google has for Androids, which allows you to run an emulator and do some real time debugging. The difference is the Google one works in Linux, Windows, and Mac, and it only requires one computer (which could be remotely accessed).

Do we really need to use the Netronome SDK? I guess it depends on what we want to do. For now, let's see if we can get something running using command line only to the point we can compile in Micro-C and run something in the card.

Get the packages

First we need a few packages available for either CentOS or Ubuntu.

Note: The Netronome Linux SDK officially only support CentOS and Ubuntu. So we will only be covering those distros.

Ubuntu:

apt-get install libftdi1 libjansson4 build-essential \
 linux-headers-`uname -r` dkms git

CentOS: (Still using yum; will do a dnf version when I feel like.

yum -y install epel-release && yum update -y
yum -y install libftdi jansson pciutils kernel-devel dkms wget git

Netronome does require you to have an account to get the SDK packages. I can't help with that; what I can tell you is that once I got the account I downloaded everything which was available at the time I wrote this article:

raub@desktop:~$ ls Downloads/netronome/
SDK
agilio-nfp-driver-dkms-2018.01.11.2333.f40482a-1.el7.noarch.rpm
agilio-nfp-driver-dkms_2018.01.11.2333.f40482a_all.deb
firmware
readme
raub@desktop:~$ ls Downloads/netronome/SDK/
6.0.4.1 6.1.0.1
raub@desktop:~$ ls Downloads/netronome/SDK/6.1.0.1/
nfp-sdk-6.1.0.1-preview-3286-setup.exe
nfp-sdk-6.1.0.1_preview-0-3243.x86_64.rpm
nfp-sdk-p4-rte-6.1.0.1-preview-3202.centos.x86_64.tar
nfp-sdk-p4-rte-6.1.0.1-preview-3214.ubuntu.x86_64.tar
nfp-sdk-sim-6.1.0.0-preview-3179.x86_64.tar
nfp-sdk_6.1.0.1-preview-3243-2_amd64.deb
nfp-toolchain-6.1.0.1-preview-3243.x86_64.tar
raub@desktop:~$

and then copied them all to the development vm guest we created in the previous article, desktop1

What are those files? I put what I have gathered about their function in the readme file (it covers the old file version but it should get an idea):

Programmer Studio IDE
nfp-sdk-6.0.4.1-3276-setup.exe - Windows

Run Time Environment (RTE)
nfp-sdk-p4-rte-6.0.4.1-3195.ubuntu.x86_64.tgz
nfp-sdk-p4-rte-6.0.4.1-3191.centos.x86_64.tgz

Hosted Toolchain (to be used with BSP and SmartNIC)
nfp-sdk_6.0.4.1-3227-2_amd64.deb
nfp-sdk-6.0.4.1-0-3227.x86_64.rpm

NFP Simulator
nfp-sdk-sim-6.0.4.1-3177.x86_64.tgz

Hosted Toolchain (to be used with NFP Simulator)
nfp-toolchain-6.0.4.1-3227.x86_64.tgz

Note: There are two versions of the SDK. Just pick the latest.

Then install the basic SDK

Ubuntu:

sudo dpkg -i nfp-sdk_6.1.0.1-preview-3243-2_amd64.deb

CentOS:

sudo rpm -ivh nfp-sdk-6.1.0.1_preview-0-3243.x86_64.rpm

This creates a /opt/netronome directory.

And add to the path where the binaries will be installed.
```
cat >> ~/.bash_profile << 'EOF'

# Netronome SDK
PATH=$PATH:/opt/netronome/bin
export PATH
EOF
source ~/.bash_profile
```
Note: If the user you are building your code on does not have rights to write to the card, you should edit the root's .bash_profile file as well.

We do need the Netronome modified but open source nfp driver which has the development features (specificially, it has a nfp_dev_cpp option we will need to expose the low-level user space access ABIs of non-netdev mode) we need. So, we install it, which requires installing the Netronome repo:

Ubuntu:

wget https://deb.netronome.com/gpg/NetronomePublic.key
apt-key add NetronomePublic.key
add-apt-repository "deb https://deb.netronome.com/apt stable main"
apt-get update
apt-get install agilio-nfp-driver-dkms

CentOS:

wget https://rpm.netronome.com/gpg/NetronomePublic.key
rpm --import NetronomePublic.key
cat << 'EOF' > /etc/yum.repos.d/netronome.repo
[netronome]
name=netronome
baseurl=https://rpm.netronome.com/repos/centos/
gpgcheck=0
enabled=1
EOF
yum makecache
yum install -y agilio-nfp-driver-dkms --nogpgcheck

and then reboot.

Now we can install the RTE

Ubuntu:

tar xvf nfp-sdk-p4-rte-6.1.0.1-preview-3214.ubuntu.x86_64.tar
cd nfp-sdk-6-rte-v6.1.0.1-preview-Ubuntu-Release-r2750-2018-10-10-ubuntu.binary/
sudo ./sdk6_rte_install.sh install

CentOS:

tar xvf nfp-sdk-p4-rte-6.1.0.1-preview-3202.centos.x86_64.tar
cd nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary/
sudo ./sdk6_rte_install.sh install

This should cause /opt/netronome/bin/ to fill with many more files; this is a good way to check progress.

NOTE:Chances are It will get pissed:

[...]
Loaded plugins: fastestmirror
Examining /home/centos/netronome/SDK/6.1.0.1/nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary/dependencies/nfp-bsp/rpm//nfp-bsp-dkms_2018.08.17.1104_all.rpm: nfp-bsp-dkms-2018.08.17.1104-1dkms.noarch
Marking /home/centos/netronome/SDK/6.1.0.1/nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary/dependencies/nfp-bsp/rpm//nfp-bsp-dkms_2018.08.17.1104_all.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package nfp-bsp-dkms.noarch 0:2018.08.17.1104-1dkms will be installed
--> Processing Conflict: agilio-nfp-driver-dkms-2019.04.02.0225.bf81349-1.el7.noarch conflicts nfp-bsp-dkms
Loading mirror speeds from cached hostfile
 * base: packages.oit.ncsu.edu
 * epel: mirror.umd.edu
 * extras: packages.oit.ncsu.edu
 * updates: packages.oit.ncsu.edu
No package matched to upgrade: nfp-bsp-dkms
--> Finished Dependency Resolution
Error: agilio-nfp-driver-dkms conflicts with nfp-bsp-dkms-2018.08.17.1104-1dkms.noarch
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
Error! There are no instances of module: nfp-bsp-dkms
located in the DKMS tree.
[centos@desktop1 nfp-sdk-6-rte-v6.1.0.1-preview-CentOS-Release-r2749-2018-10-09-centos.binary]$

but it will get over and will work fine.

Ensure that nfp_dev_cpp = 1

theuser@desktop1:~$ cat /sys/module/nfp/parameters/nfp_dev_cpp
1
theuser@desktop1:~$

If not, say, yet get an error message like this

[theuser@desktop1 ~]# cat /sys/module/nfp/parameters/nfp_dev_cpp
cat: /sys/module/nfp/parameters/nfp_dev_cpp: No such file or directory
[theuser@desktop1 ~]#

uninstall nfp and install it back with the option set. There are ways to load said option at boot time; I will leave that as an exercise to the reader.

theuser@desktop1:~$ sudo modprobe -r -v nfp && sudo modprobe nfp nfp_dev_cpp=1
theuser@desktop1:~$

Ensure nfp-hwinfo is talking to the card. The expected outcome should look like this:

theuser@desktop1:~$ sudo /opt/netronome/bin/nfp-hwinfo
nfp.interface=pci.0.0
nfp.model=0x40010010
nfp.serial=00:15:4d:13:5d:2b
board.exec=bootloader.bin
uart.baud=115200
preinit.setup.version=nfp-bsp-6000-b0 (4ef1e19ba176)
pcie0.type=ep
assembly.revision=11
assembly.model=lithium
assembly.partno=AMDA0096-0001
assembly.serial=17290647
assembly.vendor=SMC
ddr0.spd=spi:1:0:0x3F0F00
ddr1.spd=spi:1:0:0x3F0F00
ddr2.spd=none
ddr3.spd=none
ddr4.spd=none
ddr5.spd=none
emu1.type=cache
emu2.type=cache
ethm.mac=00:15:4d:13:5d:2b
eth.mac=00:15:4d:13:5d:2c
eth.macs=2
vpd=fis:1:0:vpd.bin
board.setup.version=nfp-bsp-6000-b0 (4ef1e19ba176)
chip.model=NFP4001
chip.revision=B0
core.speed=633
me.speed=633
arm.speed=475
chip.model.device=0x62006c20
chip.identifier=0x219b8546c
chip.model.hard=0x5
chip.model.soft=0x40010096
chip.route=0xc96f1e8e
chip.island=0x1001f13000112
mem.setup.version=nfp-bsp-6000-b0 (4ef1e19ba176)
ddr0.mem.size=1024
ddr1.mem.size=1024
ddr0.mem.speed=1600
ddr1.mem.speed=1600
emu0.mem.size=2048
emu0.mem.base=0x2000000000
emu1.mem.size=3
[...]
theuser@desktop1:~$

If it looks like this:

theuser@desktop1:~$ sudo /opt/netronome/bin/nfp-hwinfo
/opt/netronome/bin/nfp-hwinfo: Failed to open NFP device 0 (No such device)
Please check that:
 -lspci -d 19ee: shows atleast one Netronome device
 -the nfp device number is correct
 -the user has read and write permissions to the Netronome device
 -the nfp.ko module is loaded
 -the nfp_dev_cpp option is enabled (please try modinfo nfp to see all params)
theuser@desktop1:~$

stop, do not continue. Go back and check if nfp_dev_cpp = 1 and also if the vm was configured to support PCIe cards. Do not continue until you have checked and addressed these two items.

Coding, at last!

This Hello World was stolen from the Netronome appropriately named Hello World example. I will be rushing through it, concentrating on getting it to compile and showing some common issues. Lookup on the example docs for what each line does.

So we create our hello world project using lab_template as the, well, template.
```
mkdir dev
cd dev
git clone https://github.com/open-nfpsw/c_packetprocessing.git
cd c_packetprocessing/apps/
cp -r lab_template lab_hello_world
cd lab_hello_world
```
NOTE: This creates a ~/dev/c_packetprocessing/apps/lab_hello_world directory. If you want to move it to a different location, edit the line
```
ROOT_SRC_DIR  ?= $(realpath $(app_src_dir)/../..)
```
in the Makefile.

So far the hello world directory looks rather bare:

theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ ls
Makefile  README
theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

so let's start populating it.

cat > hello_world.c << 'EOF'
#include <nfp.h>
__declspec(ctm) int old[] = {1,2,3,4,5,6,7,8,9,10};
__declspec(ctm) int new[sizeof(old)/sizeof(int)];

int main(void)
{
        if (__ctx() == 0)
        {
                int i, size;
                size = sizeof(old)/sizeof(int);
                for (i=0; i < size; i++)
                {
                        new[i] = old[size - i - 1];
                }
        }
        return 0;
}
EOF

We add a few lines to the makefile. Their explanation is listed in.

sed -i -e '/^# Application definition starts here/ a\
$(eval $(call micro_c.compile_with_rtl,hello_world_obj,hello_world.c)) \
$(eval $(call fw.add_obj,hello_world,hello_world_obj,i32.me0 i32.me1)) \
$(eval $(call fw.link_with_rtsyms,hello_world))' Makefile

Time for some compiling!

theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ make
/opt/netronome/bin/nfcc -Fo/home/theuser/dev/c_packetprocessing/apps/lab_hello_world/ -Fe/home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world_obj.list -W3 -chip nfp-4xxx-b0 -Qspill=7 -Qnn_mode=1 -Qno_decl_volatile -single_dram_signal -Qnctx_mode=8 -I. -I/home/theuser/dev/c_packetprocessing/microc/include -I/home/theuser/dev/c_packetprocessing/microc/lib   /opt/netronome/components/standardlibrary/microc/src/rtl.c /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.c
/opt/netronome/bin/nfld -chip nfp-4xxx-b0 -mip -rtsyms -o /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw -map /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.map -u i32.me0 /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world_obj.list -u i32.me1 /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world_obj.list
theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

which creates a few intermediate files:

theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ ls
hello_world.c   hello_world.map  hello_world_obj.list  README
hello_world.fw  hello_world.obj  Makefile              rtl.obj
theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ cat hello_world.map
Memory Map file: /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.map
Date: Tue May  7 10:39:30 2019

nfld version: 6.0.4.1,  NFFW: /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw

Address       Region     ByteSize        Symbol
===================================================
0x0000000000800000    i24.emem      108                 .mip
0x0000000000000000    i32.ctm       704                 i32.me0.ctm_40$tls
0x00000000000002c0    i32.ctm       704                 i32.me1.ctm_40$tls

ImportVar                       Uninitialized Value
===================================================
theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

Next upload the firmware we created into the card. This needs to be run either as root or as an user who can write to card.

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# make load_hello_world
nfp-nffw load --no-start /home/theuser/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw
root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#

NOTE: If you see the following error message

theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ make load_hello_world
nfp-nffw load --no-start /home/centos/dev/c_packetprocessing/apps/lab_hello_world/hello_world.fw
nfp-nffw: Failed to open NFP device 0 (No such device)
Please check that:
 -lspci -d 19ee: shows atleast one Netronome device
 -the nfp device number is correct
 -the user has read and write permissions to the Netronome device
 -the nfp.ko module is loaded
 -the nfp_dev_cpp option is enabled (please try modinfo nfp to see all params)
nfp-nffw: Command 'load' failed
make: *** [load_hello_world] Error 1
theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

ou should check if

If you are running make load_hello_world as user who can write to the card.
nfp_dev_cpp = 1
the vm was configured to support PCIe cards.

Go back in this document for instructions on how to do so.

Now, if you see this error message

[F] nfp6000_nffw.c:4643: Firmware already loaded. Unload first.
Failed to load firmware: Operation not permitted
nfp-nffw: Command 'load' failed
Makefile:43: recipe for target 'load_hello_world' failed

either you or someone else had already loaded firmware into the card. All you have to do is unload it

theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$ nfp-nffw unload
theuser@desktop1:~/dev/c_packetprocessing/apps/lab_hello_world$

and then run make load_hello_world again.

In the hello world instructions, the next step is to see the card memory since later on we will be writing to it. So, here is it.

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# nfp-rtsym --len 176 i32.me0.ctm_40\$tls:0
0x0000000000:  0x00000001 0x00000002 0x00000003 0x00000004
0x0000000010:  0x00000005 0x00000006 0x00000007 0x00000008
0x0000000020:  0x00000009 0x0000000a 0x00000000 0x00000000
0x0000000030:  0x00000000 0x00000000 0x00000000 0x00000000
*
0x0000000050:  0x00000000 0x00000000 0x00000001 0x00000002
0x0000000060:  0x00000003 0x00000004 0x00000005 0x00000006
0x0000000070:  0x00000007 0x00000008 0x00000009 0x0000000a
0x0000000080:  0x00000000 0x00000000 0x00000000 0x00000000
*

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#

Unleash the code so it does things:

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# make fw_start
nfp-nffw start
root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#

If things were successfully done, we now can see the memory contents have changed:

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# nfp-rtsym --len 176 i32.me0.ctm_40\$tls:0
0x0000000000:  0x00000001 0x00000002 0x00000003 0x00000004
0x0000000010:  0x00000005 0x00000006 0x00000007 0x00000008
0x0000000020:  0x00000009 0x0000000a 0x00000000 0x00000000
0x0000000030:  0x0000000a 0x00000009 0x00000008 0x00000007
0x0000000040:  0x00000006 0x00000005 0x00000004 0x00000003
0x0000000050:  0x00000002 0x00000001 0x00000001 0x00000002
0x0000000060:  0x00000003 0x00000004 0x00000005 0x00000006
0x0000000070:  0x00000007 0x00000008 0x00000009 0x0000000a
0x0000000080:  0x00000000 0x00000000 0x00000000 0x00000000
*

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#

Don't forget to unload the firmware by typing nfp-nffw unload!

Checking that we are done

root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world# nfp-rtsym --len 176 i32.me0.ctm_40\$tls:0
No runtime symbol named 'i32.me0.ctm_40$tls'
root@desktop1:/home/theuser/dev/c_packetprocessing/apps/lab_hello_world#

So congratulations! You not only installed the SDK and wrote and ran your first Netronome program! You may want to look into the Network Flow C Compiler User's Guide for further info on what you can do; I would put the link but right on the front page it states it is Proprietary and Confidential.

Next time we will do some openflow or P4 coding. Don't ask me to tell which one will be because I have not decided yet. Brain hurts!

What about the simulator? Maybe one day.

Friday, May 03, 2019

Passing a Network card to a KVM vm guest because we are too lazy to configure SR-IOV

This can be taken as a generic how-to about passing PCIe cards to a mv guest. I will

Why

I can come up with a lot of excuses. The bottom line is you want the vm guest to do something with the card the vm host can't or shouldn't. For instance, what if we want to give a wireless card for a given vm guest? And the card is not supported by the vm host (I am looking at you, VMWare ESXi) or the vm host does not know how to virtualize it in a meaningful way?

Note: What we are talking about here should work with any PCI/PCIe card, but we said we will be talking about network cards, so there.

The Card

The card is a PCIe network card; for this article it should be seen as a garden-variety network card. You probably will not let me leave at that so, here is the info on the specific card I will be using in this article: it is a Netronome Agilio CX 2x10GbE (the one in the picture is a CX 1x40GbE, which I happen to own hence the crappy picture), which is built around their NFP-4000 flow processor. Basic informercial on it can be found at https://www.netronome.com/m/documents/PB_Agilio_CX_2x10GbE.pdf (it used to be https://www.netronome.com/media/documents/PB_Agilio_CX_2x10GbE.pdf, but I guess they thought media was too long a word. It also means that sometime after this article is posted the link will change again; no point on making them orange links). It is supposed to do things like KVM hypervisor support (SR-IOV comes to mind) right out of the box, so why we would want to passthrough the entire card to a vm guest? Here are some reasons:

What if the card can do something and the VM abstraction layers do not expose that?
What if we want to program the card to do our bidding?
What if we want to change the firmware of the card? Some cards allow you to upgrade the firmware, or change it completely to use it for other thingies (the Netronome card in question fits this second option, details about that might be discussed in a future article).
Why did you pick this card? Hey, this is not a reason to pass the entire card, but I will answer it anyway: because I have a box with 3 of them I was going to use for something else (we may talk about that in a future article). With that said, I avoided going over any of the special sauce this card has. For the purpose of this article, it is just a PCIe card I want to give to a vm guest.

How

Finding the card

Ok, card is inserted into the vm host, which booted properly. Now what? Well, we need to find where the card is so we can tell our guests. Most Linux distros come with lspci, which probulates the PCI bus. The trick is to search for the right pattern. Let's for instance look for network devices in one of my ESXi nodes:

[root@vmhost2:~] lspci | grep 'Network'
0000:00:19.0 Network controller: Intel Corporation 82579LM Gigabit Network Connection [vmnic0]
0000:04:00.0 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic1]
0000:04:00.1 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic2]
0000:05:00.0 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic3]
0000:05:00.1 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [vmnic4]
[root@vmhost2:~]

Notes
ESXi is really not Linux but freebsd with gnu packages sprinkled over

I just mentioned ESXi here because I needed another system I could run lscpi on.

The lscpi options in ESXi are not as extensive as in garden-variety Linux. But, it is good enough to show it in action.

If we had searched for Intel Corporation we would get much much more replies including the CPU itself. So, taking the time to get the right search string pays off.

If we were going to probulate in a Linux host, Ethernet works better than Network as the search pattern. We can even look at virtual interfaces KVM is feeding to a vm guest:

theuser@desktop1:~$ lspci |grep Ethernet
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:06.0 Ethernet controller: Red Hat, Inc. Virtio network device
theuser@desktop1:~$

Note that the 0000: is assumed. A very useful option available in the Linux version of lspci but not the ESXi one is -nn:

theuser@desktop1:~$ lspci -nn |grep Ethernet
00:03.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
00:06.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
theuser@desktop1:~$

The [1af4:1000] means [vendor_id:product_id]; remember it well.

For the Netronome cards we can just look for netronome since there should be no other devices matching that name besides the cards made by them:

raub@vmhost ~$ sudo lspci -nn|grep -i netronome
11:00.0 Ethernet controller [0200]: Netronome Systems, Inc. Device [19ee:4000]
raub@vmhost ~$

The card's PCI address is 11:00.0

Handing out the card to the guest

Two things we need to do when passing a PCI device to a vm guest (a.k.a. desktop1 in this example):

Tell the vm host to keep its hands off it. The reason is that, in the case of a network card, it might want to configure it, creating interfaces (in the /dev/ directory tree) which either the host server (vmhost) can use for its own nefarious uses or so KVM can then virtualize (as a Virtio network device or some other emulation) to hand out to the guests. Since we want to use said card for our personal private personal nefarious purposes within a specific vm guest (desktop1), we are not going to be nice and share it.
So we need to tell vmhost to leave it alone.
- KVM knows it exists because it can look in the PCI chain by itself:
```
[root@vmhost ~]# virsh nodedev-list | grep pci_0000_11
pci_0000_11_00_0
[root@vmhost ~]#
```
- So now we can tell vmhost to leave pci-0000:11:00.0 alone:
```
[root@vmhost ~]$ sudo virsh nodedev-dettach pci_0000_11_00_0
Device pci_0000_11_00_0 detached

[root@vmhost ~]$
```

Tell the vm guest there is this shiny card it can lay its noodly appendages on.

Shut the vm guest down.
Edit the desktop.
```
virsh edit desktop
```

Add something like

<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
          <address domain='0x0000' bus='0x11' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

to the end of the devices session. When you save it, it will properly place
and configure the entry.

Restart vm guest check if it can see the card using dmesg (Ubuntu 19.04 example. Note it is being listed as pci-0000:04:00.0 inside the vm guest). I expect to see something like

[    7.348276] Netronome NFP CPP API
[    7.352347] nfp-net-vnic: NFP vNIC driver, Copyright (C) 2010-2015 Netronome Systems
[    7.361865] nfp 0000:04:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe
[    7.372133] nfp 0000:04:00.0: RESERVED BARs: 0.0: General/MSI-X SRAM, 0.1: PCIe XPB/MSI-X PBA, 0.4: Explicit0, 0.5: Explicit1, free: 20/24
[    7.396094] nfp 0000:11:00.0: Model: 0x40010010, SN: 00:15:4d:13:5d:58, Ifc: 0x10ff

But what I am getting is something more like this:

[    1.768683] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[    1.773014] nfp 0000:00:07.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe
[    1.774066] nfp 0000:00:07.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[    1.775212] nfp 0000:00:07.0: can't find PCIe Serial Number Capability
[    1.776252] nfp 0000:00:07.0: Interface type 15 is not the expected 1
[    1.777285] nfp 0000:00:07.0: NFP6000 PCI setup failed

What is going on? The answer to that is the next topic. You see,

PCIe is more demanding

Do you remember the can't find PCIe Serial Number Capability message? This is a PCIe card, meaning we need to setup the vm guest machine type to q35, which supports the ICH9 chipset which can handle a PCIe bus. The default (I440FX) can only do PCI bus. QEMU has a nice description on the difference. So, let's give it a try by recreating the KVM guest:

virt-install \
   --name desktop1 \
   --disk path=/home/raub/desktop1.qcow2,format=qcow2,size=10 \
   --ram 4098 --vcpus 2 \
   --cdrom /export/public/ISOs/Linux/ubuntu/ubuntu-16.04.5-server-amd64.iso  \
   --os-type linux --os-variant ubuntu19.04 \
   --network network=default \
   --graphics vnc --noautoconsole \
   --machine=q35 \
   --arch x86_64

When we try to build that vm guest, we get an error message stating that

ERROR    No domains available for virt type 'hvm', arch 'x86_64', machine type 'q35'

What now? You see, at the time I wrote this, the CentOS KVM package does not support q35 out of the box. We need more packages!

yum install centos-release-qemu-ev
yum update
reboot

And we try again, this time when we login to the guest, desktop1, it looks more promising (note the PCI address changed to 0000:01:00.0; this is a new vm guest):

theuser@desktop1:~$ dmesg |grep -i netro
[    1.922051] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[    1.954196] nfp 0000:01:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe
[    2.239018] nfp 0000:01:00.0: nfp:   netronome/serial-00-15-4d-13-5d-46-10-ff.nffw: not found
[    2.239059] nfp 0000:01:00.0: nfp:   netronome/pci-0000:01:00.0.nffw: not found
[    2.239913] nfp 0000:01:00.0: nfp:   netronome/nic_AMDA0096-0001_2x10.nffw: found, loading...
[   11.954477] nfp 0000:01:00.0 eth0: Netronome NFP-6xxx Netdev: TxQs=2/32 RxQs=2/32
[   11.971175] nfp 0000:01:00.0 eth1: Netronome NFP-6xxx Netdev: TxQs=2/31 RxQs=2/31
theuser@desktop1:~$

Which then becomes

theuser@desktop1:~$ ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d4:9e:50 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.105/24 brd 192.168.122.255 scope global dynamic enp0s3
       valid_lft 3489sec preferred_lft 3489sec
    inet6 fe80::5054:ff:fed4:9e50/64 scope link
       valid_lft forever preferred_lft forever
3: enp1s0np0:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:15:4d:13:5d:47 brd ff:ff:ff:ff:ff:ff
4: enp1s0np1:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:15:4d:13:5d:48 brd ff:ff:ff:ff:ff:ff
theuser@desktop1:~$

And now we can do something useful with it.

References

https://stackoverflow.com/questions/14061840/kvm-and-libvirt-wrong-cpu-type-in-virtual-host
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-kvm_guest_virtual_machine_compatibility-supported_cpu_models
https://github.com/libvirt/libvirt/blob/v4.0.0/src/util/virarch.c#L37

Wednesday, January 16, 2019

Getting the profile name for installing a ESXi patch from command line

So I go to the vmware page for ESXi patches and downloaded the patch

ESXi670-201811001

Product:ESXi (Embedded and Installable) 6.7.0

Download Size:317.5 MB

But when I try to install it, I need a "patch" name, whatever that is

[root@vmhost2:/tmp] esxcli software profile update -d /vmfs/volumes/VOL1
/ISO/ESXi670-201811001.zip
Error: Missing required parameter -p|--profile

Usage: esxcli software profile update [cmd options]

Description:
  update                Updates the host with VIBs from an image profile in a
                        depot. Installed VIBs may be upgraded (or downgraded
                        if --allow-downgrades is specified), but they will
                        not be removed. Any VIBs in the image profile which
                        are not related to any installed VIBs will be added
                        to the host. WARNING: If your installation requires a
                        reboot, you need to disable HA first.

Cmd options:
  --allow-downgrades    If this option is specified, then the VIBs from the
                        image profile which update, downgrade, or are new to
                        the host will be installed. If the option is not
                        specified, then the VIBs which update or are new to
                        the host will be installed.
  -d|--depot=[  ... ]
                        Specifies full remote URLs of the depot index.xml or
                        server file path pointing to an offline bundle .zip
                        file. (required)
  --dry-run             Performs a dry-run only. Report the VIB-level
                        operations that would be performed, but do not change
                        anything in the system.
[...]
  -p|--profile=    Specifies the name of the image profile to update the
                        host with. (required)
  --proxy=         Specifies a proxy server to use for HTTP, FTP, and
                        HTTPS connections. The format is proxy-url:port.
[root@vmhost2:/tmp]

Maybe it is named after the file? Let's punt

[root@vmhost2:/tmp] esxcli software profile update -d /vmfs/volumes/VOL1/IS
O/ESXi670-201811001.zip  -p ESXi-6.7.0-201811001-standard
 [NoMatchError]
 No image profile found with name 'ESXi-6.7.0-201811001-standard'
         id = ESXi-6.7.0-201811001-standard
 Please refer to the log file for more details.
[root@dentistryesxi2:/tmp]

After feeling a bit frustrated, I found some docs on updating a host using image profiles with the following example:

esxcli --server=server_name software sources profile list --depot=http://webserver/depot_name

Well, we know even though the docs seem to imply the "depot" is a server somewhere with the patch bundles as .zip files, we can just download and give the path. So, let's see if that applies here too:

[root@vmhost2:/tmp] esxcli software sources profile list -d /vmfs/volume
s/VOL1/ISO/ESXi670-201811001.zip
Name                             Vendor        Acceptance Level  Creation Time        Modification Time
-------------------------------  ------------  ----------------  -------------------  -------------------
ESXi-6.7.0-20181104001-no-tools  VMware, Inc.  PartnerSupported  2018-11-08T08:39:27  2018-11-08T08:39:27
ESXi-6.7.0-20181104001-standard  VMware, Inc.  PartnerSupported  2018-11-08T08:39:27  2018-11-08T08:39:27
[root@dentistryesxi2:/tmp]

Aha! I was close but no cookies. I really would never have guessed it was 20181104001 instead of 201811001; that is why it pays to ask whoever knows the right answer, in this case the file itself. Let's try it:

[root@vmhost2:/tmp] esxcli software profile update -d /vmfs/volumes/VOL1/ISO/ESXi670-201811001.zip \
 -p ESXi-6.7.0-20181104001-standard
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true 
   VIBs Installed: VMW_bootbank_bnxtroce_20.6.101.0-20vmw.670.1.28.10302608, VMW_bootbank_brcmfcoe_11.4.1078.5-11vmw.670.1.28.10302608, VMW_bootbank_elxnet_11.4.1095.0-5vmw.670.1.28.10302608, VMW_bootbank_i40en_1.3.1-22vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMW_bootbank_ixgben_1.4.1-16vmw.670.1.28.10302608, VMW_bootbank_lpfc_11.4.33.3-11vmw.670.1.28.10302608, VMW_bootbank_lsi-mr3_7.702.13.00-5vmw.670.1.28.1030260
[...]
otbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-
2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VM
W_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb
-storage_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.0-0.0.8169922, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, VMware_bootbank_esx-xserver_6.7.0-0.0.8169922, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMwa
re_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMware_bootbank_
native-misc-drivers_6.7.0-0.0.8169922, VMware_bootbank_qlnativefc_3.0.1.0-5vmw.
670.0.0.8169922, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922
[root@vmhost2:/tmp]

Much better.

Monday, September 24, 2018

Using ldapsearch and ldapmodify to talk to Active Directory

Why?

Great question! Here are a few lame excuses I was able to come with:

I like to use command line. This is a lame excuse because Windows have powershell. But,
I am more comfortable with Linux than Windows. Lame excuse since
1. How many posts in this very blog I have made about using Windows?
2. How many of said posts I have used the GUI when I could take care of business with Powershell?
3. What is wrong with Powershell, at least of the applications I have used it for here so far?

I do have a couple of not so lame ones though:

I like to be able to access the network resources from any machine in the network running any OS. If I have a Linux box in an Active Directory-controlled network, chances are I will need to authenticate the Linux box against Active Directory (AD so I can save some keytaps). AD is Kerberos + ldap + sprinkles, so I better be able to use the usual kerberos/ldap Linux tools as one day I will need to figure out why things are boink.
It feels like I get more info using ldapsearch than the Windows tools, which is good when I do not know the name of an attribute, or how many instaces of said attribute are in use.

Using ldapsearch

Before we go mindlessly typing things, we need some data.

We need to know the name of the ldapserver.

Yes, if you have it configured in your ldap.conf file, you should not need it. But I prefer to assume nothing. If the domain was setup properly, we can ask it directly by typing nslookup -type=srv _ldap._tcp.DOMAIN where DOMAIN is the Active Directory domain name, not the DNS one; that caught me off guard. So, if our DNS dmain is example.com and the AD domain (we are very original) is ad.example.com, we have

raub@desktop:/tmp$ nslookup -type=srv _ldap._tcp.ad.example.com
Server:         192.168.0.10
Address:        192.168.0.10#53

Non-authoritative answer:
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC0.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC2.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC1.ad.example.com.

Authoritative answers can be found from:
ad.example.com      nameserver = addc1.ad.example.com.
ad.example.com      nameserver = ns.example.com.
ad.example.com      nameserver = ns2.example.com.
ad.example.com      nameserver = addc0.ad.example.com.
ad.example.com      nameserver = addc2.ad.example.com.
ADDC0.ad.example.com        internet address = 192.168.1.100
ADDC1.ad.example.com        internet address = 192.168.1.102
ADDC2.ad.example.com        internet address = 192.168.1.101
ns.example.com      internet address = 192.168.0.10
ns2.example.com     internet address = 192.168.0.10

raub@desktop:/tmp$

and we can use any of the ADDCN.ad.example.com (where N=0,1,2). Notice in my setup, just using ad.example.com also worked. I found out by using netcat to see if port 636 was open (I will leave the answer for "why port 636?" as an exercise to the reader)

raub@desktop:~$ nc -v ad.example.com 636
Connection to ad.example.com 636 port [tcp/ldaps] succeeded!
^C
raub@desktop:~$

We need to be able to authenticate against AD somehow.

For this discussion I will be using a username and password; we can also do it using a Kerberos TGT ticket.

Fun Fact: I have a user account, my normal one, which can look into some things in LDAP/AD but then I have another ("admin") account I can see more and edit stuff in AD. You will see later on me forgetting completely about that and how it affects me. But we are getting ahead of ourselves.

With that taken care of, we are going to begin by looking for some user: me

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" 
Enter LDAP Password:
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
objectClass: top
objectClass: posixAccount
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Wrong Droid
sn: Droid
title: Entropy Creators
description: Orthodontics
givenName: Wrong
initials: B 
distinguishedName: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
instanceType: 4
whenCreated: 20080109142820.0Z
whenChanged: 20180905201157.0Z
displayName: Droid, Wrong
uSNCreated: 243336
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
[...]
proxyAddresses: sip:raub@ad.example.com
proxyAddresses: smtp:raub@ad.example.com
proxyAddresses: X500:/o=UNC Exchange/ou=Exchange Administrative Group (FYDIBOH
 F23SPDLT)/cn=Recipients/cn=raub
proxyAddresses: SMTP:raub@email.example.com
displayNamePrintable: Wrong Droid
name: Wrong Droid
[...]
sExchPoliciesExcluded: {26491cfc-9e50-4857-861b-0cb8df22b5d7}
msExchUserAccountControl: 0
msExchELCMailboxFlags: 2
msRTCSIP-PrimaryHomeServer:
[...]
msExchOWAPolicy: CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=ad,DC=example,DC=com

# refldaps://ForestDnsZones.ad.example.com/DC=ForestDnsZones,DC=ad,DC=example,DC=com

# refldaps://DomainDnsZones.ad.example.com/DC=DomainDnsZones,DC=ad,DC=example,DC=com

# refldaps://ad.example.com/CN=Configuration,DC=ad,DC=example,DC=com

raub@desktop:~$

That is probably more info than you wanted to know about someone, but this has its applications. Since we now know every attribute associated with a given user (I should not be that special, at least as far as AD is concerned), we can build customized queries looking for only a specific bit of info. For instance, let's just get the groups I belong to or am a member of (hint hint):

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" memberOf 
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com

Fancy, huh? Now, if instead of doing (CN=raub) we did (CN=*raub*), it would return every entry that has a CN with raub in it. In my case that would mean two entries, raub and raub.admin (if you remember our Fun Fact you will know about it), but it could also have returned a device whose name matches that. And, they would have been printed one after the other (I do wonder if the order depends on the order they were added to LDAP/AD).

Using ldapmodify

Ok, we established we can probulate Active Directory using common household Linux/UNIX query tools. What if we want to change something? Let's say we have an AD group (specifically a distribution list) called moustache_operators (for those who own and operate moustaches) and want to add a member and delete another.

Why I do like ldapmodify to edit LDAP/AD
Main reason is because I can create a file (in the LDIF format) at my leisure (i.e. think about what I want to do) with pretty commands and comments describing what I want to do. If I like what I did, I can then document it and maybe even save the file in a wiki or somewhere that can be fed to Ansible/Puppet/Chef/Docker and reused.

Our little LDIF file, let's call it change.ldif, could look like this:

# Let's define the entity we will be fiddling with
dn: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com
# And then what we will be doing with it
changetype: modify
delete: member
member: CN=baldone,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
# Separator because we will be doing another change
-
add: member
member: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com

So let's try it:

raub@desktop:~$ ldapmodify -H "ldaps://addc0.ad.example.com:636" -D "raub@ad.example.com" -x -W -f change.ldif
Enter LDAP Password:
modifying entry "CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com"
ldap_modify: Insufficient access (50)
        additional info: 00002098: SecErr: DSID-03150F93, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

raub@desktop:~$

Why is it not working? Well, do you remember the Fun Fact I mentioned earlier in this article? This is how it shows it's ugly head. I should have used my raub.admin@ad.example.com account instead of raub@ad.example.com. If we do it right, it then works. I will not show the output of a successful connection here; what matters is verifying the deed is done, and we can do it using dear ol' ldapsearch:

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub) memberOf" dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com

What about Mac/OSX?

They too have ldapsearch; just use the terminal and off you go.

Wednesday, August 29, 2018

Customizing the prompt in OSX (or anything running bash)

Quick and dirty article whose only claim to fame is to show another example of treating a Mac running OSX a UNIX box: I have two Mac, an old Mini and a MacBook Air (I think I talked about it before). If you have been reading this blog, you expect me to use the shell a lot in them, and you would be right. One thing that annoys me is the default prompt used in both. The old, slow mini has the most useless one:

bash-3.2$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
Copyright (C) 2007 Free Software Foundation, Inc.
bash-3.2$

Why would I care about knowing at all times which bash version I am running, and even then it is incomplete info (compared to bash --version)? The MacBook Air is just slightly better, showing (in order) the name of the machine, the path, and then the username.

littleguy:~ raub$

It is similar in content to what I would see in my Linux boxes,

raub@desktop:~$

but I do not like the order; it does not flow right in my eyes. So I am going to change both to look more like Linux.

First let's see how those prompts are defined in both machines; that is found by looking at the content of the environment variablePS1:

bash-3.2$ echo $PS1
\s-\v\$
bash-3.2$

and

littleguy:~ raub$ echo $PS1
\h:\W \u\$
littleguy:~ raub$

From the official gnu page on controlling the Bash prompt, we know that

\h : hostname
\s : shell name
\u : username
\v : shell version
\w : current working directory
\$ : the generic symbol for users (as opposite to # for root)

Based on the above, what I really want is something like PS1="\u@\h:\w\$ ". So let's try it out without permanently committing to it

bash-3.2$ PS1="\u@\h:\w\$ "
raub@slowmac:~$

That seems to be exactly what I want. And, since it only exists in memory (i.e. did not commit it), if I did not like it, restarting the shell or opening a new tab in iTerm would brought it back to the original config. So, how do we make it permanent? In the slowmac, we have

raub@slowmac:~$ cat ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/opt/local/bin:/opt/local/sbin

export PATH

raub@slowmac:~$

So we might as well put it in the .bashrc file the lazy way.

cat >> ~/.bashrc << "EOF"
PS1="\u@\h:\w\$ "
EOF

Interestingly enough, the MacBook Air does not have a ~/.bash_profile. Maybe the slowmac did not have it either and I added it because I was developing programs in it a while ago. No problem, we can solve the MacBook Air problem by creating a ~/.bash_profile or just appending the proper lines to it, which is done the same was as we did to add the prompt lines to the slowmac's .bashrc:

cat >> ~/.bash_profile << "EOF"

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
EOF

Why do we put the prompt in the .bashrc instead of .bash_profile. You see, while ~/.bash_profile is executed when you login, ~/.bashrc is run whenever you start a new shell. By having our prompt defined in ~/.bashrc and then having ~/.bashrc called by ~/.bash_profile we cover our bases.

I am happy, but if you want to make your prompt fancier with colour and cows (really), I would suggest you to check a few more articles such as

In addition to the gnu page I mentioned above. Incidentally, everything I mentioned above should work with any Linux and UNIX distro using bash

Monday, August 27, 2018

VMWare ESXI Gripes: service-control error message not particularly useful

I posted this at the vmware support forum, whose sophisticated text processing interface decided that it knew better than me and reimagineered my pure html post into a rather flat one. I feel like venting and will do so by posting here what it should have looked like there. Note that right now it is just a bunch of WTF-like questions.

We begin this by stating the ESXi cluster in question uses Windows vcenter server; that is not by my choice. I would rather use the appliance

C:\Users\raub> "C:\Program Files\VMware\vCenter Server\bin\service-control" --status --all
Running:
 VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS rhttpproxy vmon vmonapi vmware-cis-config vmware-license vmwareServiceControlAgent
Stopped:
 EsxAgentManager VMWareCAMService VServiceManager content-library mbcs vPostgres vapiEndpoint vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vmware-perfcharts vpxd vpxd-svcs vsan-health vsphere-ui vspherewebclientsvc

C:\Users\raub>

C:\Users\raub> "C:\Program Files\VMware\vCenter Server\bin\service-control" --start vspherewebclientsvc
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsphere-client...
Error executing start on service vsphere-client. Details {
    "detail": [
        {  
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'vsphere-client'",
            "args": [
                "vsphere-client"
            ]
        }
    ],
    "resolution": null,
    "problemId": null,
    "componentKey": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'vsphere-client'",
            "args": [
                "vsphere-client"
            ]
        }
    ],
    "resolution": null,
    "problemId": null,
    "componentKey": null
}

C:\Users\raub>

I understand that "An error occurred while starting service 'vsphere-client'", but what is it? Maybe the log file is more helpful. https://kb.vmware.com/s/article/2121043 claims log dir is

C:\ProgamData\VMware\vCenterServer\logs\vsphere-client\logs\

But I can see lots of directories in there but the log one:

C:\Users\raub>dir  "C:\Program Files\VMware\vCenter Server\"
 Volume in drive C has no label.
 Volume Serial Number is F020-F58F

 Directory of C:\Program Files\VMware\vCenter Server

05/21/2018  07:36 PM    <DIR>          .
05/21/2018  07:36 PM    <DIR>          ..
05/21/2018  07:11 PM    <DIR>          apachetomcat
05/21/2018  07:15 PM    <DIR>          autodeploy
05/21/2018  07:17 PM    <DIR>          bin
05/21/2018  07:13 PM    <DIR>          cis-license
05/21/2018  07:10 PM    <DIR>          cis_upgrade_runner
05/21/2018  07:13 PM    <DIR>          cm
05/21/2018  07:10 PM    <DIR>          common-jars
05/21/2018  07:10 PM    <DIR>          common-libs
05/21/2018  07:15 PM    <DIR>          content-library
05/21/2018  07:15 PM    <DIR>          eam
05/21/2018  07:36 PM    <DIR>          eula
05/21/2018  07:12 PM    <DIR>          fips
05/21/2018  07:21 PM    <DIR>          firstboot
05/21/2018  07:15 PM    <DIR>          imagebuilder
05/21/2018  07:10 PM    <DIR>          jmemtool
05/21/2018  07:10 PM    <DIR>          jre
05/21/2018  07:12 PM    <DIR>          jre_ext
05/21/2018  07:15 PM    <DIR>          mbcs
05/21/2018  07:13 PM    <DIR>          netdump
05/21/2018  07:10 PM    <DIR>          openSSL
04/09/2018  01:24 PM         7,398,602 open_source_license.txt
05/21/2018  07:16 PM    <DIR>          perfcharts
05/21/2018  07:11 PM    <DIR>          python
05/21/2018  07:16 PM    <DIR>          python-modules
05/21/2018  07:13 PM    <DIR>          rhttpproxy
05/21/2018  07:12 PM    <DIR>          ruby
05/21/2018  07:15 PM    <DIR>          rvc
05/21/2018  07:13 PM    <DIR>          sca
05/21/2018  07:09 PM    <DIR>          TlsReconfigurator
05/21/2018  07:13 PM    <DIR>          vapi
04/09/2018  01:24 PM            25,214 vcs.ico
05/21/2018  07:14 PM    <DIR>          virgo
05/21/2018  07:12 PM    <DIR>          visl-integration
05/21/2018  07:12 PM    <DIR>          vmafdd
05/21/2018  07:12 PM    <DIR>          vmcad
05/21/2018  07:15 PM    <DIR>          vmcamd
05/21/2018  07:12 PM    <DIR>          vmdird
05/21/2018  07:13 PM    <DIR>          vmdns
05/21/2018  07:13 PM    <DIR>          vmon
05/21/2018  07:15 PM    <DIR>          vmsyslogcollector
05/21/2018  07:13 PM    <DIR>          VMware Identity Services
05/21/2018  07:11 PM    <DIR>          vmware-sasl
04/25/2018  01:20 PM    <DIR>          vmware-sps
05/21/2018  07:13 PM    <DIR>          vmware-sso
05/21/2018  07:14 PM    <DIR>          vPostgres
05/21/2018  07:25 PM    <DIR>          vpxd
05/21/2018  07:14 PM    <DIR>          vpxd-svcs
05/21/2018  07:32 PM    <DIR>          vsan-health
05/21/2018  07:34 PM    <DIR>          vsm
05/21/2018  07:16 PM    <DIR>          vsphere-client
05/21/2018  07:17 PM    <DIR>          vsphere-ui
               2 File(s)      7,423,816 bytes
              51 Dir(s)  69,727,727,616 bytes free

C:\Users\raub>

C:\Users\raub>dir  "C:\Program Files\VMware\vCenter Server\logs"
 Volume in drive C has no label.
 Volume Serial Number is F020-F58F

 Directory of C:\Program Files\VMware\vCenter Server

File Not Found

C:\Users\raub>

Where can I find where vcenter thinks the log files are at?

If you want to see how the ticket looks like at vmware, https://communities.vmware.com/message/2796432#2796432. Try to read the version posted at vmware and you will understand why I am frustrated.

Saturday, June 02, 2018

Finding disk space hogs in a Windows server/workstation

If you have any doubts, we will be doing it from the command line. Just want to put that out before we start. Also, it turned out this is a long and boring article; deal with it.

So, where were we? Disk space and what is using it. That is a problem common to all OS: you have a partition without infinite disk space (sorry ZFS, it happens sometimes) and is running out of space:

Sometimes it is a careless user; some OS allow you to tell non-system programs, like the ones run by a user, can only use up to 95% of the disk. This way we have some space to fix things.
Sometimes it is actually a program being run as a system/root/admin account, which is trouble since it can use up the entire partition.

If the machine in question is a server, a Windows server since that is what we wrote in the name of the article, we might not be able to just ignore it; others will be affected by this. So, how to take care of this problem? The lazy fix is to throw more space at it and move on. The proper solution is to find out who is hoarding all the space and why. I would like to talk about doing the right thing.

The standard Windows approach would be to search for some app online, which must have a graphics interface and ideally from some site with a name like "finddiskusage.com" because such domain names do inspire confidence, right? Specially when the site's text is pretty much "You do not know what is using your disk space? Click here to download the solution!" Any relationship with a phishing email is merely coincidental.

So, after downloading this shady program from the suspicious website, we then install it and make sure to turn off the firewall and run it with admin rights. And after it does what we hope it is supposed to do, we then take a screenshot of the output and paste it to our documentation.

I do not know about you but I really do not like to install programs in any server, be it windows/linux/mac/solaris/aix/whatever. I think they should have only the bare minimum to do their job; you should see my Linux servers. Since I am the one writing this article, I will put my dictator hat and look for something that fits my style.

In Unix in general and Linux as a special case there is a program called du which allows you to check the disk usage at a given location. You can be short and only show, say, how much all the files and directories (folders in Windows) inside a given directory, or go recursively and show detailed views for every single directory inside the original one. Output is text, which means you can feed it to something else like sort or some program that will make a decision based on the data.

It would be really cool if there was something like that in Windows. One can dream...

Thing is, we do not have to wait for unicorns and fairies to come up with a solution. Nor we have to reinvent the wheel. You see,

There is something like that natively for windows. I do like cygwin but that requires installing yet another collection of packages that need to be patched and upgraded. Kinda wasteful if all you want is little du. I believe less is more.
It is called du just like the unix one.
You do not need to look for it in some shady or just compromised site. You can find it right at Microsoft as part of the sysinternals package(s).
You do not even need to install anything. Just put its directory somewhere you want to use it, including a USB or network drive, and run it from there.

Not bad if you ask me. Enough talk, let's use it.

Using du

The most common way I use du in Windows is like I use in Linux:

C:\Documents and Settings\raub>"\Documents and Settings\raub\My Documents\DU\du.exe" -l 1 \windows >> du.log

Du v1.5 - report directory disk usage
Copyright (C) 2005-2013 Mark Russinovich
Sysinternals - www.sysinternals.com


C:\Documents and Settings\raub>

Ok, it is old but it is not like it is getting more and more useless features to make it bloated. Like interfacing with your bluetooth-enabled IoT-based massage chair. Let me show it in action with a real (!) example: at work I have a Windows 10 vm for desktop. And as you can see, it ran out of disk disk space:

That's not much free space left! If you know Windows, it will get really slow when its boot/OS disk is that full. I am going to assume I should first check the Users directory. If I am wrong, I would then check the Windows one, First I would like to make a point I will be running du.exe off a network fileshare, strongbadia

PS C:\Users\raub> ls \\spacemoose\users\raub\bin


    Directory: \\strongbadia\users\raub\bin


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        5/21/2018   9:32 AM         169072 du.exe
-a----        5/21/2018   9:32 AM         191616 du64.exe
-a----        6/28/2018   4:06 PM            543 GetDirSize4DateRange.ps1
-a----        7/18/2017  10:12 AM         854072 putty.exe


PS C:\Users\raub>

You are now getting to learn a few secret things about me! Yes I have a bin dir. Inside it you can see the du.exe and du64.exe. Both are very tiny compared to the crazy GUI programs you can get off suspicious sites to do the very same thing. And, that is all you need: those two files. Well, I will be running du.exe even though my guest is a 64bit windows vm. Because I can. So, let's see what is in the root dir for the users dir:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

      31,480  c:\users\bob.adm
       2,665  c:\users\Default
  51,390,386  c:\users\raub
     629,717  c:\users\raub.adm
       2,727  c:\users\raub.tst
       2,768  c:\users\Public
     129,592  c:\users\windows-user
Files:        2429983
Directories:  9581
Size:         54,003,166,071 bytes
Size on disk: 59,917,407,544 bytes

PS C:\Users\raub>

Man! My homedir is full of junk. What is that and where is it? Let's check it:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\raub\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

  46,023,020  c:\users\raub\AppData
           0  c:\users\raub\Contacts
           1  c:\users\raub\Desktop
           3  c:\users\raub\dev
   1,836,295  c:\users\raub\Documents
   3,551,535  c:\users\raub\Downloads
           0  c:\users\raub\eqlgroupmgr
           0  c:\users\raub\Favorites
           1  c:\users\raub\Links
           0  c:\users\raub\Music
           0  c:\users\raub\OneDrive
       5,012  c:\users\raub\Pictures
           0  c:\users\raub\Saved Games
           3  c:\users\raub\Searches
           0  c:\users\raub\Videos
Files:        2425559
Directories:  3976
Size:         52,663,166,132 bytes
Size on disk: 58,521,224,472 bytes

PS C:\Users\raub>

AppData! A conveniently normally invisible source of many out of space drives. Not to du it is. Let's keep going in:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\raub\appdata\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

  45,780,916  c:\users\raub\appdata\Local
       2,588  c:\users\raub\appdata\LocalLow
     198,212  c:\users\raub\appdata\Roaming
Files:        2424970
Directories:  3898
Size:         47,085,278,621 bytes
Size on disk: 52,935,700,880 bytes

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\raub\appdata\local\
[...]
PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 C:\users\raub\appdata\local\Microsoft\Windows\INetCac
he\\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

         239  c:\users\raub\appdata\local\microsoft\windows\inetcache\Content.MSO
         878  c:\users\raub\appdata\local\microsoft\windows\inetcache\Content.Outlook
       5,207  c:\users\raub\appdata\local\microsoft\windows\inetcache\Content.Word
      18,852  c:\users\raub\appdata\local\microsoft\windows\inetcache\IE
  39,825,165  c:\users\raub\appdata\local\microsoft\windows\inetcache\Low
           0  c:\users\raub\appdata\local\microsoft\windows\inetcache\Virtualized
           0  c:\users\raub\appdata\local\microsoft\windows\inetcache\WebTempDir
Files:        2410022
Directories:  55
Size:         40,806,752,248 bytes
Size on disk: 46,635,979,024 bytes

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 C:\users\raub\appdata\local\Microsoft\Windows\INetCache\low\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

  39,819,877  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\IE
Files:        2408915
Directories:  34
Size:         40,780,969,849 bytes
Size on disk: 46,606,749,696 bytes

PS C:\Users\raub>

That smells like that other nemesis of web browsers: Internet Explorer or Edge. What is inside that dir?

PS C:\Users\raub> ls C:\users\raub\appdata\local\Microsoft\Windows\INetCache\Low\IE\
PS C:\Users\raub> dir C:\users\raub\appdata\local\Microsoft\Windows\INetCache\Low\IE\
PS C:\Users\raub>

WTF? Why can't I see what is inside it? Let me du inside it:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 C:\users\raub\appdata\local\Microsoft\Windows\INetCache\low\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

   1,237,378  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\10EAXOSV
   1,246,204  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\145KIAQM
   1,235,782  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\2F8FLUJD
   1,241,300  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\2VB3GL0K
   1,250,215  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\5BZQD406
   1,244,360  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\5IVFOAV9
[...]
   1,251,347  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\TQ26RPSG
   1,236,991  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\VQH18R7G
   1,240,357  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\VYHW6URF
   1,239,508  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\XYKKE45S
Files:        2408912
Directories:  33
Size:         40,775,554,697 bytes
Size on disk: 46,601,322,496 bytes

PS C:\Users\raub>

Before you ask, I am using a powershell window, where ls and dir behave the same. I came from unix so you can understand which one I prefer. So we have a ton (33) of stupid cache folders that the browser could not be bothered to delete after it quit. Thanks, Microsoft, for not cleaning after itself. Really. And, everything below c:\users\raub\appdata\local\microsoft\windows\inetcache\low is hidden (?). Alright then, off it goes! Command line cares not about hidden paths! Note: get-help can be seen as the powershell equivalent of the Unix/linux man.

PS C:\Users\raub> get-help rm

NAME
    Remove-Item

SYNOPSIS
    Deletes files and folders.


SYNTAX
    Remove-Item [-Confirm] [-Credential ] [-Exclude ] [-Filter ] [-Force] [-Include
    ] -LiteralPath  [-Recurse] [-Stream ] [-UseTransaction] [-WhatIf]
    []

    Remove-Item [-Path]  [-Confirm] [-Credential ] [-Exclude ] [-Filter ]
    [-Force] [-Include ] [-Recurse] [-Stream ] [-UseTransaction] [-WhatIf] []
   
    Remove-Item [-Stream ] []
   

DESCRIPTION
    The Remove-Item cmdlet deletes one or more items. Because it is supported by many providers, it can delete many
    different types of items, including files, folders, registry keys, variables, aliases, and functions.
    In file system drives, the Remove-Item cmdlet deletes files and folders.
   
    If you use the Stream dynamic parameter, it deletes the specified alternate data stream, but does not delete the
    file.
   
    Note: This custom cmdlet help file explains how the Remove-Item cmdlet works in a file system drive. For
    information about the Remove-Item cmdlet in all drives, type "Get-Help Remove-Item -Path $null" or see Remove-Item
    at http://go.microsoft.com/fwlink/?LinkID=113373.
   

RELATED LINKS 
    Online version: http://technet.microsoft.com/library/jj628241(v=wps.630).aspx
    Remove-Item (generic); http://go.microsoft.com/fwlink/?LinkID=113373
    FileSystem Provider
    Clear-Content
    Get-Content
    Get-ChildItem
    Get-Content
    Get-Item
    Remove-Item
    Set-Content
    Test-Path


REMARKS
    To see the examples, type: "get-help Remove-Item -examples".
    For more information, type: "get-help Remove-Item -detailed".
    For technical information, type: "get-help Remove-Item -full".
    For online help, type: "get-help Remove-Item -online"


PS C:\Users\raub> rm -force -recurse C:\users\raub\appdata\local\Microsoft\Windows\INetCache\low\IE

Note that since we are running this from the command line, I did not have to do the usual screenshot Windows blog and articles love so much. I could cut and paste the real output and put it here. I could also have piped it into some other script to use the output for its nefarious uses. FYI, the above command has been running for 38 minutes now and has not finished yet.

So, what I have done above in this real example can be also used with servers since

We are using Microsoft published program.
The program is self-contained and requires no installation.
The program fears no hidden directories.
The program has very small footprint.
The program can be run locally, off a USB, or from a network drive.

If you can put up with not having a cute window with some animation, I think sysinternals' version of du is a nice tiny add-on to a Windows server manager arsenal.

Saturday, April 28, 2018

Using sed Capture Groups (Linux/Mac)

This will be a short one and belongs to the TIL bin: until 2 days ago I did not even know about capture groups and how to use it. So, I knew how to replace a matching string/pattern in sed

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v/swallow_speed/' 
As of today swallow_speed=23kph at STD
bash-3.2$

And how to replace from a given pattern all the way to the end of the line. Or starting from the beginning of the line to said pattern:

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*/swallow_speed=42/' 
As of today swallow_speed=42
bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/^.*swallow_v=/Can you believe that swallow_speed=/' 
Can you believe that swallow_speed=23kph at STD
bash-3.2$

If you are curious, the .* in the search pattern means "any character or list of characters here, be it zero or a lot of characters". The dot (.) does the any part and the asterisk (*) the how many. I think this is from regex, but don't quote me on that. Fine, what if I want to replace everything between two patterns, but leaving the second pattern alone? Tricky. You see, replacing everything between the two patterns, inclusive is not that hard

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*k/swallow_v=25/'
As of today swallow_v=25ph at STD
raub@desktop:~$

But to preserve the second pattern we need to use the Capture Groups mentioned in the title of this article. And that makes sense because if it is on the title I better use it. So, we are supposed to surround the capture group pattern with parenthesis and then we can refer to them. If it does not make sense, I too was confused, so let's keep on using our test string:

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*(k)/swallow_v=25$1/'
As of today swallow_v=23kph at STD
raub@desktop:~$

Er, it does not seem to have worked according to the plan. In fact, it was supposed to at least grab swallow_v=23k but as you can see it did not find the pattern. Is $1 the proper way to output the captured string? Going nowhere slowly.

After much soul searching, I found the -e requires the parenthesis to be escaped. And, the capture group pattern is output using \1 instead of $1.So we try again:

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e "s/swallow_v=.*\(k\)/swallow_v=25\1/"
As of today swallow_v=25kph at STD
raub@desktop:~$

much better!

What about the Mac? Same thing (the bash-3.2$ you have seen all day is it; the raub@desktop:~$ is the Linux box):

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*\(k\)/swallow_v=25\1/' 
As of today swallow_v=25kph at STD
bash-3.2$

I am not going to say it is perfect though:

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*\([[:blank:]]\)/swallow_v=25\1/' 
As of today swallow_v=25 STD
bash-3.2$ echo "As of today swallow_v=23kph at    STD" | sed -e 's/swallow_v=.*\([[:blank:]]\)/swallow_v=25\1/' 
As of today swallow_v=25 STD
bash-3.2$

References

A site I think has lots of interesting sed examples.

Tuesday, January 14, 2020

Enter the Grubby

Final thoughts

Tuesday, November 12, 2019

Now why someone would want to do that?

Saturday, November 09, 2019

Create Key

Convert private key to something Rsync4Android can use

Copy public and private keys to the proper locations

Test it

Saturday, June 22, 2019

Thursday, May 23, 2019

Get the packages

Coding, at last!

Friday, May 03, 2019

Why

The Card

How

Finding the card

Notes

Handing out the card to the guest

PCIe is more demanding

References

Wednesday, January 16, 2019

Monday, September 24, 2018

Why?

Using ldapsearch

Using ldapmodify

Why I do like ldapmodify to edit LDAP/AD

What about Mac/OSX?

Wednesday, August 29, 2018

Monday, August 27, 2018

Saturday, June 02, 2018

Using du

Saturday, April 28, 2018

References

About Me

Followers

Blog Archive