Wednesday, September 28, 2016

DHCP renewal, dynamic DNS, and changing network cards

Another adventure that started with a simple and quick plan and then felt into the rabbit hole! If you did not get the hint, this is one of those examples of why proper documentation is important. Or, what happens when I rely on my memory...

I built a windows VM inside VMWare ESXi a while ago called dr-zaius. When I did the deed, I setup it to use an emulated Intel E1000-series ethernet card. Fast forward and I learned VMWare recommends to use their native NIC, vmxnet3. I am fine with that since that is what I do in KVM, namely use the native interface to the hypervisor and it does result in increased performance. So let's go over the steps for doing the deed.

Note: The following assumes you have the rights to edit a vm client's configuration in ESXi. If not, just contact who does and explain what you want to accomplish.

The Steps

  1. We are going to be lazy and use the vsphere client. Which we will be using while the vm is up and running. When we select the Edit Virtual Machine Settings option, we have the option to add an additional interface, so let's add it and later on delete the old one.

    By default in this version of the vsphere client, we are offered the E1000E emulated interface. And that is not what we want so we need to change the interface as shown on the right (they are calling it an adapter) and select the vmxnet3 one. Don't forget to select the proper VLAN -- Network Label in this dialog box -- while you are there. Also notice by default it will be connected at power on, which is what we want in this article. And then hit the Next button until it finally gets to the screen saying the deed is done.

  2. As soon as we do that, login to the windows machine via console. It will tell us it detected a new device and will start with its driver installation dance. Once that is done, this vm client will end up with two network interfaces. Now we need to set it up.

  3. The best thing to do is disable the old interface -- you are accessing it from console -- inside the vm client, just as you normally do in a physical host. If you use static IP, you might want to delete the IP or change it to a bogus address so the other interface will not get pissed (Windows does like to be very helpful).

  4. Now configure the new interface, be it to use static IP or DHCP. In this example we will be using DHCP.

  5. After testing everything works, we can then shut the vm client down at our convenience and remove the old network interface using the vmsphere client.

And that should be the end of this article. Maybe in another article we can talk about doing the same deed in KVM (and whatever else I get my hands on).

But things did not happen according to the plan

I am not very good with words, so let me just show it:

raub@desktop:~$ ping dr-zaius
PING dr-zaius.in.example.com (192.168.42.105) 56(84) bytes of data.
From desktop.in.example.com (192.168.42.102) icmp_seq=1 Destination Host Unreachable
From desktop.in.example.com (192.168.42.102) icmp_seq=2 Destination Host Unreachable
From desktop.in.example.com (192.168.42.102) icmp_seq=3 Destination Host Unreachable
^C
--- dr-zaius.in.example.com ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3036ms
pipe 3
raub@desktop:~$

But when I login to dr-zaius from a console, I get the IP for its new interface is 192.168.42.126, which seems to work fine when I probulate using netcat:

raub@desktop:~$ nc -v 192.168.42.126 3389
Connection to 192.168.42.126 3389 port [tcp/*] succeeded!
^C

raub@desktop:~$

Er, what is going on? So, when I went to the dhcp/dns server (nameserver) to check. It uses the ISC DHCPD and BIND (I never knew it was an acronym until I wrote this article. Who said you can't learn anything in the web?) to do its thing, and they are configured to do dynamic DNS. So, we cannot just grab the zone files

[root@nameserver ~]# ls /var/named/data/in.example.com.*
/var/named/data/in.example.com.db
/var/named/data/in.example.com.db.jnl
/var/named/data/in.example.com.rev.db
/var/named/data/in.example.com.rev.db.jnl
[root@nameserver ~]#

and edited them. The telltale hint we are doing some dynamic DNS'ing is the journal (.jnl) files. Let's first stop the DNS updates in both forward and reverse zones.

rndc freeze in.example.com
rndc freeze 42.168.192.in-addr.arpa

We now can take a look at the zone files. Here is the entry for dr-zaius in the forward zone file:

dr-zaius                A       192.168.42.105
                        TXT     "316335b72f00f1bf82eb484894d5263cdf"

I do not know about you but it sure seems like it still thinks dr-zaius's IP is ,tt>192.168.42.105. so, we delete those two lines (I like to also update the serial entry on the top of the file just in case). The reverse zone is a bit easier as it has only one line

126                     PTR     dr-zaius.in.example.com.

to be deleted. Once those are done, we can let dynamic DNS take place in that zone again.

rndc thaw 42.168.192.in-addr.arpa
rndc thaw in.example.com

The log seems to show that it is now properly associating the hostname to the IP

Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#48383: updating zone 
'in.example.com/IN': adding an RR at 'dr-zaius.in.example.com' A
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#48383: updating zone 
'in.example.com/IN': adding an RR at 'dr-zaius.in.example.com' TXT
Sep 23 13:10:47 nameserver dhcpd: Added new forward map from dr-zaius.in.example.com 
to 192.168.42.126
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#51133: signer 
"dhcpupdate" approved
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#51133: updating zone 
'42.168.192.in-addr.arpa/IN': deleting rrset at '126.42.168.192.in-addr.arpa' PTR
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#51133: updating zone 
'42.168.192.in-addr.arpa/IN': adding an RR at '126.42.168.192.in-addr.arpa' PTR
Sep 23 13:10:47 nameserver dhcpd: added reverse map from 126.42.168.192.in-addr.arpa. 
to dr-zaius.in.example.com.
Sep 23 13:10:47 nameserver dhcpd: DHCPREQUEST for 192.168.42.126 from 
00:0c:29:84:95:8c (dr-zaius) via eth0
Sep 23 13:10:47 nameserver dhcpd: DHCPACK on 192.168.42.126 to 00:0c:29:84:95:8c 
(dr-zai$s) via eth0

So let's try again. nameserver seems to be able to resolve the hostname and IP:

[root@nameserver ~]# nslookup 192.168.42.126
Server:         127.0.0.1
Address:        127.0.0.1#53

126.0.0.10.in-addr.arpa name = dr-zaius.in.example.com.

[root@nameserver ~]# nslookup 192.168.42.126
Server:         127.0.0.1
Address:        127.0.0.1#53

126.0.0.10.in-addr.arpa name = dr-zaius.in.example.com.

[root@nameserver ~]#

Now we go back to the desktop and try again

raub@desktop:~$ nslookup dr-zaius
Server:         192.168.4.1
Address:        192.168.4.1#53

Non-authoritative answer:
Name:   dr-zaius.in.example.com
Address: 192.168.42.105

raub@desktop:~$

What is going on here? I thought I had taken care of this! And who is this 192.168.4.1? nameserver is at 192.168.55.10! We probably need to see which DNS servers desktop is using

raub@desktop:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.4.1
nameserver 192.168.55.10
nameserver 192.168.4.1
search in.example.com mgmt.example.com
raub@desktop:~$

A Candle just lit above mt head! Now things make sense! You see, desktop is dual homed in both 192.168.42.0/24 and 192.168.4.0/24 networks, the latter is called mgmt.example.com. desktop also gets its IPs on both networks using dhcp and as a result its nameservers. I do not know why /etc/resolv.conf has two entries for 192.168.4.1, the nameserver as seen in 192.168.4.0/24, but the point is that it is the first nameserver. And, this is not an IP used by the real nameserver. You see, the Juniper router that is doing all this routing is configured to be a forwarding DNS server for a few networks, 192.168.4.0/24 included. And it seems it has not updated its cache. Let's see if I am right:

raub@desktop:~$ nslookup dr-zaius 192.168.4.1
Server:         192.168.4.1
Address:        192.168.4.1#53

Non-authoritative answer:
Name:   dr-zaius.in.example.com
Address: 192.168.42.105

raub@desktop:~$ nslookup dr-zaius 192.168.55.10
Server:         192.168.55.10
Address:        192.168.55.10#53

Name:   dr-zaius.in.example.com
Address: 192.168.42.126

raub@desktop:~$ 

So, all I have to do is wait a bit for the router to catch up.

I feel better now

PS: if you feel that I brushed over the setup of DHCPD and BIND, or how to configure a Juniper router, you are quite right. Those topics, while interesting, were not what we are trying to solve here. And what is that since it seems changing the Windows vm client interface in VMWware is not it? Actually, it is. But things sometimes do not happen according to the plan and we have to figure out what is going on. In this case, the culprit was DNS lease renew times at the DNS primary and forwarding (the router in my example) servers. If you think every time I set it up it is all rainbows and unicorns and things work perfectly at the first time, I have some news for you sunshine.

Wednesday, September 14, 2016

Setting up atftp in (ubuntu) Linux

I have a switch whose firmware was in dire need of being updated. Thing is, the switch will only take the upgrade firmware if it is offered by a tftp server, don't ask me why. And that is something I tend not to have running; last time I used one was to network boot Solaris boxes. Sounds like an excuse to write another article.

Just to be different I will be deploying this on a Ubuntu Linux host instead of a CentOS/RedHat one as I usually do. Why? Doing the same all the time gets boring quickly. So, let's see which versions of tftpd we can pick and choose:

raub@desktop:~$ apt-cache search tftpd
tftpd-hpa - HPA's tftp server
atftpd - advanced TFTP server
libnet-tftpd-perl - Perl extension for Trivial File Transfer Protocol Server
tftpd - Trivial file transfer protocol server
uec-provisioning-tftpd - the UEC Provisioning TFTP server
raub@desktop:~$

After careful scientific consideration, the atftpd one sounds more interesting (Multi-threaded TFTP server implementing extension and multicast), so we will pick that one. I think this is the part of the show in which we go through the steps to do the deed:

  1. A good place to start is to install it. I like command line and apt-get, so I think

    sudo apt-get install atftpd

    should do the trick. Of course you can use aptitude or the GUI. But I am lazy.

  2. Traditionally the directory used to put stuff that will be shared by tftp is /tftpboot, but current practice is to use /srv/tftp. In fact, atftpd does create /srv/tftp for you. For the same of showing how to customize things, let's say we want to be old school. And that means creating /tftpboot ourselves:

    sudo mkdir /tftpboot
    sudo chmod -R 777 /tftpboot
    sudo chown -R nobody:nogroup /tftpboot
  3. This might be a good time to put the files we want to be shared, in this example a file called image.bin in /tftpboot

  4. We need to configure it by editing /etc/default/atftpd. Here is what mine looks like

    raub@desktop:~$ cat /etc/default/atftpd
    USE_INETD=false
    OPTIONS="--tftpd-timeout 300 --retry-timeout 5 --port=69 --mcast-port 1758 
    --mcast-addr 192.168.0.0-255 --mcast-ttl 1 --maxthread 100 --verbose=7 /tftpboot"
    raub@desktop:~$

    where:

    • --port=69 we are forcing it to use the default tftp port
    • --mcast-addr 192.168.0.0-255 specifies the multicast address range we will be using. Being lazy, I am using the entire 192.168.0.0/24 range
    • /tftpboot is the directory we will be sharing as explained above. By default the config file specifies /srv/tftp which means if we put our file in /tftpboot we would get a message like

      Sep 13 13:54:00 desktop atftpd[28045]: File /srv/tftp/image.bin 
      not found

      when we try to fetch the file

    • --verbose=7 is the highest amount of verbose we can use. By default its value is set to 5.

    Once it starts properly (service atftpd start should be enough to start it), you should see something like

    raub@desktop:~$ ps -ef|grep ftp
    raub     16510 11504  0 15:26 pts/11   00:00:00 grep ftp
    nobody   28161     1  0 Sep13 ?        00:00:00 /usr/sbin/atftpd --daemon --tftpd-timeout 300 --retry-timeout 5 --port=69 --mcast-port 1758 --mcast-addr 192.168.0.0-255 --mcast-ttl 1 --maxthread 100 --verbose=7 /tftpboot
    raub@desktop:~$

How to get the file using tftp is beyond this discussion because it depends on your OS and the tftp client you are using. For instance the switch might show a webpage where you can configure the name of the tftp server -- 192.168.0.102 in my example -- and the name of the file you want to grab. What is more interesting is to see how the entire enchilada from we started the tftp server until we get the file image.bin looks like. By default (can be configured) we would see that in /var/log/syslog:

Sep 13 13:55:52 desktop systemd[1]: Starting LSB: Launch atftpd server...
Sep 13 13:55:52 desktop atftpd[28160]: Advanced Trivial FTP server started (0.7)
Sep 13 13:55:52 desktop atftpd[28153]: Starting Advanced TFTP server: atftpd.
Sep 13 13:55:52 desktop atftpd[28161]:   running in daemon mode on port 69
Sep 13 13:55:52 desktop atftpd[28161]:   logging level: 7
Sep 13 13:55:52 desktop atftpd[28161]:   directory: /tftpboot/
Sep 13 13:55:52 desktop atftpd[28161]:   user: nobody.nogroup
Sep 13 13:55:52 desktop atftpd[28161]:   log file: syslog
Sep 13 13:55:52 desktop atftpd[28161]:   not forcing to listen on local interfaces.
Sep 13 13:55:52 desktop atftpd[28161]:   server timeout: Not used
Sep 13 13:55:52 desktop atftpd[28161]:   tftp retry timeout: 5
Sep 13 13:55:52 desktop atftpd[28161]:   maximum number of thread: 100
Sep 13 13:55:52 desktop atftpd[28161]:   option timeout:   enabled
Sep 13 13:55:52 desktop atftpd[28161]:   option tzise:     enabled
Sep 13 13:55:52 desktop atftpd[28161]:   option blksize:   enabled
Sep 13 13:55:52 desktop atftpd[28161]:   option multicast: enabled
Sep 13 13:55:52 desktop atftpd[28161]:      address range: 192.168.0.0-255
Sep 13 13:55:52 desktop atftpd[28161]:      port range:    1758
Sep 13 13:55:52 desktop systemd[1]: Started LSB: Launch atftpd server.
Sep 13 13:55:59 desktop atftpd[28161]: socket may listen on any address, including broadcast
Sep 13 13:55:59 desktop atftpd[28161]: Creating new socket: 192.168.0.102:45115
Sep 13 13:55:59 desktop atftpd[28161]: Serving image.bin to 192.168.0.3:2295
Sep 13 13:56:03 desktop atftpd[28161]: End of transfer
Sep 13 13:56:03 desktop atftpd[28161]: Server thread exiting

The underline on the entry when the file image.bin is transferred was added to make it easy to see. And that is pretty much all I had to do. Once file was transfered, I stopped atftpd and then remove it

sudo apt-get remove --purge atftpd

because I do not like to have unused services running.

PS: Always backup your switch/network device's config before upgrading its firmware in case it reverts to default as part of the upgrade process. Guess who forgot to do that? Good thing I had good notes and could reconfigure it using the time-honored cut-n-paste technique