Discovering (tagged only?) vlans using tcpdump

Why to use VLANs? well, there is the part of having a bunch of different devices -- computers, printers, toasters -- networks that are connected to the same switches but in different networks so you can control how/if they can see and talk to each other. And those switches are connected to each other and to routers who need to route traffic between the different networks. And that probably means we are trunking and tagging the vlans between those switches and routers. But, if we now add virtual machines to the mix, the physical machines they are running on -- the virtual servers -- need to be able to connect to all the networks their virtual clients belong to. And that once again is done using trunking and vlan tagging.

What if it is not working properly?

Let me elaborate it by using an example: I recently had a router running OpenWRT which I assumed I had configured to handle multiple vlans and was connected to the rest of the network through a 802.1q trunk; think of it as a router-on-a-stick setup. Now, the router was configured to have one IP -- 192.168.2.1 and 192.168.3.1 to make it easier -- in each of the two vlans, which I shall call vlan2 and vlan3. Problem was I could only reach it through vlan2; nothing on vlan3.

First thing I did was to recheck the switch to see if that port was configured for 802.1q trunk. Not only it was but both vlans were tagged; I did not have an untagged/default vlan in this trunk. So for all practical purposes both vlans should look the same for the router. So why weren't they?

At this point I decided to do some packet sniffing. And, this would be the cue to unleash Wireshark on it. But, we are talking about a little router running a OS designed to fit a resource limited device. Why not see if we can use a command line alternative, like the humble tcpdump? It can show you not only any vlan traffic on the wire but also only traffic on a specific vlan just as it can show traffic related to a specific IP or protocol. For instance, since we really only care about traffic on vlan 3 on interface eth0 we can

root@router:~# tcpdump -i eth0.3
tcpdump: WARNING: eth0.3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes

And then after letting it run for ten minutes, all I got back were tumbleweeds. In the same time frame, if had asked for traffic on vlan 2 it would have filled the screen before I finished this sentence (and, yes, I am a slow typist but still). The logical thing to do now is to see if there is any traffic on the wire that has a 802.1Q tag (vlan):

tcpdump -i eth0 -e -n vlan
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
01:32:27.440782 e2:46:9a:5e:a7:45 > c0:ff:ee:3e:06:eb, ethertype 802.1Q (0x8100)
, length 122: vlan 2, p 0, ethertype IPv4, 192.168.2.1.22 > 192.168.2.249.56762: Flags
 [P.], seq 2026124025:2026124077, ack 2216938037, win 21298, options [nop,nop,TS
 val 346274301 ecr 697366229], length 52
01:32:27.441234 e2:46:9a:5e:a7:45 > c0:ff:ee:3e:06:eb, ethertype 802.1Q (0x8100)
, length 122: vlan 2, p 0, ethertype IPv4, 192.168.2.1.22 > 192.168.2.249.56762: Flags
 [P.], seq 52:104, ack 1, win 21298, options [nop,nop,TS val 346274301 ecr 697366232], length 52
01:32:27.441946 e2:46:9a:5e:a7:45 > c0:ff:ee:3e:06:eb, ethertype 802.1Q (0x8100)
, length 138: vlan 2, p 0, ethertype IPv4, 192.168.2.1.22 > 192.168.2.249.56762: Flags
 [P.], seq 104:172, ack 1, win 21298, options [nop,nop,TS val 346274301 ecr 6973
66232], length 68

Do you see the vlan info on those lines? Let's trim it a bit:

tcpdump -i eth0 -e -n vlan | cut -f 6,10,11 -d ' '
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
^C21 packets captured
29 packets received by filter
0 packets dropped by kernel

So all the traffic we have seen above is for vlan 2. In fact the 192.168.2.1.22 > 192.168.2.249.56762 means someone in 192.168.2.249 is connecting from its port 56762 to port 22 (ssh) on 192.168.2.1, which is the router. And that is indeed the case because that is how I connected to the router!

NOTE: A long time ago there was a bug in tcpdump, which would require you to run it twice,

tcpdump -Uw - | tcpdump -en -r - vlan 

But that is no longer the case.

But, where's vlan 3? That is a great question! At first I thought I had it setup wrong n the router even though both vlans have similar confiugrations, save the network info. And that kept me awake for a few nights.

It turns out that there is a ongoing bug which would not allow me to run more than one tagged vlan. The workaround is to "kick" it by adding a

option enable_vlan4k 1

to /etc/config/network

We feel confident we solved the problem, so instead of listening to all vlans this time we will only listen for vlan 3 traffic:

root@router:~# tcpdump -i eth0.3
tcpdump: WARNING: eth0.3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes
19:15:25.822964 IP 0.0.0.0 > all-systems.mcast.net: igmp query v2
19:15:25.823010 IP6 fe80::e046:9aff:fe5e:a745 > ff02::1: HBH ICMP6, multicast li
stener querymax resp delay: 10000 addr: ::, length 24
19:15:26.568728 ARP, Request who-has 192.168.3.10 tell 192.168.3.1, length 28
19:15:26.569022 ARP, Reply 192.168.3.10 is-at c0:ff:ee:14:25:ad (oui Unknown), l
ength 42
19:15:26.569092 IP 192.168.3.1.32101 > 192.168.3.10.domain: 29094+ PTR? 1.0.0.0.
0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
19:15:26.606872 IP 192.168.3.10.domain > 192.168.3.1.32101: 29094 NXDomain 0/1/0
 (160)
19:15:27.797110 IP6 fe80::c2ff:eeff:fe14:25ad > ff02::1:ff14:25ad: HBH ICMP6, mu
lticast listener reportmax resp delay: 0 addr: ff02::1:ff14:25ad, length 24
19:15:28.014715 IP6 fe80::be5f:f4ff:fe54:d78d > ff02::1:ff54:d78d: HBH ICMP6, mu
lticast listener reportmax resp delay: 0 addr: ff02::1:ff54:d78d, length 24
19:15:28.615536 IP 192.168.3.1.16915 > 192.168.3.10.domain: 15895+ PTR? d.a.5.2.
4.1.e.f.f.f.e.e.f.f.2.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
19:15:28.616059 IP 192.168.3.10.domain > 192.168.3.1.16915: 15895 NXDomain* 0/1/
0 (125)
19:15:28.618233 IP 192.168.3.1.36929 > 192.168.3.10.domain: 54136+ PTR? d.a.5.2.
4.1.f.f.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
19:15:28.646935 IP 192.168.3.10.domain > 192.168.3.1.36929: 54136 NXDomain 0/1/0
 (160)

And that is the kind of output I wanted to see!

Moral of the story: when vlan and their trunks get funky it might be time to unleash some packet analysis tool to figure out where in the network is the problem. And, even though Wireshark is a pretty nice tool, sometimes you can get the work done with something a bit more lightweight.

DHCP renewal, dynamic DNS, and changing network cards

Another adventure that started with a simple and quick plan and then felt into the rabbit hole! If you did not get the hint, this is one of those examples of why proper documentation is important. Or, what happens when I rely on my memory...

I built a windows VM inside VMWare ESXi a while ago called dr-zaius. When I did the deed, I setup it to use an emulated Intel E1000-series ethernet card. Fast forward and I learned VMWare recommends to use their native NIC, vmxnet3. I am fine with that since that is what I do in KVM, namely use the native interface to the hypervisor and it does result in increased performance. So let's go over the steps for doing the deed.

Note: The following assumes you have the rights to edit a vm client's configuration in ESXi. If not, just contact who does and explain what you want to accomplish.

The Steps

1. We are going to be lazy and use the vsphere client. Which we will be using while the vm is up and running. When we select the Edit Virtual Machine Settings option, we have the option to add an additional interface, so let's add it and later on delete the old one.

   By default in this version of the vsphere client, we are offered the E1000E emulated interface. And that is not what we want so we need to change the interface as shown on the right (they are calling it an adapter) and select the vmxnet3 one. Don't forget to select the proper VLAN -- Network Label in this dialog box -- while you are there. Also notice by default it will be connected at power on, which is what we want in this article. And then hit the Next button until it finally gets to the screen saying the deed is done.

2. As soon as we do that, login to the windows machine via console. It will tell us it detected a new device and will start with its driver installation dance. Once that is done, this vm client will end up with two network interfaces. Now we need to set it up.

3. The best thing to do is disable the old interface -- you are accessing it from console -- inside the vm client, just as you normally do in a physical host. If you use static IP, you might want to delete the IP or change it to a bogus address so the other interface will not get pissed (Windows does like to be very helpful).

4. Now configure the new interface, be it to use static IP or DHCP. In this example we will be using DHCP.

5. After testing everything works, we can then shut the vm client down at our convenience and remove the old network interface using the vmsphere client.

And that should be the end of this article. Maybe in another article we can talk about doing the same deed in KVM (and whatever else I get my hands on).

But things did not happen according to the plan

I am not very good with words, so let me just show it:

raub@desktop:~$ ping dr-zaius
PING dr-zaius.in.example.com (192.168.42.105) 56(84) bytes of data.
From desktop.in.example.com (192.168.42.102) icmp_seq=1 Destination Host Unreachable
From desktop.in.example.com (192.168.42.102) icmp_seq=2 Destination Host Unreachable
From desktop.in.example.com (192.168.42.102) icmp_seq=3 Destination Host Unreachable
^C
--- dr-zaius.in.example.com ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3036ms
pipe 3
raub@desktop:~$

But when I login to dr-zaius from a console, I get the IP for its new interface is 192.168.42.126, which seems to work fine when I probulate using netcat:

raub@desktop:~$ nc -v 192.168.42.126 3389
Connection to 192.168.42.126 3389 port [tcp/*] succeeded!
^C

raub@desktop:~$

Er, what is going on? So, when I went to the dhcp/dns server (nameserver) to check. It uses the ISC DHCPD and BIND (I never knew it was an acronym until I wrote this article. Who said you can't learn anything in the web?) to do its thing, and they are configured to do dynamic DNS. So, we cannot just grab the zone files

[root@nameserver ~]# ls /var/named/data/in.example.com.*
/var/named/data/in.example.com.db
/var/named/data/in.example.com.db.jnl
/var/named/data/in.example.com.rev.db
/var/named/data/in.example.com.rev.db.jnl
[root@nameserver ~]#

and edited them. The telltale hint we are doing some dynamic DNS'ing is the journal (.jnl) files. Let's first stop the DNS updates in both forward and reverse zones.

rndc freeze in.example.com
rndc freeze 42.168.192.in-addr.arpa

We now can take a look at the zone files. Here is the entry for dr-zaius in the forward zone file:

dr-zaius                A       192.168.42.105
                        TXT     "316335b72f00f1bf82eb484894d5263cdf"

I do not know about you but it sure seems like it still thinks dr-zaius's IP is ,tt>192.168.42.105. so, we delete those two lines (I like to also update the serial entry on the top of the file just in case). The reverse zone is a bit easier as it has only one line

126                     PTR     dr-zaius.in.example.com.

to be deleted. Once those are done, we can let dynamic DNS take place in that zone again.

rndc thaw 42.168.192.in-addr.arpa
rndc thaw in.example.com

The log seems to show that it is now properly associating the hostname to the IP

Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#48383: updating zone 
'in.example.com/IN': adding an RR at 'dr-zaius.in.example.com' A
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#48383: updating zone 
'in.example.com/IN': adding an RR at 'dr-zaius.in.example.com' TXT
Sep 23 13:10:47 nameserver dhcpd: Added new forward map from dr-zaius.in.example.com 
to 192.168.42.126
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#51133: signer 
"dhcpupdate" approved
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#51133: updating zone 
'42.168.192.in-addr.arpa/IN': deleting rrset at '126.42.168.192.in-addr.arpa' PTR
Sep 23 13:10:47 nameserver named[1536]: client 192.168.55.10#51133: updating zone 
'42.168.192.in-addr.arpa/IN': adding an RR at '126.42.168.192.in-addr.arpa' PTR
Sep 23 13:10:47 nameserver dhcpd: added reverse map from 126.42.168.192.in-addr.arpa. 
to dr-zaius.in.example.com.
Sep 23 13:10:47 nameserver dhcpd: DHCPREQUEST for 192.168.42.126 from 
00:0c:29:84:95:8c (dr-zaius) via eth0
Sep 23 13:10:47 nameserver dhcpd: DHCPACK on 192.168.42.126 to 00:0c:29:84:95:8c 
(dr-zai$s) via eth0

So let's try again. nameserver seems to be able to resolve the hostname and IP:

[root@nameserver ~]# nslookup 192.168.42.126
Server:         127.0.0.1
Address:        127.0.0.1#53

126.0.0.10.in-addr.arpa name = dr-zaius.in.example.com.

[root@nameserver ~]# nslookup 192.168.42.126
Server:         127.0.0.1
Address:        127.0.0.1#53

126.0.0.10.in-addr.arpa name = dr-zaius.in.example.com.

[root@nameserver ~]#

Now we go back to the desktop and try again

raub@desktop:~$ nslookup dr-zaius
Server:         192.168.4.1
Address:        192.168.4.1#53

Non-authoritative answer:
Name:   dr-zaius.in.example.com
Address: 192.168.42.105

raub@desktop:~$

What is going on here? I thought I had taken care of this! And who is this 192.168.4.1? nameserver is at 192.168.55.10! We probably need to see which DNS servers desktop is using

raub@desktop:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.4.1
nameserver 192.168.55.10
nameserver 192.168.4.1
search in.example.com mgmt.example.com
raub@desktop:~$

A Candle just lit above mt head! Now things make sense! You see, desktop is dual homed in both 192.168.42.0/24 and 192.168.4.0/24 networks, the latter is called mgmt.example.com. desktop also gets its IPs on both networks using dhcp and as a result its nameservers. I do not know why /etc/resolv.conf has two entries for 192.168.4.1, the nameserver as seen in 192.168.4.0/24, but the point is that it is the first nameserver. And, this is not an IP used by the real nameserver. You see, the Juniper router that is doing all this routing is configured to be a forwarding DNS server for a few networks, 192.168.4.0/24 included. And it seems it has not updated its cache. Let's see if I am right:

raub@desktop:~$ nslookup dr-zaius 192.168.4.1
Server:         192.168.4.1
Address:        192.168.4.1#53

Non-authoritative answer:
Name:   dr-zaius.in.example.com
Address: 192.168.42.105

raub@desktop:~$ nslookup dr-zaius 192.168.55.10
Server:         192.168.55.10
Address:        192.168.55.10#53

Name:   dr-zaius.in.example.com
Address: 192.168.42.126

raub@desktop:~$ 

So, all I have to do is wait a bit for the router to catch up.

I feel better now

PS: if you feel that I brushed over the setup of DHCPD and BIND, or how to configure a Juniper router, you are quite right. Those topics, while interesting, were not what we are trying to solve here. And what is that since it seems changing the Windows vm client interface in VMWware is not it? Actually, it is. But things sometimes do not happen according to the plan and we have to figure out what is going on. In this case, the culprit was DNS lease renew times at the DNS primary and forwarding (the router in my example) servers. If you think every time I set it up it is all rainbows and unicorns and things work perfectly at the first time, I have some news for you sunshine.