Showing posts with label vlan. Show all posts
Showing posts with label vlan. Show all posts

Saturday, May 18, 2024

Configuring an interface in Linux without(!) nmcli

In ages past, when you wanted to configure a network interface eth0 with MAC CC:00:FF:EE:12:34 with static IP 128.227.3.20 and using uranus (128.227.3.1) as the gateway in Red Hat Linux or its derivatives, you could have something like this

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << 'EOF'
DEVICE="eth0"
BOOTPROTO="static"
HWADDR="CC:00:FF:EE:12:34"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
DHCP_HOSTNAME=vmhost
IPADDR=128.227.3.20
NETMASK=255.255.255.0
GATEWAY=128.227.3.1
EOF

in your notes and cut-n-paste it as needed. The same would work for a vlan trunk (think 802.1q), where the interface using 128.227.3.20 is now associated to tagged vlan 3; all we needed is 3 files.

  • Define the base interface (we could have used uuid instead of MAC address)
    cat /etc/sysconfig/network-scripts/ifcfg-eno1 << 'EOF'
    TYPE=Ethernet
    NAME="eno1"
    DEVICE="eno1"
    HWADDR="CC:00:FF:EE:12:34"
    BOOTPROTO="none"
    ONBOOT=yes
  • Define the tagged vlan 3, and which bridge it is associated with
    cat /etc/sysconfig/network-scripts/ifcfg-eno1.3 << 'EOF'
    DEVICE="eno1.3"
    BOOTPROTO="none"
    NM_CONTROLLED="no"
    ONBOOT="yes"
    VLAN=yes
    BRIDGE=dmzbr
    EOF
  • Define the bridge with static IP
    cat /etc/sysconfig/network-scripts/ifcfg-dmzbr << 'EOF'
    DEVICE=dmzbr
    TYPE=Bridge
    BOOTPROTO="static"
    NM_CONTROLLED="no"
    ONBOOT="yes"
    TYPE="Ethernet"
    DHCP_HOSTNAME=vmhost
    IPADDR=128.227.3.20
    NETMASK=255.255.255.0
    GATEWAY=128.227.3.1
    EOF

We could get away with two but the 3rd one is there because I like to use bridges. The awake reader will have noticed I switched from eth0 to eno1; I will leave that to the reader but the point here is the above applies to whatever networking naming convention you have to deal with.

You shalll count to three (Monty Python)

As some (those who do not fall asleep reading these posts) know, I like automation and one of my tools of choice is ansible. Creating the above files for a given host from network declarations in its host_vars/host is very convenient using ansible.

But, there is Network Manager. Until recently -- CentOS 8, Rocky 8, Alma 8 -- you could still tell Network Manager to leave these interfaces alone (the NM_CONTROLLED entry), but now I am building a rocky 9 host, I am being forced to use nmcli instead. And what would it take to nmcli all of that? Let's find out (I think I missed a step here, so don't trust this):

nmcli con add ifname dmzbr type bridge con-name dmzbr
nmcli con modify dmzbr ipv6.method disabled
nmcli connection modify dmzbr ipv4.address 192.168.3.20/24
nmcli connection modify dmzbr ipv4.gateway 192.168.3.1
nmcli connection modify dmzbr ipv4.dns 192.168.3.1
nmcli connection add type vlan con-name eno1.3 ifname eno1.3 dev eno1 id 3
nmcli connection modify eno1.3 master dmzbr slave-type bridge
nmcli connection up eno1.3
nmcli connection up dmzbr

And this should end up with something like this

[root@testbox ~]# nmcli con show
NAME       UUID                                  TYPE      DEVICE
DMZ        6a97eddf-ac72-45d9-ba29-d12c7d59b511  vlan      eno1.3
dmzBridge  ccffabb8-0c8a-47d8-a2d6-15cab0e9b53b  bridge    dmzbr
eno1       9b1b155f-8197-3a7a-a9cc-79cab8b92da1  ethernet  eno1
lo         2dd22561-0fd3-436a-9da0-a53c61d63848  loopback  lo
[root@testbox ~]#

Before you get excited, that UUID is not set in stone. If you are going to do this in Ansible, take a look at the official docs on the nmcli_module. Short version is you are doing all those nmcli commands I did before, which I know I need to check since I know I missed something (I changed the bridge name later). And that is the proper official way to do the deed.

And then there is you

Yep, there is me. I know I will make a mistake. So, let's take a look on this network manager thing. No matter what, the configuration of all of these network interfaces have to go somewhere? If I am unlucky, some kind of binary-only database like Microsoft. If I am lucky, a text file. Well, it turned out my luck still holds: the files are hidden in /etc/NetworkManager/system-connections/:

[root@testbox ~]# ls /etc/NetworkManager/system-connections/
dmzbr.nmconnection  eno1.nmconnection   DMZ.nmconnection
[root@testbox ~]# 

Let's take a look at eno1:

[root@testbox ~]# cat /etc/NetworkManager/system-connections/eno1.nmconnection 
[connection]
id=eno1
uuid=9b1b155f-8197-3a7a-a9cc-79cab8b92da1
type=ethernet
interface-name=eno1
timestamp=1711852227

[ethernet]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=eui64
method=disabled

[proxy]
[root@testbox ~]# 

admit it, it may have a different, more grandiose format than /etc/sysconfig/network-scripts/ifcfg-eno1 but it describes the same thing. What would it take to make my own files? Long story short. not much:

[root@testbox ~]# more /etc/NetworkManager/system-connections/dmzbr.nmconnection /etc/NetworkManager/system-connections/DMZ.nmconnection 
::::::::::::::
/etc/NetworkManager/system-connections/dmzbr.nmconnection
::::::::::::::
[connection]
id=dmzBridge
type=bridge
interface-name=dmzbr

[ethernet]

[bridge]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=default
method=disabled

[proxy]
::::::::::::::
/etc/NetworkManager/system-connections/DMZ.nmconnection
::::::::::::::
[connection]
id=DMZ
type=vlan
interface-name=eno1.3
master=dmzbr
slave-type=bridge

[ethernet]

[vlan]
flags=1
id=3
parent=eno1

[bridge-port]
[root@testbox ~]# 

Note I did not even bother to declare the timestamp or uuid; the later can be created on the fly as the interface comes online. What does that mean? I can keep a copy of these files in a safe place, In fact, I have the following file (that is the filename, I swear)

[root@testbox ~]# cat NICs/eno1-oh_shit.nmconnection 
[connection]
id=eno1
uuid=9b1b155f-8197-3a7a-a9cc-79cab8b92da1
type=ethernet
autoconnect-priority=-999
interface-name=eno1

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]
[root@testbox ~]# 

The idea of this file is if I screw the network up and need to get it back quickly, I copy that to /etc/NetworkManager/system-connections/eno1.nmconnection, make sure it is connected to an untagged vlan switchport, and then restart network manager. That will then get a dhcp IP address and off it goes.

What about Ansible?

The same way I created /etc/sysconfig/network-scripts/ifcfg-eno1.3 I can create these files; the template looks a bit different but everything else is similar. Heretic? Surely, but that is how I roll.

Tuesday, September 26, 2017

Forcing a fuse (sshfs) network fileshare to unmount in OSX

As some of you already know, I do have an old MacBook Air which I use as my main (as in the computer I sit in front of, not the computer I store data on. Laptops can be stolen, you know) machine until I find a new Linux laptop replacement. For this reason I need it to play nice with other machines, and that requires sometimes to mount a fileshare. If the other host is in the same VLAN, that is rather easy because there are ways to mount a Windows (SMB/CIFS) and even a Linux/UNIX (nfs) fileshare without breaking a sweat. But what if the machine is remote? If we can ssh into it, why not then use sshfs?

As we are aware of (since we read the link. There are a few more sshfs examples here), sshfs requires fuse. Since I am using OSX, which at the present time does not have it, I need to install. If you are curious, the one I use is FUSE for MacOS.

Mounting: business as usual

Let's say we are in the machine boris as user pickles trying to mount my home directory off desktop. We create the mountpoint (Let's use /tmp/D or ~/D so it looks more like what we would do in Linux:

boris:Documents pickles$ mkdir /tmp/D; sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ df -h
Filesystem                     Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk1                    112Gi   79Gi   33Gi    71% 20783599  8546848   71%   /
devfs                         364Ki  364Ki    0Bi   100%     1259        0  100%   /dev
map -hosts                      0Bi    0Bi    0Bi   100%        0        0  100%   /net
map auto_home                   0Bi    0Bi    0Bi   100%        0        0  100%   /home
raub@desktop.in.example.com:.  492Gi  389Gi  102Gi    80%   408428 32359572    1%   /private/tmp/D
boris:Documents pickles$

So far so good. To unmount it we can use diskutil, as in (Mac)

boris:Documents pickles$ diskutil umount /tmp/D
Unmount successful for /tmp/D
boris:Documents pickles$

or (Linux)

fusermount -u /tmp/D

Or go old school (both):

sudo mount /tmp/D

Since boris is a laptop, sometimes if we just let it go to sleep it will unmount it. Then, all we have to do is mount it again.

Mounting again: not so fast

Thing is, sometimes it does not work.

boris:Documents pickles$ mkdir /tmp/D; sshfs raub@desktop.in.example.com:. /tmp/D
mkdir: /tmp/D: File exists
fuse: bad mount point `/tmp/D': Input/output error
boris:Documents pickles$ 

Ok, maybe it did not automagically unmounted while laptop was off. So, let's tell it to do so:

boris:Documents pickles$ diskutil umount /tmp/D
Unmount failed for /tmp/D
boris:Documents pickles$ 

Just before you ask, sudo mount /tmp/D did not work either. What if the old sshfs processes did not cleanly closed and as a result are still lingering? To answer that we must elicit some help from one of grep's cousins, pgrep:

boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
7356 sshfs user@other.in.example.com:. /tmp/D
boris:Documents pickles$

Just as we guessed, there are not only but quite a few unhappy sshfs instances. Let's see if we can kill them:

boris:Documents pickles$ kill 384 1776 7356
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ kill 384
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ kill 1776
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$
Hmmm, this is going nowhere slowly. Let's crank up a notch and force it to kill the mount.
boris:Documents pickles$ kill -9 1776
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ kill -9 384
boris:Documents pickles$ pgrep -lf sshfs
\boris:Documents pickles$

Sounds like we got them all. Now, let's try and mount once more:

boris:Documents pickles$ mkdir /tmp/D; sshfs raub@desktop.in.example.com:. /tmp/D
mkdir: /tmp/D: File exists
raub@desktop.in.example.com's password:
boris:Documents pickles$

I think we have a winner!

Monday, January 30, 2017

Using tcpdump to see vlan traffic in xenserver

Short version: it is a bit convoluted, bordering into the Rube Goldberg domain.

If you are in a hurry, you can now move onto something more interesting. If you instead want to hear me ranting and typing annoying commands, read on.

If we talking about building servers to virtualize hosts, we will end up talking about multiple networks organized in vlans which then need to be fed to this vm server. Of course running a 802.1q trunk sometimes does not work perfectly, so we need to be prepared to look behind the curtain. If the vm server is running Linux, like if it was running KVM or Xen, we can unleash tcpdump just like we did when trying to diagnose some trunking issues with a router. What about the Xenserver you mentioned in the title of this article? I thought it was Linux based. Good question. Very good question. I started this assuming it would be just Linux business as usual. You know what they say about assuming.

What I found out is that you cannot just use eth0.12 if you want to look for vlan 12 like you would do in KVM. In fact, it does not use /proc/net/vlan. It just ain't there

[root@thexen ~]# ls /proc/net/
anycast6      ip6_flowlabel        netfilter            route         tcp
arp           ip_conntrack         netlink              rpc           tcp6
dev           ip_conntrack_expect  netstat              rt6_stats     udp
dev_mcast     ip_mr_cache          nf_conntrack         rt_acct       udp6
dev_snmp6     ip_mr_vif            nf_conntrack_expect  rt_cache      udplite
fib_trie      ip_tables_matches    packet               snmp          udplite6
fib_triestat  ip_tables_names      protocols            snmp6         unix
icmp          ip_tables_targets    psched               sockstat
if_inet6      ipv6_route           ptype                sockstat6
igmp          mcfilter             raw                  softnet_stat
igmp6         mcfilter6            raw6                 stat
[root@thexen ~]#

You see, there is no eth0.12 defined here; by default xenserver will try to configure all available network cards (NICs for those craving for acronyms) in a managed (by xenserver) mode. Once they are added, it creates bridges, called xapiN, and then associates them with each network. And how do we find out which of those bridges is being used by our vlan? Er, it requires a few steps using the xen commands (xe something-or-another) which I have not found out how to automate yet.

  1. We begin by finding out which vlans are defined in this server. And that can be done using xe pif-list:
    root@thexen ~]# xe pif-list
    uuid ( RO)                  : 540f3b24-0606-6380-c10c-c2f8c2f4c2ce
                    device ( RO): eth1
        currently-attached ( RO): true
                      VLAN ( RO): 2
              network-uuid ( RO): a874cb50-1c87-0bde-390d-66d0a4e1576c
    
    
    uuid ( RO)                  : 8684e63f-3d1c-241b-8e75-3b2e37f8c859
                    device ( RO): eth0
        currently-attached ( RO): true
                      VLAN ( RO): -1
              network-uuid ( RO): ed2325c5-1f3b-7f25-6104-61902a13d3ac
    
    
    uuid ( RO)                  : 82b9dee1-52db-a6ae-cb42-11ae7f6d3d25
                    device ( RO): eth1
        currently-attached ( RO): true
                      VLAN ( RO): -1
              network-uuid ( RO): 993c9237-5961-9808-36cd-729827e005d8
    
    uid ( RO)                  : 592339bf-cc03-4048-9075-946f5bcc47fb
                    device ( RO): eth1
        currently-attached ( RO): true
                      VLAN ( RO): 12
              network-uuid ( RO): 0d28f847-3da6-11f3-3600-8a033435168c
    
    
    uuid ( RO)                  : 3d60399c-bb8d-5e5a-e01b-8986b8808f12
                    device ( RO): eth0
        currently-attached ( RO): true
                      VLAN ( RO): 3
              network-uuid ( RO): c8726e09-a0a5-b026-013e-2c5edd5062b3
    
    
    uuid ( RO)                  : 19f1fe37-16d1-6fcd-4bbd-4e566abc74c4
                    device ( RO): eth0
        currently-attached ( RO): true
                      VLAN ( RO): 8
              network-uuid ( RO): 2f94b1c8-be16-14d5-a149-90ae35528c22
    
    
    uuid ( RO)                  : 6df7b741-9cef-d34f-e487-fa2abe422068
                    device ( RO): eth1
        currently-attached ( RO): true
                      VLAN ( RO): 100
              network-uuid ( RO): 9ec62435-ec2a-2bfc-9f29-2ea5c9756971
    
    
    [root@thexen ~]#

    What can we gather from this output:

    • This machine has 2 physical interfaces, eth0 and eth1. And each of them have a few vlans groing through them. So, there are two 802.1q trunks. Deal with it.
    • We have two uuid entries per interface uuid (the one after "uuid ( RO) ") and a network-uuid.
    • If we only wanted to see the VLAN number, both the uuids, and the physical NIC/device each virtual interface is using, we could have instead said
      xe pif-list params=device,VLAN,network-uuid,uuid

      But if we wanted to know everything about each virtual interface,

      xe pif-list params=all
    • To get more info on a giver interface (or bridge) you need the uuid associated with uuid ( RO). So if you wanted to know everything about VLAN 100, you could say
      xe pif-list uuid=6df7b741-9cef-d34f-e487-fa2abe422068 params=all
    • The ones with VLAN ( RO): -1 are the untagged networks; we have one per interface even if we do not have it defined.
OK< smart guy, how do we go from this to this crazy xapiN interface? Oh, you mean what xenserver calls a bridge? We shall use the xe network-list command. If you run that, it will give back which xapiN is associated with which vlan. It will also show which bridge is being used for the console to this xenserver, which usually is an untagged vlan. There are ways to make that a tagged vlan but that will be for another episode. What is important is the uuid being shown is the network-uuid we got using xe pif-list. And, we can feed it to the network-list command if we just care about, say, VLAN 12:

[root@thexen ~]# xe network-list uuid=0d28f847-3da6-11f3-3600-8a033435168c params=bridge,name-label
name-label ( RW)    : vlan 12
        bridge ( RO): xapi4


[root@thexen ~]#

We finally found out that vlan 12 is attached to xapi4. Time for some tcpdumping:

[root@thexen ~]# tcpdump -i xapi4 -e
tcpdump: WARNING: xapi4: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xapi4, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@thexen ~]#

Crickets

What is going on here? I let it run for 10s; that should have been enough to fill the screen. Let's try again, letting it run for longer 94 minutes?) while we do, say, tracepath to the gateway.

[root@thexen ~]# tcpdump -i xapi4 -e -n
tcpdump: WARNING: xapi4: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xapi4, link-type EN10MB (Ethernet), capture size 65535 bytes
10:46:10.696176 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:11.698310 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:12.700355 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:13.705644 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:14.706364 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:15.708375 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:18.710913 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
[...]
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28
10:46:30.724414 c0:ff:ee:70:63:a4 > Broadcast, ethertype ARP (0x0806), length 42
: Request who-has 192.168.12.241 tell 192.168.12.242, length 28

^C
15 packets captured
15 packets received by filter
0 packets dropped by kernel
[root@thexen ~]#

Fifteen packets in four minutes? I could have done that by hand! What is going on here?

But it really does not work as well as it should

Do I sound bitter? I am just being factual. I can't use tcpdump when this bridge thing is only giving me like 4 packets a minute. Even if there was no talkign between servers, just the ARP requests should have been more often. So, we need to rething this.

We did the proper thing so far. Now it's time to cheat.

We know the network associated with vlan 12 is 192.168.12.0/24, so why not tell tcpdump to look at eth1 for anything that matches that?

tcpdump -i eth1 -e -n | grep '192.168.12'

I will not bother to show the output of that but it will look much more like what we would have expected. Of course, it will only get traffic in that wire matching that pattern so if you have a host trying to reach out for a dhcp server in that network you will not detect that. Nor it would find IPv6 traffic (you would need to feed the proper pattern). But, it is better than using xapi4.

Saturday, December 31, 2016

Discovering (tagged only?) vlans using tcpdump

Why to use VLANs? well, there is the part of having a bunch of different devices -- computers, printers, toasters -- networks that are connected to the same switches but in different networks so you can control how/if they can see and talk to each other. And those switches are connected to each other and to routers who need to route traffic between the different networks. And that probably means we are trunking and tagging the vlans between those switches and routers. But, if we now add virtual machines to the mix, the physical machines they are running on -- the virtual servers -- need to be able to connect to all the networks their virtual clients belong to. And that once again is done using trunking and vlan tagging.

What if it is not working properly?

Let me elaborate it by using an example: I recently had a router running OpenWRT which I assumed I had configured to handle multiple vlans and was connected to the rest of the network through a 802.1q trunk; think of it as a router-on-a-stick setup. Now, the router was configured to have one IP -- 192.168.2.1 and 192.168.3.1 to make it easier -- in each of the two vlans, which I shall call vlan2 and vlan3. Problem was I could only reach it through vlan2; nothing on vlan3.

First thing I did was to recheck the switch to see if that port was configured for 802.1q trunk. Not only it was but both vlans were tagged; I did not have an untagged/default vlan in this trunk. So for all practical purposes both vlans should look the same for the router. So why weren't they?

At this point I decided to do some packet sniffing. And, this would be the cue to unleash Wireshark on it. But, we are talking about a little router running a OS designed to fit a resource limited device. Why not see if we can use a command line alternative, like the humble tcpdump? It can show you not only any vlan traffic on the wire but also only traffic on a specific vlan just as it can show traffic related to a specific IP or protocol. For instance, since we really only care about traffic on vlan 3 on interface eth0 we can

root@router:~# tcpdump -i eth0.3
tcpdump: WARNING: eth0.3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes

And then after letting it run for ten minutes, all I got back were tumbleweeds. In the same time frame, if had asked for traffic on vlan 2 it would have filled the screen before I finished this sentence (and, yes, I am a slow typist but still). The logical thing to do now is to see if there is any traffic on the wire that has a 802.1Q tag (vlan):

tcpdump -i eth0 -e -n vlan
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
01:32:27.440782 e2:46:9a:5e:a7:45 > c0:ff:ee:3e:06:eb, ethertype 802.1Q (0x8100)
, length 122: vlan 2, p 0, ethertype IPv4, 192.168.2.1.22 > 192.168.2.249.56762: Flags
 [P.], seq 2026124025:2026124077, ack 2216938037, win 21298, options [nop,nop,TS
 val 346274301 ecr 697366229], length 52
01:32:27.441234 e2:46:9a:5e:a7:45 > c0:ff:ee:3e:06:eb, ethertype 802.1Q (0x8100)
, length 122: vlan 2, p 0, ethertype IPv4, 192.168.2.1.22 > 192.168.2.249.56762: Flags
 [P.], seq 52:104, ack 1, win 21298, options [nop,nop,TS val 346274301 ecr 697366232], length 52
01:32:27.441946 e2:46:9a:5e:a7:45 > c0:ff:ee:3e:06:eb, ethertype 802.1Q (0x8100)
, length 138: vlan 2, p 0, ethertype IPv4, 192.168.2.1.22 > 192.168.2.249.56762: Flags
 [P.], seq 104:172, ack 1, win 21298, options [nop,nop,TS val 346274301 ecr 6973
66232], length 68

Do you see the vlan info on those lines? Let's trim it a bit:

tcpdump -i eth0 -e -n vlan | cut -f 6,10,11 -d ' '
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
802.1Q vlan 2,
^C21 packets captured
29 packets received by filter
0 packets dropped by kernel

So all the traffic we have seen above is for vlan 2. In fact the 192.168.2.1.22 > 192.168.2.249.56762 means someone in 192.168.2.249 is connecting from its port 56762 to port 22 (ssh) on 192.168.2.1, which is the router. And that is indeed the case because that is how I connected to the router!

NOTE: A long time ago there was a bug in tcpdump, which would require you to run it twice,

tcpdump -Uw - | tcpdump -en -r - vlan 

But that is no longer the case.

But, where's vlan 3? That is a great question! At first I thought I had it setup wrong n the router even though both vlans have similar confiugrations, save the network info. And that kept me awake for a few nights.

It turns out that there is a ongoing bug which would not allow me to run more than one tagged vlan. The workaround is to "kick" it by adding a

option enable_vlan4k 1

to /etc/config/network

We feel confident we solved the problem, so instead of listening to all vlans this time we will only listen for vlan 3 traffic:

root@router:~# tcpdump -i eth0.3
tcpdump: WARNING: eth0.3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes
19:15:25.822964 IP 0.0.0.0 > all-systems.mcast.net: igmp query v2
19:15:25.823010 IP6 fe80::e046:9aff:fe5e:a745 > ff02::1: HBH ICMP6, multicast li
stener querymax resp delay: 10000 addr: ::, length 24
19:15:26.568728 ARP, Request who-has 192.168.3.10 tell 192.168.3.1, length 28
19:15:26.569022 ARP, Reply 192.168.3.10 is-at c0:ff:ee:14:25:ad (oui Unknown), l
ength 42
19:15:26.569092 IP 192.168.3.1.32101 > 192.168.3.10.domain: 29094+ PTR? 1.0.0.0.
0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
19:15:26.606872 IP 192.168.3.10.domain > 192.168.3.1.32101: 29094 NXDomain 0/1/0
 (160)
19:15:27.797110 IP6 fe80::c2ff:eeff:fe14:25ad > ff02::1:ff14:25ad: HBH ICMP6, mu
lticast listener reportmax resp delay: 0 addr: ff02::1:ff14:25ad, length 24
19:15:28.014715 IP6 fe80::be5f:f4ff:fe54:d78d > ff02::1:ff54:d78d: HBH ICMP6, mu
lticast listener reportmax resp delay: 0 addr: ff02::1:ff54:d78d, length 24
19:15:28.615536 IP 192.168.3.1.16915 > 192.168.3.10.domain: 15895+ PTR? d.a.5.2.
4.1.e.f.f.f.e.e.f.f.2.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
19:15:28.616059 IP 192.168.3.10.domain > 192.168.3.1.16915: 15895 NXDomain* 0/1/
0 (125)
19:15:28.618233 IP 192.168.3.1.36929 > 192.168.3.10.domain: 54136+ PTR? d.a.5.2.
4.1.f.f.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
19:15:28.646935 IP 192.168.3.10.domain > 192.168.3.1.36929: 54136 NXDomain 0/1/0
 (160)

And that is the kind of output I wanted to see!

Moral of the story: when vlan and their trunks get funky it might be time to unleash some packet analysis tool to figure out where in the network is the problem. And, even though Wireshark is a pretty nice tool, sometimes you can get the work done with something a bit more lightweight.