Disassembling a Brocade 2800 fabric switch (and finding its serial port/resetting its password in the process!)

What we have here is a Brocade 2800 fabric switch, but you probably new that by reading the title of today's chapter. In previous jobs I have worked with 24000 and 48000 Brocade switches like this hairball here:

Yes, they are bigger (256 fibre ports for the 24000 and 512 for the 48000 vs 16 for the 2800) and faster (up to 4GB compared to 1GB) but they all do the same thing: they were originally designed to be used in storage, connecting SAN and NAS devices to the hosts that use them. The fabric part of the name means just that. But, we are here talking about the 2800, so sit down and take notes because I will not repeat myself.

As mentioned before, the 2800 has 16 fibre ports... and one ethernet port. The ethernet port, a 100Base-TX one, is normally used as your console: you assign an IP to it, connect to the switch either using a web interface or ssh, and do some switch managing.

Tearing it apart.

All that talk about what these switches can do is most interesting, but that is not what we are here to do today. We want to see what is inside the box! So, let's get busy!

The first thing we should do is to remove the two redundant power supplies. To do so, you first pop the clip that goes around the power connector; you will note it becomes a neat handle (shown here is the left power supply sitting on the top of the right, which is still in the switch).

Then, you press down on the tab the clip hinges around. That unlocks the power supply from the case, allowing you to slide it out from the front.

Here is another view of the hole the right power supply hides. Note the tabs on the left. They are what hold the top in place, which we will be removing in the next step.

Ok, I was lying. We are not removing the top of the switch just yet. First, we need to remove the plastic cover that surrounds the front LCD and the buttons that control the menu on the display. It will come off if you grab it on one side and work it out, I promise! Just do not be too aggressive.

Once the cover is off, well, the cover is of. Now we can go back to the business of removing the top cover.

This next step require a bit of force and traction: you need to slide the lid forward which will mean you (or a friend) will have to hold the switch in place while wiggling the cover and slowly moving it. I did that by myself but will say finding a place to hold the cover was tricky. This is in my opinion the hardest step in this entire adventure.

With a judicious use of curse works, strength, and even finesse, it should eventually slide forward until it can't go any further. Yeah, as the picture below shows it is not much, but remember all it has to do is to line its tabs with the holes you saw through the power supply bays. Now, you have to grab the back of the cover and tilt it up around the front. Then, a bit of wiggling should get the cover out.

Here is how the switch looks like topless. Those wires on the top come from the front panel. You can also see the fans on the rear. Also note the titled plate those wires go through. It is probably that way to help direct the air from the motherboard to the fans; we will talk about them in a moment.

And here is a rear view of the case. Can you see the motherboard hiding there?

I promised I was going to talk a bit about the fans, and talk about them I will. If you look on the back of the case, you will find 4 thumbscrews that look like this:

They hold the fan assembly to the case. So, if you unscrew them, you can remove the fans without much fuss.

Overall I will say this is very easy switch to take apart, which means you can replace any of its components in minutes.

Hunting for the serial port.

If you are like me, you will make sure you keep all documentation on the switch somewhere safe... or safer than piling up by the trash bin. The more confidential stuff, like its configuration and passwords, you probably have a secure encrypted file somewhere with them; it sure beats post-it notes! But, what do you do if you lose your password or the previous guy left without letting anybody know it? Well, two things you can do: call Brocade and have them send you a file to reset the password (which they will gladly do depending on your contract) or you have to take matters on your own hand. Brocade was actually nice: in addition to the normal ethernet console port, they hid a serial port inside the switch. As I looked online for info on it, it seemed that its location was only told to the initiated after they proven their mastery of the secret network society handshake.

Jokes apart, I have never found pictures of the inside of this switch, much less detailed information of the location of the serial port (I guess it rather be called IDC but I will ignore its wishes). And this is why I decided to do this entire writeup. The part about tearing the siwtch apart is more of a bonus feature; this is the neat stuff:

Getting to the serial port will require one tool and one tool only: a flathead screwdriver. If you look on the front of the switch, you will find a slotted cylinder thingie about in the middle of the ports; that is where you put the screwdriver on:

That actually turns a long shaft which ends inside the case. The threaded end of the shaft turns around a nut that is attached to a bracket. Just click on the pict below to see it; it is kinda behind where the wires from the front panel go down.

As you unscrew it, the drawer containing the motherboard and the switch ports will slide forward. It will not go very far; you will know when you unscrewed it as far as it can go. After that you will need to work it out. It is tight in there. I wiggled it and eventually it slid enough I could grab it and pull it out like a drawer:

Here is what the motherboard looks like once it is removed from the case. Now, what about the serial port you talked about? Calm down and look to its upper left corner, by the left large white connector. Can't see it? Let me zoom a bit.

How about now? Can you see it? Hint: I circled it in yellow.

And here is yet an even closer look. Can you see the pin numbers? Hint: White dot usually indicates pin 1.

If you want to make your serial cable, here is the pinout:

I will add how to reset the password later.

Indentifying hard drives

Some weeks ago I was in a AIX forum where one of the members asked to know the difference between a 68pin SCSI hard drive and an 80pin one. So, what I did was to grab some of those hard drives I had sitting around and take a picture of their business end, i.e. where you can recognize which one is which.

If you look at the top hard drive, you will notice it has a large connector to the left and a power connector (the same one used in the old PATA drives) to the right. That would be a 68pin drive; if you want to count, there will be 68 pins on that large connector. Go ahead; I will wait. While you count, I will pass the time by saying some older UNIX workstations, such as the IBM RS6000 up to (as far as I know) P3, and the old DEC Alphas use those drives. The largest 68 pin drive I have ever seen is 32 (or is it 36?) GB.

By elimination, the two drives on the bottom are of the 80pin variety. As you can see, they have only one large connector right smack in the middle; power and signals go through that connector. And, it does have 80 pins. Sun workstations since the Sparc 20 vintage have been using these drives whenever they wanted SCSI. In addition to these workstations, they also find their way into some SANs. If you are looking to buy them, sometimes they go for the acronym SCA. I have personally seen 147GB SCA hard drives, but I do know there are 500GB ones out there.

ZFSing around (or, using ZFS under Solaris)

ZFS is a different file system than the old UFS. Best way to describe it is to think on it as a file system with LVM built on it. If you know LVM, you will understand what I am trying to say and recognize a lot of things here. If not, I promise that soon I will try to talk about it and compare ZFS, Linux LVM, and AIX LVM. For now, accept my simplistic explanation:

Logical Volume Manager, or LVM, is a way the separate the physical storage (think hard drives and networks drives and such things) from the storage seen by the user. In other words, what the users see is a pseudo disk (or logical volume if we are to use its terminology) of sorts which can be made of a collection of other disks, raid arrays, or just partitions of those disks. Now, while Linux LVM provides the logical disk which is then formatted and partitioned as they were normal disks, in ZFS these two steps are done become just one.

I think this will make sense using an example: in this Solaris 10 machine, we have two 73GB SCA hard drives. One of which, c0t0d0 (yes, Sun likes to call their drives differently than everybody else. We will talk about that in some other episode. For now, it suffices to say that d0 is the SCSI disk with SCSI ID 0), was formated as ZFS during the installation of the operating system and currently looks like this:

# df -h
Filesystem             size   used  avail capacity  Mounted on
boot/ROOT/root          67G   6.8G    40G    15%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   1.9G   1.4M   1.9G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
fd                       0K     0K     0K     0%    /dev/fd
boot/ROOT/root/var      67G   727M    40G     2%    /var
swap                   1.9G    32K   1.9G     1%    /tmp
swap                   1.9G    40K   1.9G     1%    /var/run
boot                    67G   176K    40G     1%    /boot
boot/export             67G    20K    40G     1%    /export
boot/export/home        67G    15G    40G    28%    /export/home
# 

Then we have c0t1d0, also a 73GB HD that we plan to add to the hard drive. For this machine, I would like to have the following layout:

/
/var
/tmp
/export/home   (accounts)
/export/hosts  (virtual hosts, in case we have them)
/export/images (as this may end up being a netboot/jumpstart server)

Each of those would reside in a separate partition. Some of that has already been taken care of by Solaris' default install, but we are left with /export/hosts and /export, which will be run off c0t1d0. Before ZFS, we would divide the hard drive using format into two partitions and would edit /etc/vfstab to mount the two partitions at boot time:

# cat /etc/vfstab                      
#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/zvol/dsk/boot/swap -       -       swap    -       no      -
/devices        -       /devices        devfs   -       no      -
sharefs -       /etc/dfs/sharetab       sharefs -       no      -
ctfs    -       /system/contract        ctfs    -       no      -
objfs   -       /system/object  objfs   -       no      -
swap    -       /tmp    tmpfs   -       yes     -
/dev/dsk/c0t1d0s4     /dev/rdsk/c0t1d0s4      /export/hosts        ufs     2       yes     -
/dev/dsk/c0t1d0s6     /dev/rdsk/c0t1d0s6      /export/images       ufs     2       yes     -
# 

but, in ZFS things are a bit different. First we need to create the pseudo disk mentioned above. In ZFS terminology that is called a pool; I guess they want to remind us that you usually create it by adding disks to it. So we create the pool, which shall be named storagepool. By now you may have realized that boot is the name of the pool created during the installation of the operating system.

# zpool create  storagepool c0t1d0
# df -h
Filesystem             size   used  avail capacity  Mounted on
boot/ROOT/root          67G   6.8G    40G    15%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   1.9G   1.4M   1.9G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
fd                       0K     0K     0K     0%    /dev/fd
boot/ROOT/root/var      67G   727M    40G     2%    /var
swap                   1.9G    32K   1.9G     1%    /tmp
swap                   1.9G    40K   1.9G     1%    /var/run
boot                    67G   176K    40G     1%    /boot
boot/export             67G    20K    40G     1%    /export
boot/export/home        67G    15G    40G    28%    /export/home
storagepool             67G    19K    67G     1%    /storagepool
#

You may have noticed that the pool we created, storagepool is also a filesystem: as soon as we created the pool, it becomes a filesystem mounted under /storagepool. That is different than what is done under Linux LVM, where you would first create the pool, then partition and format that pool into the partitions you are going to use, and then mount them using /etc/fstab (the Linux equivalent to /etc/vfstab and mount -a. In Solaris and ZFS, on the other hand, that is done with just one command. I mean, we did not even have to edit /etc/vfstab. The current list of ZFS mountpoints defined can be seen using zfs filelist:

# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
boot                26.5G  40.5G   176K  /boot
boot/ROOT           7.46G  40.5G    18K  legacy
boot/ROOT/root      7.46G  40.5G  6.75G  /
boot/ROOT/root/var   727M  40.5G   727M  /var
boot/dump           2.00G  40.5G  2.00G  -
boot/export         15.0G  40.5G    20K  /export
boot/export/home    15.0G  40.5G  15.0G  /export/home
boot/swap              2G  42.5G    16K  -
storagepool         89.5K  66.9G     1K  /storagepool
# 

If you remember, from storagepool we will be creating /export/hosts. That is done as follows:

# zfs create storagepool/hosts
# df -h
Filesystem             size   used  avail capacity  Mounted on
boot/ROOT/root          67G   6.8G    40G    15%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   1.9G   1.4M   1.9G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
fd                       0K     0K     0K     0%    /dev/fd
boot/ROOT/root/var      67G   727M    40G     2%    /var
swap                   1.9G    32K   1.9G     1%    /tmp
swap                   1.9G    40K   1.9G     1%    /var/run
boot                    67G   176K    40G     1%    /boot
boot/export             67G    20K    40G     1%    /export
boot/export/home        67G    15G    40G    28%    /export/home
storagepool             67G    19K    67G     1%    /storagepool
storagepool/hosts       67G    18K    67G     1%    /storagepool/hosts
#

If we just leave at that, a mount point, /storagepool/hosts, is created. But, we really do not want that. We want to mount storagepool/hosts in /export/hosts, and mount we shall.

# zfs set mountpoint=/export/hosts storagepool/hosts
# 

Now, it seems to be mounted where we want

# df -h
Filesystem             size   used  avail capacity  Mounted on
boot/ROOT/root          67G   6.8G    40G    15%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   1.9G   1.4M   1.9G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
fd                       0K     0K     0K     0%    /dev/fd
boot/ROOT/root/var      67G   727M    40G     2%    /var
swap                   1.9G    32K   1.9G     1%    /tmp
swap                   1.9G    40K   1.9G     1%    /var/run
boot                    67G   176K    40G     1%    /boot
boot/export             67G    20K    40G     1%    /export
boot/export/home        67G    15G    40G    28%    /export/home
storagepool             67G    19K    67G     1%    /storagepool
storagepool/hosts       67G    18K    67G     1%    /export/hosts
#

Neat, huh? There is more we can do, like establish quotas and such stuff. But that will be for another episode.

Some parting thoughts

Remember that Sun call boot and storagepool pools. Even though we've only used one disk per pool, we could have used more. For instance, we could have grouped them together like in Linux LVM,

# zpool create  anotherpool c0t0d0 c0t1d0
# 

or, we could have made a pool where one of the disks mirrors the other, as in a raid 1

# zpool create  anotherpool mirror c0t0d0 c0t1d0
# 

Another option is instead of doing a mirror is to create a raid 5 (Sun claims it is a variation of raid 5, bigger, faster, better) using the raidz option and at least 3 drives (min for such raid),

# zpool create  anotherpool raidz c0t0d0 c0t1d0 c0t2d0
# 

We could also create hotspares for those arrays; check the man page for zpool for more info on that. I hope that will make you think on what ZFS can do and why I think it is rather neat.

Of Macs and serial ports

My trusty iBook, as all Macintosh computers manufactured in the last few years, have no serial port. That has never stopped me from doing work as I had a Linux laptop, a Dell Latitude D600, which I would bring whenever I needed to talk to a Cisco switch or use as console for a Unix workstation (say, Sun Solaris or IBM AIX box).

But, then, the Dell laptop died. And I needed to configure a cisco switch from scratch... at least configure it enough so I could then telnet to it. To do that I needed to connect the famous Cisco blue console cable to the Mac. I needed a usb-to-serial cable.

Not knowing where to find one of those usb-to-serial cables, I decided to try one of my favorite places: geeks.com. I not only found it but here is a picture of the cable:

Clicking on the image *should* lead you to the link for the cable. After I received it, I connected it to the Mac. The laptop was aware of the device, even recognizing its chipset. But, it would not be available for use. Here is what I mean:

Mireille:~ dalek$ ls /dev/tty.*
/dev/tty.Bluetooth-Modem                /dev/tty.Nokia6103-NokiaPCSuite-1
/dev/tty.Bluetooth-PDA-Sync             /dev/tty.modem
/dev/tty.Nokia6103-Dial-upnetwor-2
Mireille:~ dalek$ 

Clearly, I need a driver for it. Examining the information shown by the machine about the driver, we see the chipset is made by prolific. After a bit of searching online, I found the manufacturer's site and downloaded the drivers from its site. Do note in that page that they also have drivers for Windows and even Linux. I do not know if Linux would ever need such a driver; finding that out is for a different episode. Anyway, after installing it, we had to reboot the laptop. After that, it was time to connect the usb-to-serial cable and find out if it was seen as a device we could use. Can you spot the new entry?

Mireille:~ dalek$ ls /dev/tty.*
/dev/tty.Bluetooth-Modem                /dev/tty.Nokia6103-NokiaPCSuite-1
/dev/tty.Bluetooth-PDA-Sync             /dev/tty.modem
/dev/tty.Nokia6103-Dial-upnetwor-2      /dev/tty.usbserial
Mireille:~ dalek$ 

Now we have a device, tty.usbserial, we can try it out. We could install minicom using fink, but we can be a bit lazy and use, of all things, screen. Believe it or not, screen can also be used to connect to a terminal device. So, if you type something like

Mireille:~ dalek$ screen /dev/tty.usbserial 9600

you would be telling screen to connect to our usb-to-serial cable, identified as tty.usbserial, at 9600baud which happens to be the default port speed for a Cisco switch. Neat, huh?