Showing posts with label USB. Show all posts
Showing posts with label USB. Show all posts

Friday, May 15, 2015

USB Passthrough in KVM/libvirt 2: Problems and workarounds on Arduino development in a vm client

We will talk a bit about setting up an Arduino development environment. Originally I thought this was going to be a short article, but that was not the case. The more I worked on this the longer it became. So I got annoyed and put the part about setting USB passthrough and then dealing with waking up a vm client that lost its USB device after it was saved in their own articles. I want to have this article as focused on building an Arduino development environment in a vm guest as possible; we already talked about USB passthrough in general and even accessed a UPS as example.

Last time I did setup an Arduino development environment I used my laptop and my Arduino. That worked fine but required me to have my Arduino with me and the laptop just in case I decided to do something; which I did: I carried the board, cables, and even a USB drive with useful stuff in a little box in my backpack at all times. With time that got old, I now prefer to do remote development as much as I can get away with, so let's see if I can do it with the Arduino.

The Arduino boards I will be using here are the Funduino Uno/YourDuinoRoboRED and a Duemilanove.

The scientific reason I am using those boards is because that is what I have; I bought both of them with my own money. The first one I ever got was the Duemilanove; it is in fact the one I used to carry in my bag. Recently (as in last month) I bought the Funduino board. I decided to start this article by choosing this red board because it comes with a miniUSB port and included a USB cable just the right size so it can be precariously hung from vmhost. Maybe also because it is more colourful; you'll be the judge.

We will be adding an Arduino device to the vm client desktop, which is an Ubuntu Desktop vm in the vmhost called, for reasons the go beyond the topic of this article, vmhost, which runs KVM with libvirt.

Setting the mess up

The title of this article mentions USB passthrough. Main difference between that and PCI passthrough is that the USB one allows hotplugging. And that means we should be able to add the Arduino board to desktop while this vm client is running. At least that is the hope.

Without further ado, let's get busy.

  1. We probably should start this by connecting the arduino board to on of vmhost's USB port. The picture on the right shows the actual board hanging in front of the actual vmhost.
  2. As we did in an earlier article on USB passthrough, we should ask if vmhost knows about the Arduino board:
    [raub@vmhost ~]$ lsusb
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
    Bus 001 Device 003: ID 2341:0001 Arduino SA Uno (CDC ACM)
    [raub@vmhost ~]$
    The line we are interested in is
    Bus 001 Device 003: ID 2341:0001 Arduino SA Uno (CDC ACM)
    Note that it identifies itself as an Arduino Uno; I do not know if the real Arduino Uno also identifies itself the same way. Also note that
    <vendor id='0x2341' />
    <product id='0x0001' />
  3. With that info we can now tell desktop about the Arduino board. As we have seen before, we can do it live by creating a xml config file containing info about the Arduino board:
    cat > arduino.xml << 'EOF'
        <hostdev mode='subsystem' type='usb' managed='yes'>
          <source>
            <vendor id='0x2341'/>
            <product id='0x0001'/>
          </source>
        </hostdev>
    EOF
  4. Attach it
    [root@vmhost tmp]# virsh attach-device desktop ./arduino.xml
    Device attached successfully
    
    [root@vmhost tmp]#
  5. Check if the vm client reports it as attached
    raub@desktop:~$ lsusb
    Bus 001 Device 004: ID 2341:0001 Arduino SA Uno (CDC ACM)
    Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
    Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
    Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
    raub@desktop:~$
    Don't forget since we are accessing this board as a normal user, we need to be in the group that owns that device. I know that its USB port is shown as /dev/ttyACM0, so we need to find who owns it.
    raub@desktop:~/Downloadz/Arduino$ ls -l /dev/ttyACM0
    crw-rw---- 1 root dialout 166, 0 May  8 09:56 /dev/ttyACM0
    raub@desktop:~/Downloadz/Arduino$ id
    uid=1000(raub) gid=1000(raub) groups=1000(raub),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),107(lpadmin),124(sambashare),127(debian-tor)
    raub@desktop:~/Downloadz/Arduino$
  6. Get and install the Arduino SDK. Since I am doing all of this in a Linux vm, installing the SDK is really uncompressing the file, arduino-1.6.4-linux64.tar.xz at the time I wrote this article, into where you want to run the SDK from; I put it in ~/bin.
    tar xJvf arduino-1.6.4-linux64.tar.xz
  7. And run and configure the SDK. I connected to the vm running the SDK from my laptop, telling ssh to do X11 port forwarding:
    ssh -X desktop
    Once there, I started the Arduino SDK
    ./bin/arduino-1.6.4/arduino
    which popped the usual GUI on my laptop screen thanks to the magic of X Windows. Now, do tell it that you are connected to an Arduino Uno board through port /dev/ttyACM0.
  8. Now we need to test things up. I think one of the best programs to test an Arduino SDK setup is blink, which controls the little LED that a lot of boards have built-in. Here is the original code if you are too lazy to look it up:
    /*
      Blink
      Turns on an LED on for one second, then off for one second, repeatedly.
    
      This example code is in the public domain.
     */
    
    // Pin 13 has an LED connected on most Arduino boards.
    // give it a name:
    int led = 13;
    
    // the setup routine runs once when you press reset:
    void setup() {
      // initialize the digital pin as an output.
      pinMode(led, OUTPUT);
    }
    
    // the loop routine runs over and over again forever:
    void loop() {
      digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
      delay(1000);               // wait for a second
      digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
      delay(1000);               // wait for a second
    }
    Upload the file and see if it blinks the way you told it to do so (1s on, 1s off).
And all seems to be nice an peachy; we should end this article here and pat ourselves on the back. But, this is where things start to go wrong.

Things Did Not Happen According to the Plan

  1. Let's say I want to change the blink frequency in the blink program. Maybe I want to have the LE stay on 2s and off 1s. That part of the code would change like this:
    void loop() {
      digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
      delay(2000);               // wait for a second
      digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
      delay(1000);               // wait for a second
    }
    Great! now, let's upload the new code and stare at the LED.

    And no changes.

    We then look at the GUI and find the following messages:

    Sketch uses 1,068 bytes (3%) of program storage space. Maximum is 32,256 bytes.
    Global variables use 11 bytes (0%) of dynamic memory, leaving 2,037 bytes for local variables. Maximum is 2,048 bytes.
    avrdude: stk500_recv(): programmer is not responding
    avrdude: stk500_getsync() attempt 7 of 10: not in sync: resp=0x00
    avrdude: stk500_recv(): programmer is not responding
    avrdude: stk500_getsync() attempt 8 of 10: not in sync: resp=0x00
    avrdude: stk500_recv(): programmer is not responding
    avrdude: stk500_getsync() attempt 9 of 10: not in sync: resp=0x00
    avrdude: stk500_recv(): programmer is not responding
    avrdude: stk500_getsync() attempt 10 of 10: not in sync: resp=0x00
    Problem uploading to board.  See http://www.arduino.cc/en/Guide/Troubleshooting#upload for suggestions.
    Sometimes doing virsh detach-device and then reattaching it will work, but other times it does not, which leads to...
  2. The Funduino board likes to disappear.
    [root@vmhost tmp]# virsh attach-device desktop arduino.xml
    error: Failed to attach device from arduino.xml
    error: internal error Did not find USB device 2341:1
    
    [root@vmhost tmp]#  lsusb
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
    [root@vmhost tmp]#
    By that I mean I still have not found what triggers that. Sometimes it is connected to vmhost for 3 days doing nothing but always showing on lsusb even if it is not attached to a vm client. And then, it is gone. Just like that. Only workaround I found was to physically unplug the Funduino out of vmhost and then plug it back in.

What happens if you use the Duemilanove?

I have to say I have not been able to replicate any of the above issues with the Duemilanove. It uses a different USB-to-Serial chip, so it is seen by lsusb as

Bus 001 Device 012: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
I do not know if the chipset makes a difference or there is something in my Funduino board that is boink (maybe the chipset is more picky?). I have another Funduino board, but not an Arduino Uno, so I can only test so much. What I can say is as of now I will use the Duemilanove to do remote development, but when I want to take a board in the field -- say, attach to my car -- I will grab the Funduino one. After all it works fine if I just install new code once.

Update on the Funduino/RoboRED: Success!

So I got home last night, took my second Funduino out of its shrinkwrap, and tried it. It works as well as the Duemilanove:

  1. If I disconnect it physically from vmhost, desktop gracefully reports it as gone. virsh dumpxml still reports an entry
    [root@vmhost tmp]# virsh dumpxml desktop|grep 2341
            
            
    [root@vmhost tmp]# 
    but that could be how KVM does things.
  2. If I unload it using
    [root@vmhost tmp]# virsh detach-device desktop arduino.xml                      
    Device attached successfully
    
    [root@vmhost tmp]#
    it does get removed from desktop's config file. If there is more than one entry, it might take a bit more time but it will come out.
  3. If we did not physically remove the Arduino device as above, instead just using virsh detach-device, it will be removed from destkop's config and running lsusb in that vm client will no longer report it.
  4. Once lsusb in vmhost shows the Arduino board back (say, you physically connected it back to that machine),
    [root@vmhost tmp]# virsh attach-device desktop arduino.xml                      
    Device attached successfully
    
    [root@vmhost tmp]#
    will once again make the device available for desktop.
Since the problem is on the first one; I will see about getting it replaced.

Thursday, May 14, 2015

Restoring a saved vm client that cannot find its attached USB device

The title is a mouthfull, I know. I really want to talk about Arduino development in a vm client. In fact, this was going to be one of the issues I had found while doing that, but I later on felt it deserves its own article. Reason is that it might be helpful on its own.

So I have a KVM-based (with libvirt) vm host, vmhost. Running in it is a vm guest, desktop. vmhost is configured to use managesave to automagically save and restore vm clients when it reboots. I like doing that because it saves the state and all that exciting stuff. Now, I configured desktop to access an Arduino board using USB passthrough, which seemed to have worked. But then I needed to reboot vmhost. I expected all my vms to come back running, but that was not the case:

[root@vmhost tmp]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     nameserver                     running
 3     nagios                         running
 4     win7                           running
 5     puppet                         running
 -     desktop                        shut off

[root@vmhost tmp]#
As you can see, desktop did not come back. Let's see if I can persuade it to start manually:
[root@vmhost tmp]# virsh start desktop
error: Failed to start domain desktop
error: internal error Did not find USB device 2341:1 bus:1 device:5

[root@vmhost tmp]#
Ok, who is that usb guy?
[root@vmhost tmp]# virsh dumpxml desktop | grep 2341
[root@vmhost tmp]#
Hmmm, nobody matching that vendor ID. But, wait! If managedsave saved the vm client before vmhost shutdown, the saved image should be in /var/lib/libvirt/qemu/save/, right?
[root@vmhost tmp]# ls /var/lib/libvirt/qemu/save/
desktop.save  lost+found
[root@vmhost tmp]#
We are making progress! Now, /var/lib/libvirt/qemu/save/desktop.save contains, amongst other things, a copy of the config as it was when the file was created. So, if we cheat a bit, we can see that file has the following entries:
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='5'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='6'/>
  </source>
</hostdev>
which show two hostdev entries related to our Arduino device. We can also notice the first one is the one related to the
error: internal error Did not find USB device 2341:1 bus:1 device:5
error message.

Our priority right now is to get desktop to boot. The quickest way to do so is to add startupPolicy='optional' to the

<source missing='yes'>
lines, which would tell virsh that it is ok to boot if those hostdevs are not around.
[root@vmhost tmp]# virsh save-image-edit --file /var/lib/libvirt/qemu/save/desktop.save
State file /var/lib/libvirt/qemu/save/desktop.save edited.
[root@vmhost tmp]#
If all goes well, we should now be able to start desktop:
[root@vmhost tmp]# virsh start desktop
Domain desktop started
[root@vmhost tmp]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     nameserver                     running
 3     nagios                         running
 4     win7                           running
 5     puppet                         running
 9     desktop                        running

[root@vmhost tmp]#
Success! desktop is once again up and running! And, it is not a reboot. I will not take the time to show it, so you will have to have faith on my lies, but it restored the session and the state as it was before vmhost shut down. I really like that fact (hence putting it in italics), and think this is the most important point in this entire article.

Now that desktop is up and running, virsh dumpxml should work and can be used to verify this is the vm client we edited and then restored. In other words, we should see our USB hostdevs.

[root@vmhost tmp]# virsh dumpxml desktop | less
[...]
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source startupPolicy='optional' missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='5'/>
  </source>
  <alias name='hostdev0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
  <source startupPolicy='optional' missing='yes'>
    <vendor id='0x2341'/>
    <product id='0x0001'/>
    <address bus='1' device='6'/>
  </source>
  <alias name='hostdev1'/>
</hostdev>
[root@vmhost tmp]#
I do not know about you, but I think it is high time to remove those bastards. Our plan is to create a file similar to what we used to tell desktop about the USB device. Since we have two entries we want to delete, we need to be more specific. However, we can be lazy and just copy the hotsdev entries from above (we will only delete one of them at a time so we can show we can specify exactly which hostdev we want to excise) into a .xml file:
[root@vmhost tmp]# cat > arduino-out.xml << 'EOF'
<hostdev mode='subsystem' type='usb' managed='yes'>
<source startupPolicy='optional' missing='yes'>
<vendor id='0x2341'/>
<product id='0x0001'/>
<address bus='1' device='5'/>
</source>
<alias name='hostdev0'/>
</hostdev>
EOF
In a previous article, we showed how to use virsh attach-device to, well, add the USB device. Therefore, it should not surprise us that virsh detach-device does the opposite:
[root@vmhost tmp]# virsh detach-device desktop arduino-out.xml
Device detached successfully

[root@vmhost tmp]#
If it worked, we should only have one USB device with vendor ID=0x2341 here.
[root@vmhost tmp]# virsh dumpxml desktop | grep 2341
<vendor id='0x2341'/>
[root@vmhost tmp]# virsh dumpxml desktop | less
<hostdev mode='subsystem' type='usb' managed='yes'>
<source startupPolicy='optional' missing='yes'>
<vendor id='0x2341'/>
<product id='0x0001'/>
<address bus='1' device='6'/>
</source>
<alias name='hostdev1'/>
</hostdev>
[root@vmhost tmp]#
So, if we edit the arduino-out.xml file so it says device='6', and run virsh detach-device desktop arduino-out.xml again, the last USB entry related to the Arduino should finally be gone.

Moral of the Story

  1. Use startupPolicy='optional' so vm client will restart even if it cannot find a USB device. This might also work with a PCI passthrough device but I have not checked yet.
  2. We can edit the config of a saved vm.
  3. All of our cocking about did not cause the vm client to reboot. It was restored to the exact status it was before we saved it.
  4. This article was long enough to be its own thing.

USB Passthrough in KVM/libvirt: How to talk to a UPS

I was writing an article about how to connect an Arduino to a kvm vm host so a vm guest can access it. Since there were some issues I would like to share, I decided to break it down in two parts. First would be this very article, dealing with setting up USB passthrough. In a future article we will talk about issues using Arduino with USB passthrough.

Last time I talked about passthrough was in a post about using grep to look for a vm client which uses PCI passthrough. I never really mentioned how to do the deed there because I assumed everyone knew how to do that in KVM. Should we talk about how to set that up? You tell me. Since I will be talking about using USB passthrough here, you can see if it is enough to understand and use PCI passthrough. If not, let me know and I will write a few lies about it.

Since I kept talking about the Arduino, I will not use it as the example here. Instead, an APC-brand UPS is chosen for the unbiased version that it is what I have here. Here's the idea: let's say you have a vm client running something like Nagios or Icinga in it. Why not use it to monitor the UPS and let us know (or initiate some automated operation) if it is in use and/or the remaining charge drops below a certain level?

We will be adding the APC to the vm client scan, which is an CentOS vm client/guest in the KVM-based with libvirt vm host called, for reasons the go beyond the topic of this article, vmhost.

Install and Setup

  1. The UPS in question has a USB cable that is used to monitor/talk to it. Now I know as matter of fact that apcupsd works well with that setup, so we should install it in our nagios (I am using this as the vm client's name for the sake of lazyness) server. To make a long story short, since this is RedHat-based distro we can get it by
    yum install apcupsd --enablerepo=epel
  2. Let's make sure the vm can see the APC box. To do so, we first need to see if the vm host, affectionately known as vmhost because of some unexplained reason, know we plugged the console USB cable from the APC:
    [root@vmhost tmp]# lsusb
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
    [root@vmhost tmp]#
    You have to agree the line
    Bus 002 Device 003: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
    is a dead giveaway that this is the APC-brad UPS. Now we identified the entry, we want to get the USB vendor and product IDs. That will be the 051d:0002, respectively.
  3. Next we need to reconfigure scan so it knows it has a new USB device attached to it. A USB device is a seen by KVM as hostdev element. From the libvirt docs, a hostdev is the container that describe host devices. We could shutdown scan and edit its config file, but we should/better be able to do it live. First we create a little file containing the hostdev definition:
    cat > apc.xml << 'EOF'
        <hostdev mode='subsystem' type='usb' managed='yes'>
          <source startupPolicy='optional'>
            <vendor id='0x051d'/>
            <product id='0x0002'/>
          </source>
        </hostdev>
    EOF
    You can see the vendor and product IDs, which we gathered from last step. The reason for the startupPolicy='optional' is that if the machine needs to reboot, or come back from being saved, it will do so even if the usb device is no longer reachable. Now, we should let the vm client know about the UPS:
    [root@vmhost tmp]# virsh attach-device nagios apc.xml
    Device attached successfully
    
    [root@vmhost tmp]# 
  4. Does our vm client see it now?
    [raub@scan ~]$ lsusb
    Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
    Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
    Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
    Bus 001 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
    [raub@scan ~]$
    It seems it is there, so we should test the connection using apcupsd
    [root@scan ~]# service apcupsd start
    grep: /etc/nologin: No such file or directory
    Starting UPS monitoring:                                   [  OK  ]
    [root@scan ~]# service apcupsd status
    apcupsd (pid  19373) is running...
    APC      : 001,036,0917
    DATE     : 2015-05-14 12:30:43 -0400
    HOSTNAME : scan.in.example.com
    VERSION  : 3.14.10 (13 September 2011) redhat
    UPSNAME  : ups1
    CABLE    : USB Cable
    DRIVER   : USB UPS Driver
    UPSMODE  : Stand Alone
    STARTTIME: 2015-05-14 12:30:42 -0400
    MODEL    : Back-UPS RS 1500G
    STATUS   : ONLINE
    LINEV    : 124.0 Volts
    LOADPCT  :  21.0 Percent Load Capacity
    BCHARGE  : 100.0 Percent
    TIMELEFT :  38.4 Minutes
    MBATTCHG : 5 Percent
    MINTIMEL : 3 Minutes
    MAXTIME  : 0 Seconds
    SENSE    : Medium
    LOTRANS  : 088.0 Volts
    HITRANS  : 147.0 Volts
    ALARMDEL : 30 seconds
    BATTV    : 27.0 Volts
    LASTXFER : Automatic or explicit self test
    NUMXFERS : 0
    TONBATT  : 0 seconds
    CUMONBATT: 0 seconds
    XOFFBATT : N/A
    SELFTEST : NO
    STATFLAG : 0x07000008 Status Flag
    SERIALNO : 3B1416X00241
    BATTDATE : 2014-04-14
    NOMINV   : 120 Volts
    NOMBATTV :  24.0 Volts
    NOMPOWER : 865 Watts
    FIRMWARE : 865.L5 .D USB FW:L5
    END APC  : 2015-05-14 12:30:47 -0400
    [root@scan ~]#
    Smells like our USB passthrough adventure worked.
We could go on and configure Nagios to do the UPS monitoring, but that will be left for another article; if you want to get ahead, a starting point would be to search for "nagios apcupsd". Remember, all we wanted here is to get the USB passthrough part working. For the next article we will talk about when things do not work as peachy.

Removing the USB device from the vm guest

  1. If you want to remove it programatically,
    virsh detach-device nagios apc.xml
    will remove it from nagios's config. lsusb on the vm client (nagios) will show it is gone.
  2. If you just physically pluck the USB device from vmhost, the client will report it as gone However, it will still be in the vm client's config file. Here is an example of a USB device with vendor ID=2341 that I physically plucked from more than once (and then put it back and added to the client):
    [root@vmhost tmp]# virsh dumpxml desktop|grep 2341
            <vendor id='0x2341'/>
            <vendor id='0x2341'/>
    [root@vmhost tmp]#

    The solution for that is to keep removing it (virsh detach-device nagios apc.xml) until virsh dumpxml stops reporting it is there. And then add it again.

    Let me show this step-by-step with the AUP example:

    1. We start by not seeing the APC UPS in the vm client:
      [root@scan ~]# lsusb
      Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
      Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
      Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
      [root@scan ~]#
    2. The vm host tells us there are 3 entries in the vm client's config:
      [root@vmhost tmp]# virsh dumpxml nagios|grep  051d
              <vendor id='0x051d'/>
              <vendor id='0x051d'/>
              <vendor id='0x051d'/>
      [root@vmhost tmp]#
    3. Let's start removing them:
      [root@vmhost tmp]# virsh detach-device nagios apc.xml
      Device detached successfully
      
      [root@vmhost tmp]# virsh detach-device nagios apc.xml
      Device detached successfully
      
      [root@vmhost tmp]# virsh dumpxml nagios|grep  051d
              <vendor id='0x051d'/>
      [root@vmhost tmp]# 
    4. Sounds like we can't count. We need to do the deed 3 times, so we do it one last time and see if the device is finally gone:
      [root@vmhost tmp]# virsh detach-device nagios apc.xml
      Device detached successfully
      
      [root@vmhost tmp]# virsh dumpxml nagios|grep  051d
      [root@vmhost tmp]# 
    5. Success! Now we add it
      [root@vmhost tmp]# virsh attach-device nagios apc.xml
      Device attached successfully
      
      [root@vmhost tmp]#
    6. And check in the vm client if it is back
      [root@scan ~]# lsusb
      Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
      Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
      Bus 001 Device 003: ID 0409:55aa NEC Corp. Hub
      Bus 001 Device 009: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
      [root@scan ~]#
  3. If you save the vm client (virhs managedsave or virsh save), exciting things will happen.

Sunday, March 23, 2014

When upgrades go bad: Installing JunOS from USB in a SRX router

So, I screwed up pretty bad. I decided to upgrade the JunOS release in this Juniper SRX210 router to the one (at the time I type this) recommended by Juniper, 11.4R10.3. When it booted up after the install, it crashed during the boot process. Well, I could have spent the time kicking myself but I am doing this upgrade off-hours and I did account for things going badly in my downtime estimate. And, this router is part of a redundant router setup using the Virtual Router Redundancy Protocol (VRRP); being down will not affect production. In other words, this is more of an annoyance than a real issue. Since I have to deal with this, how about if we learn how to restore the OS in this juniper router?

I tried a few ways and thought that the easiest one was to use a USB drive. Of course, it will not work well if you are not physically close to said router (other things will also not work well in these circumstances but that is another topic), but since I can I am doing the USB upgrade.

Procedure

  1. Get a USB drive. I know, this is a pretty obvious step but it is step 1. Ideally use a 1GB/2GB USB drive, formatted as fat16/fat32. Honestly I do not know how critical that is, but my experience with Cisco, which seems not to like the higher capacity ones, made me be leery. On the plus side, you should be able to find those rather easily as people replace their old ones with newer larger ones. If not, there are always the usual sources such as ebay or amazon.
  2. Download and copy OS image you are going to use, say junos-srxsme-11.4R10.3-domestic.tgz, into USB drive. If you are smarter than me, you would have gone to the Juniper downloads site and got all the OS images you need, placing them in your file server. I wasn't so I had to go the SRX210 download page and fetch it.
  3. Have your trusty serial cable and connect it to the router's console port. The default setup is the time-honored 9600 8N1. If you changed it, make sure you wrote than somewhere. I am lazy and I kinda like that setting.
  4. Connect USB drive to router.
  5. Reboot router after you attack the usb drive to it. It needs to know the drive exists as it boots up. Otherwise, it will bark like this:
    loader> install file:///junos-srxsme-11.4R10.3-domestic.tgz
    cannot open package (error 22)
    loader>

    When you try to install it.

  6. Now, if you boot with USB already connected to router, it will first say something like this:

    Running U-Boot CRC Test... OK.
    Flash:  4 MB
    USB:   scanning bus for devices... 4 USB Device(s) found
           scanning bus for storage devices... 2 Storage Device(s) found
    Clearing DRAM........ done
    BIST check passed.

    Some of you noticed the 2 storage devices message. It is talking about the inboard one (probably where the OS should be) and the external drive.

  7. Now, when you see

    POST Passed
    Press SPACE to abort autoboot in 1 seconds

    Please keep your fingers in your pockets. If you press space here, you will end up in the => prompt (U-boot). If you wait you will then see

    Protected 1 sectors
    Loading /boot/defaults/loader.conf
    /kernel data=0xb0f9c0+0x134788 DA(some hot action happening here)

    have your space-bar finger on standby for the next message will be

    Hit [Enter] to boot immediately, or space bar for command prompt.
  8. Then you will press space bar and get the loader> prompt. And now, it will start doing the install thingie:

    loader> install file:///junos-srxsme-11.4R10.3-domestic.tgz
    /kernel data=0xae82f0+0x12d2b8 syms=[0x4+0x88ce0+0x4+0xc6af6]
    Kernel entry at 0x801000d8 ...
    init regular console
    GDB: debug ports: uart
    GDB: current port: uart
    KDB: debugger backends: ddb gdb
    KDB: current backend: ddb
    Copyright (c) 1996-2013, Juniper Networks, Inc.
    All rights reserved.
    Copyright (c) 1992-2006 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    JUNOS 11.4R10.3 #0: 2013-11-15 06:56:20 UTC
    [...]
  9. After a while (I got bored and went to make me some tea), you will see it recreate the ssh key pairs and then finally be ready for business (apologies for the bad cut-n-pasting but my terminal console was being cute):

    |
    |                 |
    |  .o  ..         |
    |.+o .o.o.
    |X . .. .. E      |
    |oo ..            |
    |  .+             |
    |.-+
    root@uranus% omplete
    Setting initial options: .
    Starting optface configuration:
    additional daemons: eventd.
    Additional rout;/boot/modules -> /bo;
    kld netpfe drv: ifpfed_dialer default_adtwork setup:.
    Starting final network daemons:.
    setting ldconfig.
    Initial rc.mips initialization:.
    Local package initializationup access
    .
    kern.securelevel: -1 -> 1
    Creating JAIL MFS partitirade.uboot="0xBFC00000"
    boot.upgrade.loader="0xBFE00000"
    Boot mILE SYSTEM CLEAN; SKIPPING CHECKS
    clean, 78249 free (17 frags, ar 20 16:46:25 CDT 2014
    
    uranus (ttyu0)

    Note that it remembered the hostname for the router. I still went through the configs before letting it join the router cluster. But that is pretty much it! Router is back in business.

Closing Thoughts

  1. The universe is Murphian; things will go wrong. Try not to stress about that.
  2. When you schedule downtime for upgrades, account for things going badly in your time estimates.
  3. The hardest thing to do is figuring out what can go wrong. But, you could ask yourself "If this upgrade halts server or just this service, what would be my backup plan?" and then see if you can answer that question.
  4. Next time I need to upgrade the OS in this or another router, I will have the firmware/OS on standby in a USB drive. I do not know about you but I found out when I am prepared everything works out perfectly.
  5. If you can afford it, redundancy is a wonderful thing.
  6. Always save your configs somewhere, well, safe. Having to recreate them from scratch is a bit of a drag.

Tuesday, December 31, 2013

Notes on resetting and connecting to a Juniper router

This is another of those notes I wrote primarily to myself. It has to do with a Juniper appliance, namely a SSG5, which runs ScreenOS. I had some issues with its configuration, as I screwed up and could not log into it from either the network port or console. It felt it was high time to wipe and reconfigure the little guy.

Juniper has some notes on doing the deed, but there are a few things I would like to mention:

  1. You really want to do this resetting dance while the outer is not connected to any network. You know, just in case someone does recognize a router in default mode and have a field day.

  2. Having a good DB-9 RS-232-to-usb cable makes all the difference. I would strongly recommend one using the FTDI Chipset. Without that you might end up rather frustrated. There are a lot of companies, FTDI itself included, making such cables. For the lazy and curious amongst you, the one I personally own is the Sabrent USB2-to-RS-232 cable, model CB-FTDI.

  3. Find something convenient to reach the button, and a way to hold the router in place. When I first tried it, I used a trusty paperclip to press the reset button on the back. The brilliant (at the time) idea but was that if I could hold both the router and paper clip with one hand, I then would be able to see the lights on the front of the router. It would work fine if I was holding router with its back towards me. In real life, with me trying to see its blinking light, the paper clip kept sliding off the reset button, lodging itself between the button and the board (I think; I haven't opened it). What worked for me was a mechanical pencil. Its (7mm) tip was thick enough to just fit the reset button hole and its body felt just right to hold from the back.

  4. Resetting the router turned to be close enough to what was described in Juniper notes on resetting this router, but not exactly. Specifically,

    1. When you press the reset button, hold it until it starts blinking orange. Until that happens, just keep on pressing the button.
    2. Once it starts bliking orange, let it go. It will go green.
    3. Wait 2-4 seconds and then press the reset button again. The exact time might need some practice; in my case it was more like 3s. You know you got it right because once you press the button the LED will start blinking red.
    4. Now (led blinking red) release the router reset button and let it do the boot process continue.
  5. Know the serial port settings: 9600 8N1, the same as many Cisco devices. How you will configure that and connect to router is up to you. I have used screen (Linux/OSX/Others), minicom (Linux), and tip (Solaris), but I do know Windows also has a terminal program (HyperTerminal?) that comes with it that will work just fine. Or putty. I like putty.
  6. Running a packet acquisition package in a router LAN port is quite useful, specially if you have setup router to use a different network/IP than the default. When I first did the reset, the USB-to-serial cable I had was not working with the router's serial port.

    While I was waiting for the CB-FTDI cable mentioned above, I used wireshark (I was feeling lazy; nothign stopping you to use something fancier you already had... or write your own routine) to look at the traffic at the lan. Before it was reset, the router would keep sending arp requests in broadcast. And that would tell me which network it was configured to use, which was not the default (192.168.1.0/24). Now, as soon as I successsfully reset the router, traffic went quiet during bootup but then I started seeing traffic from 192.168.1.1. As the only device in that network -- my ethernet cable was just connected to the laptop doing the packet capture -- that told me the reset was successful. Then, I turned wireshark off, set some ip in the same network for the ethernet port connected to router, and checked if the web interface mentioned in the manual, and which I have never used, was there. Nope, all ports were closed. So I just had to wait for USP to deliver the USB-to-serial cable.

  7. As the manual and the link states, the default login and password are both nnetscreen.

Wednesday, June 15, 2011

screen + minicom (how to get out of)

So, I did it again. You see, I like to use minicom as a terminal program since it plays nice with my usb-to-serial cables and my serial devices (Sun and AIX workstations, Seagate Dockstar, Pogoplug, routers, switches). And I have used it long enough to remember some of the commands. I also like screen, which allows me to run multiple sessions on different machines and stop and resume them as needed; kinda like running vnc/rdp but in command-line.

The problem begins when I try to use them together. I really should not do that because you can connect to a serial port from screen by doing something like

:screen /dev/ttyUSB0 115200
(some devices I use have their default port speed set to 9600, others like the one in this example to 115200)

But, force of habit (read: lazyness) or distraction (did you see that ant crawling up the wall?) caused me to start minicom right in screen. At first that does not seem to be a bad idea; after all, it would be nice to connect to the console of, say, a Brocade fabric switch, start something there, log out, and then come back to it later. But, problem is that screen and minicom by default use the same CTRL-A sequence to enter commands. So, if I want to send a CTRL-A X (exit) to minicom, screen will think I am talking to it and then lock the screen. Not fun.

Some of you will argue that I could remap the escape sequences for each program so they will not match, but hindsight is 20/20: it does not do me any good once they are already running inside each other. Another option would be to close that screen session, which would quit minicom... but not gracefully. As a result, that serial port (/dev/ttyUSB0 in my case. Yes, it is USB but you know exactly what I am talking about) would not be freed even if I remove the serial-to-USB cable. We need something better.

If we look at the screen man page, we will find the escape sequence CTRL-a a. What it does is send a CTRL-a escape to the screen session, so whatever program running in that session can grab and run away with it. So, what we want to do is probably CTRL-a a x, which should quit minicom. Now, the trick here is speed: you have to press CTRL-a together, then immediately a and then x. It took a while for me because I was either not typing fast enough or just mashing keys together. But, once I got it just right, I was rewarded with the minicom quit dialog box:

+----------------------+
|    Leave Minicom?    |
|     Yes       No     |
+----------------------+

And now I was able to properly quit minicom. Life was good once again.

Do I plan on running minicom inside screen again? Not if I can avoid it. But, if I can't, now I have a way out.

Wednesday, February 11, 2009

Of Macs and serial ports

My trusty iBook, as all Macintosh computers manufactured in the last few years, have no serial port. That has never stopped me from doing work as I had a Linux laptop, a Dell Latitude D600, which I would bring whenever I needed to talk to a Cisco switch or use as console for a Unix workstation (say, Sun Solaris or IBM AIX box).

But, then, the Dell laptop died. And I needed to configure a cisco switch from scratch... at least configure it enough so I could then telnet to it. To do that I needed to connect the famous Cisco blue console cable to the Mac. I needed a usb-to-serial cable.

Not knowing where to find one of those usb-to-serial cables, I decided to try one of my favorite places: geeks.com. I not only found it but here is a picture of the cable:

Clicking on the image *should* lead you to the link for the cable. After I received it, I connected it to the Mac. The laptop was aware of the device, even recognizing its chipset. But, it would not be available for use. Here is what I mean:

Mireille:~ dalek$ ls /dev/tty.*
/dev/tty.Bluetooth-Modem                /dev/tty.Nokia6103-NokiaPCSuite-1
/dev/tty.Bluetooth-PDA-Sync             /dev/tty.modem
/dev/tty.Nokia6103-Dial-upnetwor-2
Mireille:~ dalek$ 

Clearly, I need a driver for it. Examining the information shown by the machine about the driver, we see the chipset is made by prolific. After a bit of searching online, I found the manufacturer's site and downloaded the drivers from its site. Do note in that page that they also have drivers for Windows and even Linux. I do not know if Linux would ever need such a driver; finding that out is for a different episode. Anyway, after installing it, we had to reboot the laptop. After that, it was time to connect the usb-to-serial cable and find out if it was seen as a device we could use. Can you spot the new entry?

Mireille:~ dalek$ ls /dev/tty.*
/dev/tty.Bluetooth-Modem                /dev/tty.Nokia6103-NokiaPCSuite-1
/dev/tty.Bluetooth-PDA-Sync             /dev/tty.modem
/dev/tty.Nokia6103-Dial-upnetwor-2      /dev/tty.usbserial
Mireille:~ dalek$ 

Now we have a device, tty.usbserial, we can try it out. We could install minicom using fink, but we can be a bit lazy and use, of all things, screen. Believe it or not, screen can also be used to connect to a terminal device. So, if you type something like

Mireille:~ dalek$ screen /dev/tty.usbserial 9600

you would be telling screen to connect to our usb-to-serial cable, identified as tty.usbserial, at 9600baud which happens to be the default port speed for a Cisco switch. Neat, huh?

Sunday, August 27, 2006

Backing up

It has been said that a business will not survive if it does not have a good backup system. I personally agree with it. I also have noticed that many companies only consider backup solutions after the lack of it bit them in their behinds, resulting in data and productivity losses, credibility issues, and, above all, losses where it counts the most: the wallet. So, why would they avoid it like most of us, excluding characters like a certain guy from Little Shop of Horrors, would avoid a visit to the dentist? Well, sometimes it has to do with perceived cost. Since backup does not directly translates to making profit, it is seen as waste of good money which could be used in more productive manners. Kinda like boats.

Then, we have the old adage, do not change a winning team. In other words, if the system is working, do not mess with it by adding this newflanged and unproven backup thingamajig. That also implies that a properly configured system should not need backup. So, if the system has a problem -- something that is to be expected as no systems are perfect (yes, I know, there are systems that are less perfect than others) and hardware will go boink before tea-time -- someone did not do his/hers/its share. As a result, heads in the IT group will roll.

Anywhoo, the point is that files will be lost (either by malice or by mistake) and machines will crash and hard drives will fail and other things will happen to make your original copy of the data unusable. At that point, if you have backup you only lose a few hours worth of work if that. If you do not, well, you may have lost years of data. It happened to me and I can tell you it is not a nice feeling.

All this is nice and boring. How hard is to have a backup? And, how expensive it can be? It depends on what do you want to do, how much effort you are willing to put on it, and how nice you want it to be. You have price, speed, and quality, but can only pick two of them. That is fine. How complex a backup system do you need? Sometimes not that much if your needs are small. Let me present an example based on a real story. I was asked by a company I consult for to come up with a simple, hands-off backup of their most important application (in this case a program they use to keep track of what happens in their shop). One of the most important issues is that they want to be able to continue working in case their application server goes on a holiday. And, they run Windows XP. Yes, I know this blog is supposed to be about unix, but the principle is what matters. I promise I will show later an unix example.

My approach was first to understand the program. Because it is dos-based (that is a choice done by the developer, and they do go over great lengths to justify it. It really boils down to: it is works nicely and does not need much resources to run) program, it is placed on a directory which is then exported to the machines that are authorized to mount it (sounds familiar? Can you say NFS + NIS/LDAP? I bet you can). Interestingly enough, for it to work the entire system disk of the server machine that hosts the program must be exported with permissions set for everybody who access the program be able to read and write to the disk for it to work. Why? I do not know but do think that is a safety issue. But, I digress...

When the client said he wanted to be able to continue working in case of system crash, what he really meant was that if the server crashed, he would be able to go to another PC in his network and run off the backup copy. So, it was decided the most convenient backup media was an USB hard drive formated with one NTFS partition. To keep this quite Windows-centric, we chose to use robocopy, which is found in the Microsoft website, as the underlining program we create a little script (or batch file for those of you who are anal about terminology), that would be called to do out daily backups.

:: BACKUP.BAT
:: Backs up a single directory from one location to another.  Since we are using robocopy,
:: the locations can be in local or network drives.
@echo off

:: What to copy and where to copy to
set dirname=shopmanager
set sourcepath=z:set targetpath=f:set sourcedir=%sourcepath%%dirname%
set targetdir=%targetpath%%dirname%

:: Just to be on the safe side, I am defining where robocopy has been 
:: installed. BTW, this is the default path.
set COPYPATH=C:\Program Files\Windows Resource Kits\Tools\robocopy
set COPYARGS=/e /copyall

:: Add today's day number to targetdir
set today=%date:~7,2%
set targetdir=%targetpath%_%today%

echo %COPYPATH% %sourcedir% %targetdir% %COPYARGS%

And