What that orange alarm LED light in a Juniper SRX router is trying to tell me?

If you have a Juniper SRX router thingie, you might have noticed the orange light glowing on it:

It is the alarm light, and could have been triggered by many reasons, like the one mentioned in http://unixwars.blogspot.com/2015/08/juniper-srx-router-booted-from-backup.html>later post. On its defense, it is nice to know the silly router is upset about something. And, it sure beats a blinking 200W light or a blaring air raid siren. Still, it is staring at me with its deep unmoving orange eyes demanding attention. So, let's do some probulating, shall we?

When I asked it what's up, this is what it told me:

root@uranus> show system alarms 
2 alarms currently active
Alarm time               Class  Description
2014-03-23 10:42:35 EDT  Minor  Autorecovery information needs to be saved
2014-03-23 10:42:33 EDT  Minor  Rescue configuration is not set

root@uranus>

After I saw that, it hit me like a, er, something heavy (please come up with something more original than the usual ton of bricks. I have never been hit by one and plan on staying that way) and unyielding: about that time I did a full wipe and reinstall! So, since it is being so nice to tell us what it wants, let's see about pleasing it. If we look at this thread in the juniper forums, we see the command to save the rescue configuration is:

root@uranus> request system configuration rescue save 

root@uranus> show system alarms                          
1 alarms currently active
Alarm time               Class  Description
2014-03-23 10:42:35 EDT  Minor  Autorecovery information needs to be saved

root@uranus>

One down, one to go. Now, how to save the autorecovery info? I am going to punt and assume the command should be very similar to the one we used to save the rescue info. Like Cisco's IOS, you can use ? to see which arguments a give command take. So, we try

root@uranus> request system configuration ?    
Possible completions:
  rescue               Request operation on system rescue configuration
root@uranus> request system ?                 
Possible completions:
  autorecovery         Manage autorecovery information
  certificate          Manage X509 certificates
  configuration        Request operation on system configuration
  download             Manage downloads
  firmware             Upgrade or downgrade firmware
  halt                 Halt the system
  license              Manage feature licenses
  logout               Forcibly end user's CLI login session
  power-off            Power off the system
  reboot               Reboot the system
  scripts              Manage scripts (commit, op, event)
  services             Request service applications information
  set-encryption-key   Set EEPROM stored encryption key
  snapshot             Archive data and executable areas
  software             Perform system software extension or upgrade
  storage              Request operation on system storage
  zeroize              Erase all data, including configuration and log files
root@uranus> request system ?

Aha, we found autorecovery. Which arguments does request system autorecovery take?

root@uranus> request system autorecovery ?  
Possible completions:
  state                Manage autorecovery state information
root@uranus> request system autorecovery state ?
Possible completions:
  clear                Delete previously saved autorecovery state
  recover              Check for problems and recover state if needed
  save                 Save autorecovery state
root@uranus> request system autorecovery state ?

So it seems that request system autorecovery state save will do the trick. Let's try it then:

root@uranus> request system autorecovery state save 
Saving config recovery information
Saving license recovery information
Saving BSD label recovery information

root@uranus>

and the orange light's gone! Another mystery solved...

When upgrades go bad: Installing JunOS from USB in a SRX router

So, I screwed up pretty bad. I decided to upgrade the JunOS release in this Juniper SRX210 router to the one (at the time I type this) recommended by Juniper, 11.4R10.3. When it booted up after the install, it crashed during the boot process. Well, I could have spent the time kicking myself but I am doing this upgrade off-hours and I did account for things going badly in my downtime estimate. And, this router is part of a redundant router setup using the Virtual Router Redundancy Protocol (VRRP); being down will not affect production. In other words, this is more of an annoyance than a real issue. Since I have to deal with this, how about if we learn how to restore the OS in this juniper router?

I tried a few ways and thought that the easiest one was to use a USB drive. Of course, it will not work well if you are not physically close to said router (other things will also not work well in these circumstances but that is another topic), but since I can I am doing the USB upgrade.

Procedure

1. Get a USB drive. I know, this is a pretty obvious step but it is step 1. Ideally use a 1GB/2GB USB drive, formatted as fat16/fat32. Honestly I do not know how critical that is, but my experience with Cisco, which seems not to like the higher capacity ones, made me be leery. On the plus side, you should be able to find those rather easily as people replace their old ones with newer larger ones. If not, there are always the usual sources such as ebay or amazon.
   
2. Download and copy OS image you are going to use, say junos-srxsme-11.4R10.3-domestic.tgz, into USB drive. If you are smarter than me, you would have gone to the Juniper downloads site and got all the OS images you need, placing them in your file server. I wasn't so I had to go the SRX210 download page and fetch it.
   
3. Have your trusty serial cable and connect it to the router's console port. The default setup is the time-honored 9600 8N1. If you changed it, make sure you wrote than somewhere. I am lazy and I kinda like that setting.
   
4. Connect USB drive to router.
   
5. Reboot router after you attack the usb drive to it. It needs to know the drive exists as it boots up. Otherwise, it will bark like this:

   loader> install file:///junos-srxsme-11.4R10.3-domestic.tgz
   cannot open package (error 22)
   loader>

   When you try to install it.

6. Now, if you boot with USB already connected to router, it will first say something like this:

   Running U-Boot CRC Test... OK.
   Flash:  4 MB
   USB:   scanning bus for devices... 4 USB Device(s) found
          scanning bus for storage devices... 2 Storage Device(s) found
   Clearing DRAM........ done
   BIST check passed.

   Some of you noticed the 2 storage devices message. It is talking about the inboard one (probably where the OS should be) and the external drive.

7. Now, when you see

   POST Passed
   Press SPACE to abort autoboot in 1 seconds

   Please keep your fingers in your pockets. If you press space here, you will end up in the => prompt (U-boot). If you wait you will then see

   Protected 1 sectors
   Loading /boot/defaults/loader.conf
   /kernel data=0xb0f9c0+0x134788 DA(some hot action happening here)

   have your space-bar finger on standby for the next message will be

   Hit [Enter] to boot immediately, or space bar for command prompt.

8. Then you will press space bar and get the loader> prompt. And now, it will start doing the install thingie:

   loader> install file:///junos-srxsme-11.4R10.3-domestic.tgz
   /kernel data=0xae82f0+0x12d2b8 syms=[0x4+0x88ce0+0x4+0xc6af6]
   Kernel entry at 0x801000d8 ...
   init regular console
   GDB: debug ports: uart
   GDB: current port: uart
   KDB: debugger backends: ddb gdb
   KDB: current backend: ddb
   Copyright (c) 1996-2013, Juniper Networks, Inc.
   All rights reserved.
   Copyright (c) 1992-2006 The FreeBSD Project.
   Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
           The Regents of the University of California. All rights reserved.
   JUNOS 11.4R10.3 #0: 2013-11-15 06:56:20 UTC
   [...]

9. After a while (I got bored and went to make me some tea), you will see it recreate the ssh key pairs and then finally be ready for business (apologies for the bad cut-n-pasting but my terminal console was being cute):

   |
   |                 |
   |  .o  ..         |
   |.+o .o.o.
   |X . .. .. E      |
   |oo ..            |
   |  .+             |
   |.-+
   root@uranus% omplete
   Setting initial options: .
   Starting optface configuration:
   additional daemons: eventd.
   Additional rout;/boot/modules -> /bo;
   kld netpfe drv: ifpfed_dialer default_adtwork setup:.
   Starting final network daemons:.
   setting ldconfig.
   Initial rc.mips initialization:.
   Local package initializationup access
   .
   kern.securelevel: -1 -> 1
   Creating JAIL MFS partitirade.uboot="0xBFC00000"
   boot.upgrade.loader="0xBFE00000"
   Boot mILE SYSTEM CLEAN; SKIPPING CHECKS
   clean, 78249 free (17 frags, ar 20 16:46:25 CDT 2014
   
   uranus (ttyu0)

   Note that it remembered the hostname for the router. I still went through the configs before letting it join the router cluster. But that is pretty much it! Router is back in business.

Closing Thoughts

1. The universe is Murphian; things will go wrong. Try not to stress about that.
   
2. When you schedule downtime for upgrades, account for things going badly in your time estimates.
   
3. The hardest thing to do is figuring out what can go wrong. But, you could ask yourself "If this upgrade halts server or just this service, what would be my backup plan?" and then see if you can answer that question.
   
4. Next time I need to upgrade the OS in this or another router, I will have the firmware/OS on standby in a USB drive. I do not know about you but I found out when I am prepared everything works out perfectly.
   
5. If you can afford it, redundancy is a wonderful thing.
   
6. Always save your configs somewhere, well, safe. Having to recreate them from scratch is a bit of a drag.
   

Notes on resetting and connecting to a Juniper router

This is another of those notes I wrote primarily to myself. It has to do with a Juniper appliance, namely a SSG5, which runs ScreenOS. I had some issues with its configuration, as I screwed up and could not log into it from either the network port or console. It felt it was high time to wipe and reconfigure the little guy.

Juniper has some notes on doing the deed, but there are a few things I would like to mention:

1. You really want to do this resetting dance while the outer is not connected to any network. You know, just in case someone does recognize a router in default mode and have a field day.

2. Having a good DB-9 RS-232-to-usb cable makes all the difference. I would strongly recommend one using the FTDI Chipset. Without that you might end up rather frustrated. There are a lot of companies, FTDI itself included, making such cables. For the lazy and curious amongst you, the one I personally own is the Sabrent USB2-to-RS-232 cable, model CB-FTDI.

3. Find something convenient to reach the button, and a way to hold the router in place. When I first tried it, I used a trusty paperclip to press the reset button on the back. The brilliant (at the time) idea but was that if I could hold both the router and paper clip with one hand, I then would be able to see the lights on the front of the router. It would work fine if I was holding router with its back towards me. In real life, with me trying to see its blinking light, the paper clip kept sliding off the reset button, lodging itself between the button and the board (I think; I haven't opened it). What worked for me was a mechanical pencil. Its (7mm) tip was thick enough to just fit the reset button hole and its body felt just right to hold from the back.

4. Resetting the router turned to be close enough to what was described in Juniper notes on resetting this router, but not exactly. Specifically,
   

   1. When you press the reset button, hold it until it starts blinking orange. Until that happens, just keep on pressing the button.
      
   2. Once it starts bliking orange, let it go. It will go green.
      
   3. Wait 2-4 seconds and then press the reset button again. The exact time might need some practice; in my case it was more like 3s. You know you got it right because once you press the button the LED will start blinking red.
      
   4. Now (led blinking red) release the router reset button and let it do the boot process continue.
      
5. Know the serial port settings: 9600 8N1, the same as many Cisco devices. How you will configure that and connect to router is up to you. I have used screen (Linux/OSX/Others), minicom (Linux), and tip (Solaris), but I do know Windows also has a terminal program (HyperTerminal?) that comes with it that will work just fine. Or putty. I like putty.
   

6. Running a packet acquisition package in a router LAN port is quite useful, specially if you have setup router to use a different network/IP than the default. When I first did the reset, the USB-to-serial cable I had was not working with the router's serial port.

   While I was waiting for the CB-FTDI cable mentioned above, I used wireshark (I was feeling lazy; nothign stopping you to use something fancier you already had... or write your own routine) to look at the traffic at the lan. Before it was reset, the router would keep sending arp requests in broadcast. And that would tell me which network it was configured to use, which was not the default (192.168.1.0/24). Now, as soon as I successsfully reset the router, traffic went quiet during bootup but then I started seeing traffic from 192.168.1.1. As the only device in that network -- my ethernet cable was just connected to the laptop doing the packet capture -- that told me the reset was successful. Then, I turned wireshark off, set some ip in the same network for the ethernet port connected to router, and checked if the web interface mentioned in the manual, and which I have never used, was there. Nope, all ports were closed. So I just had to wait for USP to deliver the USB-to-serial cable.
   

7. As the manual and the link states, the default login and password are both nnetscreen.