Friday, August 07, 2015

Juniper SRX router booted from backup image (or, orange LED light again!)

So, uranus is in pain once more. Power went out and it was not properly shut down before the UPS gave its last gasp (I do need to do something about that). When I rebooted it, it seemed to have come back without an issue -- it was working fine -- but it had The Light on again just like it was in last time. I ssh'd into it and this is what the motd looked like:

raub@desktop:~$ ssh janitor@uranus.example.com
janitor@uranus.example.com's password:
--- JUNOS 12.1X45-D25.1 built 2014-04-23 20:45:48 UTC

***********************************************************************
**                                                                   **
**  WARNING: THIS DEVICE HAS BOOTED FROM THE BACKUP JUNOS IMAGE      **
**                                                                   **
**  It is possible that the primary copy of JUNOS failed to boot up  **
**  properly, and so this device has booted from the backup copy.    **
**                                                                   **
**  Please re-install JUNOS to recover the primary copy in case      **
**  it has been corrupted.                                           **
**                                                                   **
***********************************************************************

janitor@uranus>
Hmmm, so the normal image (does that mean the entire OS partition or just the boot stuff?) got corrupted when power went bye-bye. Good thing it keeps a backup. Out of curiosity, let's see if that is related to the LED being on:
janitor@uranus> show chassis alarms
1 alarms currently active
Alarm time               Class  Description
2015-02-26 05:38:12 EST  Minor  Host 0 Boot from backup root

janitor@uranus>
It seems to be the case. Funny that it is labelled minor but important enough to become the motd, but I digress. Personally, we should take care of that as soon as we have a scheduled maintenance window.

Ok, maintenance window time. Before rebooting, let's prepare a few things. If we know the backup copy is good (configuration files are also backed up, and you can push them back with ansible or whatnot if you feel uncomfortable), you could be lazy like me and copy the backup into the primary partition.

janitor@uranus> request system snapshot slice alternate
Formatting alternate root (/dev/da0s1a)...
Copying '/dev/da0s2a' to '/dev/da0s1a' .. (this may take a few minutes)
The following filesystems were archived: /

janitor@uranus>
If all went well, we should see the primary snapshot having the creation date of when we ran the ,tt>request system snapshot slice alternate command.
janitor@uranus> show system snapshot media internal
Information for snapshot on       internal (/dev/da0s1a) (primary)
Creation date: Apr 19 22:34:04 2015
JUNOS version on snapshot:
  junos  : 12.1X45-D25.1-domestic
Information for snapshot on       internal (/dev/da0s2a) (backup)
Creation date: Feb 26 05:33:18 2015
JUNOS version on snapshot:
  junos  : 12.1X45-D25.1-domestic

janitor@uranus>
As you noticed, I did this a while ago but never got around to making an article about it, but there it is. However, I still had the evil LED staring at me. Time to turn it off
request system reboot media internal
So I rebooted and did not get that motd anymore not the LED:
janitor@uranus> show chassis alarms
No alarms currently active

janitor@uranus>

No comments: