Replacing a VMWare ESXi host with a KVM one

Why?

Ok, I think I did my due diligence. With all the entries I have in this blog as proof, I think I put up long enough with the ESXi box. It was not as bad as Xen but I got tired of not being able to get it to behave as it should. And when I could not do PCI passthrough -- I am not even saying the Netronome card, but every single PCI or PCIe card I had available and had no problems passing to a vm guest using KVM as the hypervisor -- it was time to move on. The writing on the wall came after almost a year I could not get an answer from VMWare.

The Plan

1. While the ESXi server, vmhost2, is still running, export the guests in .ovf format to a safe location. Just to be on the safe side, write down somewhere the specs for each guest (memory, cpu, OS, which network it is using, etc).
   
2. Build the new vmhost2 using Debian or CentOS as the base OS and kvm as the hypervisor. Some things to watch out for:
   
   • Setup Network trunk and bridges to replicate old setup.
     
   • Use the same IP as before since we are keeping the same hostname.
     
   • Setup the logical volume manager so I can move things around later on.
     
   • Configure ntp to use our internal server
     
   • Configure DNS to use our internal server
     
   • Accounts of users who need to access the vm host itself will be mounted through autofs. If that fails, can login to root using ssh keypair authentication. If that is down (say, network issues, switch to console).
     
   • Like in the old vmhost2, ISOs for the install images are available through NSF.
     
   • Add whatever kernel options we might need. Remember we are building this from scratch, not just dropping a prebuilt system like xenserver. Or ESXi.
     
3. Import enough vm guests to validate the system. Might take the opportunity to do some guest housecleaning.
   
4. Add any PCI/PCIe cards we want to passthrough.
   
5. Import the rest of the vm guests.
   
6. (Future:) set it up so it can move/load balance vm guests with vmhost, the other KVM host.
   

Note: I did not wipe the original hard drive; instead I just bought a 111.8GB (128GB in fake numbers) SSD to run the OS in. I did not get a second drive and make it a RAID for now since I plan on running the OS in that disk, which will be configured using Ansible so I can rebuilt it quickly. Any vm guest running in this vm host will either run from the fileserver (iSCSI) or in a local RAID setup of sorts. Or I might simply deploy ZFS and be done with it. With that said, I might run a few vm gusts from that drive to validate the system.

Note: This project will be broken down into many articles otherwise it will be long and boring to read on a single sitting (some of the steps are applicable to other applications besides going from ESXi to KVM). I will try to come back and add the links of those articles, treating this post as the index.

Failover and swapping controllers in a Equallogic SAN, command line involved

Equallogic as some of you know are old storage systems sold by Dell that can do at best RAID6 in a good day. As of now it has been out of support for a few years; Dell bought EMC and would rather if we used that instead. We do have them too but this article is about the Equallogic we still use.

On a Tuesday evening I got an email from one of those telling me it was unhappy. As my recurring motif, I do not like GUIs to manage devices. This one decided it would not reinforce that belief and instead just refused to start. So it is time for plan B. Most of the storage appliances and hypervisors out there either are built on Linux or Freebsd. In the last time I had to probulate it, I found it runs freebsd. And that usually means we can ssh into it. And we do it; I think I do not need to show how to do that; if you ssh into a machine, you ssh into them all.

Now we are inside we need to find what's up (I am cheating a bit because the email did tell me that the device PS4100E-01 was unhappy, which is why I am probulating it specifically):

STORGroup01> member select PS4100E-01 show
_____________________________ Member Information ______________________________
Name: PS4100E-01                       Status: online
TotalSpace: 15.98TB                    UsedSpace: 3.86TB
SnapSpace: 192.92GB                    Description:
Def-Gateway: 192.168.1.1               Serial-Number:
Disks: 12                                CN-TWO_AND_A_HALF
Spares: 1                              Controllers: 2
CacheMode: write-thru                  Connections: 4
RaidStatus: ok                         RaidPercentage: 0.000%
LostBlocks: false                      HealthStatus: critical
LocateMember: disable                  Controller-Safe: disabled
Version: V9.1.4 (R443182)              Delay-Data-Move: disable
ChassisType: DELLSBB2u12 3.5           Accelerated RAID Capable: no
Pool: default                          Raid-policy: raid6
Service Tag: H9CHSW1                   Product Family: PS4100
All-Disks-SED: no                      SectorSize: 512
Language-Kit-Version: de, es, fr, ja,  ExpandedSnapDataSize: N/A
  ko, zh                               CompressedSnapDataSize: N/A
CompressionSavings: N/A                Data-Reduction: no-capable-hardware
Raid-Rebuild-Delay-State: disabled     Raid-Expansion-Status: enabled
_______________________________________________________________________________

____________________________ Health Status Details ____________________________

Critical conditions::
Critical hardware component failure.

Warning conditions::
None
_______________________________________________________________________________


____________________________ Operations InProgress ____________________________


ID StartTime            Progress Operation Details                             

-- -------------------- -------- -----------------------------------------------
STORGroup01>

Yep, the Health Status Details menu tells us it is unhappy; we knew that already. But, what does the log say?

STORGroup01> show recentevents
[...]
6492:881:PS4100E-01:SP: 8-Jan-2019 19:34:51.700834:cache_driver.cc:1056:WARNING:
28.3.17:Active control module cache is now in write-through mode. Array performa
nce is degraded.

6491:880:PS4100E-01:SP: 8-Jan-2019 19:34:51.700833:emm.c:355:ERROR:28.4.85:Criti
cal hardware component failure, as shown next.
        C2F power module is not operating.

6490:879:PS4100E-01:SP: 8-Jan-2019 19:34:51.700832:emm.c:2363:ERROR:28.4.47:Crit
ical health conditions exist.
 Correct immediately before they affect array operation.
        Critical hardware component failure.
        There are 1 outstanding health conditions. Correct these conditions before they
 affect array operation.

OK, controller thingie is not happy. But, this device has 2 of them. Which one is it?

STORGroup01> member select PS4100E-01 show controllers
___________________________ Controller Information ____________________________
SlotID: 0                              Status: active
Model: 70-0476(TYPE 12)                BatteryStatus: failed
ProcessorTemperature: 65               ChipsetTemperature: 44
LastBootTime: 2018-04-23:15:06:55      SerialNumber:
Manufactured: 0327                       CN-I_AM_FEELING_DEPRESSED
ECOLevel: C00                          CM Rev.: A04
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011                             
_______________________________________________________________________________
_______________________________________________________________________________
SlotID: 1                              Status: secondary
Model: 70-0476(TYPE 12)                BatteryStatus: ok
ProcessorTemperature: 0                ChipsetTemperature: 0
LastBootTime: 2018-04-23:15:13:17      SerialNumber:
Manufactured: 031S                       CN-I_AM_FEELING_GREAT
ECOLevel: C00                          CM Rev.: A04
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011                             
_______________________________________________________________________________

______________________________ Cache Information ______________________________
CacheMode: write-thru                  Controller-Safe: disabled
Low-Battery-Safe: enabled                                                      
_______________________________________________________________________________
STORGroup01>

Fun fact: I haven't the foggiest idea of which slot is 1 and which one is 0. We will have to find that out as we go along.

SPOILER ALERT: Making ASSumptions here is a bad idea.

The Controller

We sent out the above info -- model, firmware -- to the vendor we have a support contract with, which sent a controller card. Here is it in its purple gloriousness.

The configuration is stored in a SD card on the socket being pointed by my finger. When swapping the controller, the laziest thing to do is put the old SD card in the new controller. This way we do not have to configure it.

We know one of the controllers is unhappy, but which one? You see, this storage device only needs one controller to work. But, it is an enterprise device; the reason to have two is that the second one is on standby: if the primary has a problem, service can failover to the secondary/backup. If you look at the picture below, you will see one controller has both lights green while the one below is has the top green and the bottom orange. That (orange light) indicates it is the either the backup, secondary, failover, or unused controller. The OR is very important here

Now, this system is designed to be hot swappable but with one proviso: you can swap the device that is not currently in use, which would make sense since chances are the bad controller failed and the backup took over. So, the failed device should be the one in standby.

So, I swapped it. And then checked again.

STORGroup01> member select PS4100E-01 show controllers
___________________________ Controller Information ____________________________
SlotID: 0                              Status: active
Model: 70-0476(TYPE 12)                BatteryStatus: failed
ProcessorTemperature: 64               ChipsetTemperature: 44
LastBootTime: 2018-04-23:15:07:14      SerialNumber:
Manufactured: 0327                       CN-I_AM_FEELING_DEPRESSED
ECOLevel: C00                          CM Rev.: A04
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011
_______________________________________________________________________________
_______________________________________________________________________________
SlotID: 1                              Status: secondary
Model: 70-0476(TYPE 12)                BatteryStatus: ok
ProcessorTemperature: 0                ChipsetTemperature: 0
LastBootTime: 2019-01-15:12:56:01      SerialNumber:
Manufactured: 025E                       CN-I_AM_THE_NEW_GUY
ECOLevel: C00                          CM Rev.: A03
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011
_______________________________________________________________________________

It seems I replaced the secondary (serial number CN-I_AM_THE_NEW_GUY), and the primary one (CN-I_AM_FEELING_DEPRESSED) is still the problematic one. Why didn't it failover when it realized it had a problem? I don't know; what I know is that I still need to replace the problematic controller. At least now we know the SlotID 0 is the top controller and SlotID 1 the bottom. Small progress but progress nevertheless.

It turns out there is a way to programmatically force the failover; that is achieved using the command restart. But, since I have 3 different storage devices -- PS4100E-01, PS4100E-00, and PS6100E -- in this setup and am currently accessing them from the system that control all of them, and restart does not seem to allow me to specify which device I want to reboot,

STORGroup01> restart
            - Optional argument to restart.
 
STORGroup01>

we should ssh into PS4100E-01 (IP 192.168.1.3 for those who are wondering) and then issue the command there. We do not need to worry about it restarting the secondary controller because it only reboots the active one, which causes it to failover to the secondary.

STORGroup01> restart

Restarting the system will result in the active and secondary control
modules switching roles. Therefore, the current active control module
will become the secondary after the system restart.


After you enter the restart command, the active control module will fail over.
To continue to use the serial connection when the array restarts, connect the
serial cable to the new active control module.

Do you really want to restart the system? (yes/no) [no]:yes
Restarting at Tue Jan 15 13:40:48 EST 2019 -- please wait...
Waiting for the secondary to synchronize with us (max 900s)
....Rebooting the active controller
Connection to 192.168.1.3 closed by remote host.
Connection to 192.168.1.3 closed.
raub@desktop:~$

And, yes, it is that anticlimatic. I just got kicked off PS4100E-01; our network access to the storage device is through the controller. Now we need to wait for it to come back. I am lazy so I will use ping to let me know when the network interface is back up.

raub@desktop:~$ ping 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.

64 bytes from 192.168.1.3: icmp_seq=10 ttl=254 time=3.50 ms
64 bytes from 192.168.1.3: icmp_seq=10 ttl=254 time=3.51 ms (DUP!)
64 bytes from 192.168.1.3: icmp_seq=11 ttl=254 time=90.8 ms
64 bytes from 192.168.1.3: icmp_seq=12 ttl=254 time=1.14 ms
64 bytes from 192.168.1.3: icmp_seq=13 ttl=254 time=1.49 ms
64 bytes from 192.168.1.3: icmp_seq=14 ttl=254 time=1.64 ms
64 bytes from 192.168.1.3: icmp_seq=15 ttl=254 time=1.06 ms
^C
--- 192.168.1.3 ping statistics ---
15 packets transmitted, 6 received, +1 duplicates, 60% packet loss, time 14078m
s
rtt min/avg/max/mdev = 1.068/14.741/90.815/31.072 ms
raub@desktop:~$

What I can't show in this article is that it took a while until I started getting packets back. In fact, I went to make coffee (which required me to find coffee first) and then came back and then run ping a few times until I got the above output. These devices are in no hurry when rebooting.

Once we see pinging, we can check if ssh is running again and then log back in and asked what's up with the controllers:

STORGroup01> member select PS4100E-01 show controllers
___________________________ Controller Information ____________________________
SlotID: 0                              Status: unknown
Model: ()                              BatteryStatus: unknown
ProcessorTemperature: 0                ChipsetTemperature: 0
LastBootTime: 2019-01-15:13:43:19      SerialNumber:
Manufactured:                          ECOLevel:
CM Rev.:                               FW Rev.:
BootRomVersion:                        BootRomBuilDate:
_______________________________________________________________________________
_______________________________________________________________________________
SlotID: 1                              Status: active
Model: 70-0476(TYPE 12)                BatteryStatus: ok
ProcessorTemperature: 60               ChipsetTemperature: 43
LastBootTime: 2019-01-15:12:56:00      SerialNumber:
Manufactured: 025E                       CN-I_AM_THE_NEW_GUY
ECOLevel: C00                          CM Rev.: A03
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011
_______________________________________________________________________________

When the output was captured, the controller in slot 0 has not come back up from the restart command. What matters, however, is the controller in slot 1 became the active one, meaning we can swap the unhappy controller. If we wait a bit, we will see the controller on slot 0 be reported as failed,

STORGroup01> member select PS4100E-01 show controllers
___________________________ Controller Information ____________________________
SlotID: 0                              Status: secondary
Model: 70-0476(TYPE 12)                BatteryStatus: failed
ProcessorTemperature: 0                ChipsetTemperature: 0
LastBootTime: 2019-01-15:13:43:09      SerialNumber:
Manufactured: 0327                       CN-I_AM_FEELING_DEPRESSED
ECOLevel: C00                          CM Rev.: A04
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011
_______________________________________________________________________________

which means we can now swap the top guy (slot 0) with the old controller we accidentally removed from slot 1. After a few minutes, all is well in the world:

STORGroup01> member select PS4100E-01 show controllers
___________________________ Controller Information ____________________________
SlotID: 0                              Status: secondary
Model: 70-0476(TYPE 12)                BatteryStatus: ok
ProcessorTemperature: 0                ChipsetTemperature: 0
LastBootTime: 2019-01-15:13:55:33      SerialNumber:
Manufactured: 031S                       CN-I_AM_FEELING_GREAT
ECOLevel: C00                          CM Rev.: A04
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011
_______________________________________________________________________________
_______________________________________________________________________________
SlotID: 1                              Status: active
Model: 70-0476(TYPE 12)                BatteryStatus: ok
ProcessorTemperature: 62               ChipsetTemperature: 42
LastBootTime: 2019-01-15:12:55:43      SerialNumber:
Manufactured: 025E                       CN-I_AM_THE_NEW_GUY
ECOLevel: C00                          CM Rev.: A03
FW Rev.: Storage Array Firmware V9.1.4 BootRomVersion: 3.6.4
   (R443182)                           BootRomBuilDate: Mon Jun 27 10:20:45
                                          EDT 2011
_______________________________________________________________________________

There might be a GUI way to do all of that but I am not smart enough to click on things.

Adding (local) storage in XenServer

Lets start by getting something straight: This is not something that is normally done in an environment that has a cluster of XenServer servers. In those, you are probably using some kind of external storage line a SAN which feeds LUNs to the servers. But, there are those who did not setup their clusters like that for whatever reason -- cost, space, resources -- and as a result built their xen servers with local storage for the vm clients. Or maybe they just have one single xen server. For those this might be a bit useful.

Note: As usual most of this to be done in command line; if you have read my previous articles you would expect nothing less from me. I am just not that original; just deal with it already.

For this example, let's say you bought a server with lots of drive bays (say, 15), a nice raid controller that is supported by XenServer. And you got it running by creating, say, a RAID1 array using two small drives to install the OS. And they happen to have enough free disk space to build a few test vm clients. This server shall henceforth be called vmhost3.

Note: If you had built a Linux box and then installed Xen on the top of it, like RedHat used to recommend before they switched to KVM, this is much easier because you can use the usual LVM commands. Since we are using XenServer things are conceptually the same but the implementation slightly different.

So you built a few test vm clients and verified all is nice and peachy. I mean, even your VLAN trunks work properly. So, now it is time to go into production.

1. You used 2 drive bays to put the OS on; you still have 13 left to play. So, you grab, say, 6 SSDs and make a nice RAID6 out of it. I will not go in details here because it is out of scope of this discussion, but I can write another article showing how to create the new RAID from command line in xenserver. Let's just assume you did it and it works.
   
2. Xenserver by default creates local storage for vm clients by using LVM: it allocates a logical volume (lv) as the drive(s) for those clients, which then format them to whatever filesystem they want. So, we need to grab the drive created by the RAID6 and then configure it to do some LVMing. Every journey starts with the first step, and in this case that is to take a quick look at the drive we created when we did the raid; I will save you some time and tell you it is called /dev/sdc:

   [root@vmhost3 ~]# fdisk -l /dev/sdc
   
   Disk /dev/sdc: 1999.8 GB, 1999844147200 bytes
   255 heads, 63 sectors/track, 243133 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   
   Disk /dev/sdc doesn't contain a valid partition table
   [root@vmhost3 ~]#

   Yes, our drive is not that large, but it will be good enough for this example.

3. Before we start passing LVs out like farts we need a lvm physical volume(pv). We could create a single partition in sdc that is as big as the disk, or just make the entire drive as the pv. I will do the latter.
   

   Do you remember my warning about XenServer vs Linux+Xen? This is where things diverge a bit. In the latter we would do something like

   pvcreate /dev/sdc
   vgcreate data /dev/sdc

   To create the pv and a volume group (vg) called data, but since this is XenServer, we should use their command line thingie, xe:

   [root@vmhost3 ~]# xe sr-create name-label=data shared=false \
   device-config:device=/dev/sdc type=lvm
   f70d9ff8-3567-ef23-42c2-7c7997a4abc6
   [root@vmhost3 ~]#

   which does the same thing, but tells xenserver that it has a new local storage device:
   

4. From here on, it is business as usual./

Entrypoint scripts and data persistency: mounting a NFS fileshare inside a docker container

This is a quick article; you have been warned.

The traditional way to have persistent data in a docker container is to feed it with a volume from the docker host. You begin by locating a directory, be it in the local docker host drive, network mounted to the docker host, or a volume in another docker container. Then feed it to the container using the VOLUME statement defined in the Dockerfile or to command line. We all know that. But what some (very few we hope) of you might not have been aware of is the volume is only mounted/attached to the container when you tell docker to run the container, which might make running/configuring something that require files in those volumes a bit challenging.

At this point I would expect one of you to shout "Aha! But you can then use an entrypoint script to do all that monkeying that needs to happen after volume is mounted and before whatever service this container provides starts!" And you would be quite right! Here's a quick example in case we lost somebody: if our Dockerfile ended up like this:

# Put the entrypoint script somewhere we can find
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

EXPOSE 22
# Start service
CMD ["/usr/sbin/sshd", "-D"]

we would have created a file called entrypoint.sh in the root path of the container, which is run just after the volume is created and before the, in this example, sshd service is started. Said entrypoint file could do, I don't know:

#!/bin/sh
set -e

# Do something in /home, which is the mounted volume
/do-something-at-home.sh

# And we  are done here
exec "$@"

point is that do-something-at-home.sh is called only after the persistent volume is mounted into the container.

What if you want to mount a network fileshare straight into the container? Before I have time to answer that, someone from the audience will jump and state "Why would you want to do that? You can mount the network fileshare in the docker host and then use VOLUME just like Wicked Witch of the West intended!" What if I don't want to do that? What if I can't for whatever reason mount a network fileshare on the docker host?

The answer is that it is completely doable; you just have to ask the entrypoint script to do the mounting for you. To show what I mean I will use the example of mounting a NFSv4 fileshare that is supposed to be used by some website. So we modify our little entrypoint script a bit

#!/bin/sh
set -e

# /export/www was created in the Dockerfile
mount.nfs4 fileserver.example.com:/export/www /www/some-website 

# And we are done here
exec "$@"

Since we are using NFSv4, chances are you might need to add something like

RUN sed -i -e '/^#Domain/a Domain = example.com' /etc/idmapd.conf

to your dockerfile in addition to telling it to get the nfs client packages, but the general idea should apply for, say, SMB or ZFS or whatever other network filesystem that you fancy: let your entrypoint script do the heavy lifting!

Resizing a shared partition in a Synology DiskStation

I bought one of those devices, specifically the DS212j, to use as network storage (NAS for you alphabet soup lovers) for my home. I slapped two green 2TB Western Digital drives in it, set them up as a raid 1, created a 100GB (which probably means using the fake gigabyte, not the power-of-two one) NFS share partition for users, and off I went. Now, since I used its default clickety-click interface (it's pronounced web-based), when I ssh into the device (I am a bit of a command-line (CLI; I did not forget you) kinda bloke), I found it is using the standard Linux lvm (ok, this one I actually use) and named the logical volume I created as volume1, formatted as ext4 mounted as /volume_1. Even though I personally like to call my volumes after their function and try to avoid mounting stuff on the root, I can live with that. But, the point is ext4 and lvm involved. i.e. sane stuff. I like that. It also means that even though a lot of those devices use a scaled down version of linux, this one is not as scaled down as you would be led to believe.

This morning I received a email from the device. Since I want to make this post look long and important, I will post it here in glorious quadrovision:

Dear user,

The available space of volume 1 on spindizzy is running out; please delete some files to free space.

Total capacity: 98.43 GB
Available capacity: 0.98 GB (1.00%)

Sincerely,
Synology DiskStation

Hmmm, that sounds kinda bad. What should I do? Well, I am lazy. Do you remember when I mentioned the sane stuff Synology is using in this device? Let's do some exploring since I still need to fill more space:

spindizzy> pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/md2   vg1  lvm2 a-   1.81T 1.72T
spindizzy> vgs
  VG   #PV #LV #SN Attr   VSize VFree
  vg1    1   2   0 wz--n- 1.81T 1.72T
spindizzy> lvs
  LV                    VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  syno_vg_reserved_area vg1  -wi-a-  12.00M                                      
  volume_1              vg1  -wi-ao 100.00G                                      
spindizzy> 

So, the entire raid (minus whatever the device needs to do its thing) is a single physical volume which is allocated as a single volume group (cleverly called vg1, inside which is our logical volume. And, as the email said and df -h can show,

/dev/vg1/volume_1        98.4G     97.4G    916.5M  99% /volume1

rather full. Well, how about if we take care of that lvm-style?

spindizzy> lvextend -L +100G /dev/vg1/volume1
  Logical volume volume1 not found in volume group vg1
spindizzy> lvextend -L +100G /dev/vg1/volume_1
  Extending logical volume volume_1 to 200.00 GB
  Logical volume volume_1 successfully resized
spindizzy> resize2fs /dev/vg1/volume_1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg1/volume_1 is mounted on /volume1; on-line resizing required
old desc_blocks = 7, new_desc_blocks = 13
Performing an on-line resize of /dev/vg1/volume_1 to 52428800 (4k) blocks.
The filesystem on /dev/vg1/volume_1 is now 52428800 blocks long.

spindizzy> df -h              
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  2.3G    425.6M      1.8G  19% /
/tmp                    121.8M    264.0K    121.5M   0% /tmp
/dev/vg1/volume_1       196.9G     97.4G     99.2G  50% /volume1
spindizzy> 

What I did was to add an extra 100G, effectively doubling its size, to the logical volume volume_1. And all that was done live. Exciting huh? For those of you who do not dabble with lvm a lot, one of its nicest features is that you can increase the size of a logical volume life, without needing to unmount it first. All you need is to have some free space in the volume group (the VFree column). Going the other way around is a bit more challenging, for you need to umount the volume first, but can be done. I will later write an article on monkeying with lvm, I promise (remind me!).

Some of you might be like big deal, you could probably have done that using the web interface, just like in many other equivalent devices. What's so special about the a Linux-based network storage thingie? Well, the fact they can (either from factory or by adding the required packages) use lvm means I do not need to recreate a partition whenever I need more space, which was a problem with other devices I had. And, I can not only take care of that through the command line instead of needing a web browser, but also I could write a script to do that for me. Are they the only ones doing it? I doubt, but it reminds me why when shopping for a NAS I look for one that runs Linux in some shape or form.