Restrictive rsync + ssh

Some of you have probably used rsync to backup files and directories from one machine to another. If one of those machines is in in an open network, you probably are doing it inside a ssh tunnel. If not, you should. And, it is really not that hard to do.

Let's say you wanted to copy directory called pickles inside the user bob's home directory at flyingmonkey.example.com, which is a Linux/Unix box out in the blue yonder. If you have rsync
installed (most Linux distros do come with it or offer it as a package), you could do something like:

rsync -az -e ssh bob@flyingmonkey.example.com:pickles /path/to/backup/dir/

The -e ssh is what tells rsync to do all of its monkeying about inside a ssh tunnel. And, when you run the above statement, it will then ask for bob's password and then proceed to copy the directory
~bob/pickles inside the directory /path/to/backup/dir. Which is great but I think we can do better.

Look Ma! No passwords!

First thing I want to get rid of is needing to enter a password. Yeah, it was great while we are testing it, but if we have a flyingmonkey loose in the internet, I would like to make it a bit harder for someone to break into it; I think I owe that to the Wicked Witch of the West.

The other reason is that then we can do the rsync dance automagically, using a script that is run whenever it feels like. In other words, backup. For this discussion we will just cover backup as in copying new stuff over old stuff; incremental backup is doable using rsync but will be the subject for another episode.

So, how are we going to do that? you may ask. Well, ssh allows you to authenticate using public/private key pairs. Before we continue, let's make sure sshd in flyingmonkey is configured to accept them:

bob@flyingmonkey:~$ grep -E 'PubkeyAuthentication|RSAAuthentication' /etc/ssh/sshd_config 
#RSAAuthentication yes
#PubkeyAuthentication yes
#RhostsRSAAuthentication no
bob@flyingmonkey:~$

Since PubkeyAuthentication and RSAAuthentication are set to yes, we are good to go. Now if flyingmonkey runs OSX, you would want to use /etc/ssh/sshd_config instead.

A quick note on ssh keys: they are very nice way to authenticate because they make life of whoever is trying to break into your machine rather hard. Now, just guessing the password does not do you much good; you need to have the key. And, to add insult to injury, you can have a passphrase in the key itself.

Enough digressing. The next step is to create the key pair. The tool I would use in Linux/Solaris/OSX is ssh-keygen because I like to do command line thingies. So, we go back to the host that will be rsnc'ing to flyingmonkey and create it by doing

ssh-keygen -b 4096 -t rsa -C backup-key -f ~/.ssh/flyingmonkey

which will create a 4096 bit (a lot of places still use 1024 and some now are announcing they have new state-of-the-art ultra secure settings of 2048 bits. So unless your server can't handle it, use 4096 or better) RSA key pair called flyingmonkey and flyingmonkey.pub in your .ssh directory:

raub@backup:~$ ls -lh .ssh/flyingmonkey*
-rw------- 1 raub raub 3.2K Dec 31 11:30 .ssh/flyingmonkey
-rw-r--r-- 1 raub raub  732 Dec 31 11:30 .ssh/flyingmonkey.pub
raub@backup:~$

During the creation process, it will ask for a passphrase. Since we are going to have a script using this keypair, it might not make sense to have a passphrase associated to it. Or it might, and there are ways to provide said passphrase to script in some secure way. But this post is getting long so I will stick to the easy basic stuff. If you remember, we said this is a public/private key authentication; that means it uses to keys: public and private. The public is taken to the machine you want to ssh into while the private stays, well, private. Let's look at the public key (it is a single line):

raub@backup:~$ cat .ssh/flyingmonkey.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACACsgpy/ihq31kv+Zji6Eknr46nbyx38uPE54X3STbaNC8oCheulVk
/+bTmrFy8Ne8RcTeWYd93wwabgBVDJYzjnsuwUgBO/JPXE4GiQrcnIz5fPsqJqslYxR5WnuUfkYPsAYgJL33XWZWi
dPu+A38OSxXf7UAfpKe5WXa93knXIERUA7NOzCKO96YzpW96i7LxAs20bsmNAw6bZbrZG8Dn3EssIK8CtEUvw4nWb
uFKZQS+b5AM8q20+IrGGVG193H6Rm3/iw0jip0VQOFozUB6yjToyZ5MTzShjb+f56o3+VUG2Bel7OMDfYXYYEKIoj
+cmTMLP5yu4v1t5dTkN3osneK/+2KHwXFTQY48TqxyqH+ZqEFy2X+kXoKff/89aD8lwj+uYKl6HNKhveKSZMNq/yc
7jCc05hLQCQkyC9/1lY9LI2UMHq2kqsgbdmR3uu3Oua2y1HhyeR9hqP9Om+kLu2K7cIXu5NBO9ro0vWBJII7T+z98
awbGH4jSryOxvAlpTT7d+POev13oOWonIwyTmkT72Q+/qJhPU/Vdtd7n5gSUomRT8dJQH+2hyA8c3+YPSW2VckBY/
Ax5aGX+AFoy1Y6WKpUWMIbwHJqizpdEd3WQWzivR1psfsjFzqrdG5SOSZFH2SHvzdNQOTz0FbYvgBV2Egq7WXv98C
se9ZDx backup-key
raub@backup:~$ 

You probably notices the backup-key string on the end of the file; we put it there using the -C (comment) option. Usually it writes the username@host, which is OK most of the times but I wanted something to remind me of what this key is supposed to do. You can change it later if you want.

So we go back to flyingmonkey and place the contents of the public key, flyingmonkey.pub in ~bob/.ssh/authorized_keys by whatever means you want. cut-n-paste works fine.

cat flyingmonkey.pub >> authorized_keys

also does a great job. Or you can even use ssh-copy-id if you feel frisky. Just remember the contents of flyingmonkey.pub is a single line. Of course, if flyingmonkey is a windows machine, you will do something else probably involving clicking on a few windows, but the principle is the same: get the bloody key into the account in the target host you want to connect to.

Once that is done, connect using ssh by providing the key

ssh -i .ssh/flyingmonkey bob@flyingmonkey.example.com

Can you login fine? Great; now try rsync

rsync -az -e 'ssh -i .ssh/flyingmonkey' bob@flyingmonkey.example.com:pickles /path/to/backup/dir/

Do not continue until the above works. Note in a real script the private key will probably be somewhere only the user which runs the backup script can access.

Limiting

So far so good. We eliminated the need to use a password so we can write a script to use the above. But, we can still ssh using that key to do other things besides just rsync. Time to finally get to the topic of this post.

If the IP/hostname of the host you are backing up flyingmonkey from does not change, you can begin by adding that to the front of the ~bob/.ssh/authorized_keys entry for the flyingmonkey public key. Now, if the backup server is in a private/NATed lan, you want to use the IP for its gateway. In this example, let's say we all all inside a private lan and the IP for backup server is 192.168.42.24:

from="192.168.42.24" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACACsgpy/ihq31kv+Zji6Eknr46nbyx38uPE54X3STbaNC8oCheulVk
/+bTmrFy8Ne8RcTeWYd93wwabgBVDJYzjnsuwUgBO/JPXE4GiQrcnIz5fPsqJqslYxR5WnuUfkYPsAYgJL33XWZWi
dPu+A38OSxXf7UAfpKe5WXa93knXIERUA7NOzCKO96YzpW96i7LxAs20bsmNAw6bZbrZG8Dn3EssIK8CtEUvw4nWb
uFKZQS+b5AM8q20+IrGGVG193H6Rm3/iw0jip0VQOFozUB6yjToyZ5MTzShjb+f56o3+VUG2Bel7OMDfYXYYEKIoj
+cmTMLP5yu4v1t5dTkN3osneK/+2KHwXFTQY48TqxyqH+ZqEFy2X+kXoKff/89aD8lwj+uYKl6HNKhveKSZMNq/yc
7jCc05hLQCQkyC9/1lY9LI2UMHq2kqsgbdmR3uu3Oua2y1HhyeR9hqP9Om+kLu2K7cIXu5NBO9ro0vWBJII7T+z98
awbGH4jSryOxvAlpTT7d+POev13oOWonIwyTmkT72Q+/qJhPU/Vdtd7n5gSUomRT8dJQH+2hyA8c3+YPSW2VckBY/
Ax5aGX+AFoy1Y6WKpUWMIbwHJqizpdEd3WQWzivR1psfsjFzqrdG5SOSZFH2SHvzdNQOTz0FbYvgBV2Egq7WXv98C
se9ZDx backup-key

This is a small improvement: only host that can connect is the one with this IP, be it legit or faking that. Test it.

Next step is specify which commands that can be run when connected using this key. And that one again will require playing with ~bob/.ssh/authorized_keys. This time we will specify the command:

from="192.168.42.24",command="/home/bob/.ssh/validate-rsync" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACACsgpy/ihq31kv+Zji6Eknr46nbyx38uPE54X3STbaNC8oCheulVk
[...]
se9ZDx backup-key

And define validate-rsync as

cat > .ssh/validate-rsync << 'EOF'
#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
rsync\ --server\ --sender\ -vlogDtprze.iLsf\ .\ pickles)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac
EOF
chmod +x .ssh/validate-rsync

And this is where it get really exciting. All that validate-rsync is doing is seeing if the command being sent is not only an rsync command but a specific one. Once we figure out how to get the proper SSH_ORIGINAL_COMMAND, we can change the line

rsync\ --server\ --sender\ -vlogDtprze.iLsf\ .\ pickles)

to what it needs to be to match our backup script and test. Note that if you change the rsync statement, you will need to change the case.

Getting the SSH_ORIGINAL_COMMAND

Let's say you want to have an account you can ssh into but only run very specific commands in it. A good way to achieve that is to write a wrapper script that is called from your authorized_keys file. So you could have a wrapper that looks like this:

#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
    "/usr/bin/rsync "*)
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Permission denied."
        exit 1
        ;;
esac

But, what if you really want to be really precise on the command? Using the above example, not only running rsync but also specifying the path and the arguments? You could cheat and find what the command you are sending is supposed to look like by replacing (temporarily) your wrapper script with this

/bin/sh

DEBUG="logger" # Linux
#DEBUG="syslog -s -l note" # OSX

if [ -n "$SSH_ORIGINAL_COMMAND" ]; then
        $DEBUG "Passed SSH command $SSH_ORIGINAL_COMMAND"
elif [ -n "$SSH2_ORIGINAL_COMMAND" ]; then
        $DEBUG "Passed SSH2 command $SSH2_ORIGINAL_COMMAND"
else
        $DEBUG Not passed a command.
fi

Then you run the ssh command and see what it looks like in the log file. Copy that to your original wrapper script, and you are good to go. So

ssh -t -i /home/raub/.ssh/le_key raub@virtualpork echo "Hey"

Results in

Dec 26 13:34:05 virtualpork syslog[64541]: Passed SSH command echo Hey

While

rsync -avz -e 'ssh -i /home/raub/.ssh/le_key' raub@virtualpork:Public /tmp/backup/

results in

Dec 26 13:28:17 virtualpork syslog[64541]: Passed SSH command rsync --server 
--sender -vlogDtprze.iLs . Public

The latter meaning our little wrapper script would then look like

#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
    "rsync --server --sender -vlogDtprze.iLs . Public")
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Permission denied."
        exit 1
        ;;
esac

Adding a disk to a libvirt/kvm vm client

So you have a kvm virtualization infrastructure which you, being lazy like me, manage using libvirt. You then created a vm client with the virtual disk partitioned just right (using LVM or not; pick your poison). But later you realized you needed another disk. Maybe it is because you need to have data encrypted to meet HIPAA or PCI requirements. Maybe you just want to keep your data in a different drive. The point is you need another drive and don't want to/should not resize the current virtual disk associated with the VM.
So, first thing you do is create the new disk. If you are using KVM, chances are you have been using qcow2 disks. Or maybe vmdk, iSCSI, or, like me, lvm. No matter what, you probably know how to create a new disk, so I will assumed you took time to figure out how large it needs to be to fit your needs and created the little bastard. Because, as I mentioned before, I am lazy, I will say we are running libvirt in linux, with a vm client called vmclient and creating LVs to use as virtual disks. So, we need a 10GB virtual disk that we'll call data because we are friends with Captain Obvious.

lvcreate -L 10G -n data vmhost_vg0

creates as we know a 10GB lv as /dev/vmhost_vg0/data. To make it easier on us, we will shut down the vm client. Once that is done (do check it using virsh list --all, will you?), we then run

virsh edit vmclient

Note we could have created a properly configured xml file and fed into the config, but I do forget how to make it properly configured so I prefer to cheat.
When the config file is open, look for the disk entries; they should look like this:

    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/vmclient_boot"/>
      <target bus="virtio" dev="vda"/>
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci"/>
    </disk>

which is how our original virtual disk is configured to be used in this vm client. Of course, if you were using qcow, vmdk, or something else, the entry might look a bit different; make a note of how it differs from my example and move on. This is why I said I like to cheat: I can see how the old disk was defined and copy that instead of trying to figure out how to do it.

Now you need to add the new drive. As you guessed from the above, we will do it by copying the above entry and changing it a bit. Now, we do not need to copy everything; we just need enough so virsh knows what we want. It will fill the blanks. So, after we copy the relevant bits below the already defined disk and change the drive name, we would have something like this

    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/data"/>
      <target bus="virtio" dev="vda"/>
    </disk>

Save it; it should close without issues. If not, got back and see if you missed something. If it did not bark, use virsh dump xml vmclient to see your handiwork. Mine looks like this:

    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/vmclient_boot"/>
      <target bus="virtio" dev="vda"/>
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci"/>
    </disk>
    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/data"/>
      <target bus="virtio" dev="vda"/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/%gt
    </disk>

As you can see, it added the pci bus address entry on its own. Now all we need to do is reboot the vmclient, format the new disk on whatever way we want, and start using it.