Creating a RSA key for rsync for Android using docker

I use Rsync4Android to backup my phone. It is convenient and you can even run a cronjob so it does its thing without your intervention (think of it backing up while you are having dinner somewhere) and it backs up to wherever you want, so you are in control. Now, one day it stopped working. Not knowing what was going on, I did what one usually does when dealing with ssh issues: run sshd in debug mode:

raub@desktop:~$ sudo /usr/sbin/sshd -D
[...]
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: inetd sockets after dupping: 3, 3
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user raub service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "raub"
debug1: PAM: setting PAM_RHOST to "10.0.0.129"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user raub service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]
Connection closed by 10.0.0.129 port 39739 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 10239
debug1: audit_event: unhandled event 12
raub@desktop:~$

The line that tells what is going on is

userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]

When creating a key pair, Rsync4Android uses DSA algorithm. As we know, DSA has been considered insecure for a while and the current releases of openssh do not support it by default. So, if I want to keep on using rsync4android, I either configure my ssh server to accept DSA keys or find a way to convince it to use a RSA key. I chose the RSA route, but how to do it?

Create Key

Nothing special here.

ssh-keygen -t rsa -b 4096 -C "Phone_backup" -f ~/.ssh/phonebackup

Don't like 4096 bits? Double it; Rsync4Android does not care.

Convert private key to something Rsync4Android can use

The public key will go to the Linux host, which runs openssh and can handle those keys just fine. Rsync4Android, on the other hand, needs dropbear style keys. As mentioned in Rsync4Android docs, the best way to convert is to use the dropbearconvert command, which for ubuntu comes in the dropbear package. As I did not want to install it in desktop, I quickly created a docker container, copied the private key, and then installed the package. And then ran it (note the path) telling I am feeding it an openssh format key and want a dropbear style key:

root@b3f7ed8c4f24:/home# apt-get install dropbear
[...]
root@b3f7ed8c4f24:/home# /usr/lib/dropbear/dropbearconvert openssh dropbear phonebackup phonebackup.dropbear
Key is a ssh-rsa key
Wrote key to 'phonebackup.dropbear'
root@b3f7ed8c4f24:/home#

Now I have the dropbear-formated private key, phonebackup.dropbear, I can finally set the circus up.

Copy public and private keys to the proper locations

Public key goes to the computer we are backing the phone to (in my case desktop): added to the account's ~/.ssh/athorized_keys file.

The private key to the android. I put it in the same directory Rsync4Android placed the original (DSA) key it created, /sdcard. Then it was a matter of renaming the key in the Rsync4Android config and running it again.

Test it

Rsync4Android has a dry run mode so you can see if it works. When testing, I also ran sshd in debug mode. Then I ran the backup in "production mode"; to the left is a screen capture of my phone. The reason you see a lot of output is because I have --partial --progress as rsync options; you should configure it to fit your needs. I suggest to also check on the --exclude option; I think you will find it quite useful.

Now we know it works, we can worry about running it as a cron job (the Rsync4Android docs have a link for how to do that) and then make it work from an external network. But that is for another article.

Ubuntu Docker container with multiple architectures

Here's yet another quickie: let's say you created a docker Ubuntu container to do some sort of cross compilation. You know, maybe your container is 64bit Intel and you need to spit out some static 32bit Intel stuff. Or PPC for that matter. Now your development environment needs the 32bit version of libncurses-dev, which for ubuntu 14.04 would be libncurses5-dev:i386. We do need to specify the architecture (:i386) if we are not using the default one..

So you scribble up a quick Dockerfile:

############################################################
# Dockerfile to build a ubuntu container image thingie
# Based on Ubuntu (Duh!)
############################################################

# Set the base image to Ubuntu
FROM ubuntu:14.04

################## BEGIN INSTALLATION ######################

RUN apt-get update && apt-get -y upgrade 

RUN apt-get install -y build-essential curl libc6-dev libncurses5-dev:i386

##################### INSTALLATION END #####################

And then you do some building:

ducker@boot2docker:~/docker/test$ docker build -t raubvogel/test.
Sending build context to Docker daemon 4.096 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
 ---> 6d4946999d4f
Step 1 : RUN apt-get update && apt-get -y upgrade
 ---> Running in 2d035053a431
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
[...]
Processing triggers for ureadahead (0.100.0-16) ...
Setting up initscripts (2.88dsf-41ubuntu6.2) ...
guest environment detected: Linking /run/shm to /dev/shm
 ---> c1434c42218e
Removing intermediate container 2d035053a431
Step 2 : RUN apt-get install -y build-essential zip bzr curl libc6-dev libncurses5-dev:i386
 ---> Running in 2ac39f9430c2
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libncurses5-dev
INFO[0017] The command [/bin/sh -c apt-get install -y build-essential zip bzr curl libc6-dev libncurses5-dev:i386] returned a non-zero code: 100
ducker@boot2docker:~/docker/test$

And then it barks at the libncurses5-dev entry. What's going on?

The Debian Multiarch HOWTO seems to have a bit of a clue. I was going to write a long winded explanation but I am bored. So here is the short version:

dpkg --add-architecture i386
apt-get update
apt-get install libncurses5-dev:i386

See the dpkg --add-architecture i386 line? It tells the host that we are can do 32bit Intell thingies. The next line is there just to feed us with the 32bit repository data, so when we look for a 32bit package we can get it. Let apply that to our Dockerfile:

############################################################
# Dockerfile to build a ubuntu container image thingie
# Based on Ubuntu (Duh!)
############################################################

# Set the base image to Ubuntu
FROM ubuntu:14.04

################## BEGIN INSTALLATION ######################

# We need i386 crap
RUN dpkg --add-architecture i386
# Business as usual
RUN apt-get update && apt-get -y upgrade 

RUN apt-get install -y build-essential curl libc6-dev libncurses5-dev:i386

##################### INSTALLATION END #####################

And build it again.

ducker@boot2docker:~/docker/test$ docker build -t raubvogel/test.
Sending build context to Docker daemon 4.608 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
 ---> 6d4946999d4f
Step 1 : RUN dpkg --add-architecture i386
 ---> Running in 89382c3c3469
 ---> f367d3357fc4
Removing intermediate container 89382c3c3469
Step 2 : RUN apt-get update && apt-get -y upgrade
 ---> Running in 6d0fc9519029
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
[...]
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 build-essential : Depends: gcc (>= 4:4.4.3) but it is not going to be installed
                   Depends: g++ (>= 4:4.4.3) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
INFO[0002] The command [/bin/sh -c apt-get install -y build-essential  zip bzr curl libc6-dev libncurses5-dev:i386] returned a non-zero code: 100
ducker@boot2docker:~/docker/test$

Hmmm, that didn't work. Crap. Let's massage Dockerfile a bit:

############################################################
# Dockerfile to build a ubuntu container image thingie
# Based on Ubuntu (Duh!)
############################################################

# Set the base image to Ubuntu
FROM ubuntu:14.04

################## BEGIN INSTALLATION ######################

# We need i386 crap
RUN dpkg --add-architecture i386
# Business as usual
RUN apt-get update && apt-get -y upgrade

RUN apt-get install -y build-essential  &&\
    apt-get install -y zip bzr curl libc6-dev libncurses5-dev:i386

##################### INSTALLATION END #####################

And this time it works fine:

ducker@boot2docker:~/docker/test$ docker build -t raubvogel/test.
Sending build context to Docker daemon 4.608 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
[...]
Setting up libpam-systemd:amd64 (204-5ubuntu20.12) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
invoke-rc.d: unknown initscript, /etc/init.d/systemd-logind not found.
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
Processing triggers for ca-certificates (20141019ubuntu0.14.04.1) ...
Updating certificates in /etc/ssl/certs... 173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
 ---> d86a5113c577
Removing intermediate container 32e0cc9a0b5a
Successfully built d86a5113c577
ducker@boot2docker:~/docker/test$ 

What I found out is that it does not like to have the build-essentials in the same install statement as libncurses5-dev:i386. The easiest solution is to have build-essentials on its own install statement and everyone else in the next one. And life is once again well.

using curl and awk to get the latest android sdk

Here is another of those short posts for me to remember how I did something. So I want to retrieve the latest version of the Android SDK. The reason is not relevant to this article, but since I need something to fill it up and make it sound more important than it is, I will tell you: I want to build a docker container to build Android packages and want to make sure the Dockerfile will grab the latest SDK.

As some of you know, the SDK is available at https://developer.android.com/sdk/index.htm. Since I want to automate this, I would love to be able to have the url for the latest release without having to know which one is that. Asterisk, for instance, allows you to grab the -latest one which would as the name says get the latest asterisk file without any thinking required. If you look at the Android SDK page, you will know what is the latest version number and can get it by clicking on it, but doing it command line seems to be not as fun as getting the -latest one.

One of my objectives is to get the file using curl instead of wget. don't get me wrong, I like and use wget a lot. And, try to forget about the usual wget vs curl wars; the reason is much more pedestrian: OSX comes with curl. Since I wanted to run this not only on Linux but also on OSX without having to use brew or macports, curl it is.

Let's see if we can make a quick script to do the deed. I was going to see if I could come up with a clever script, but someone cut me to it. I really like how they used gsub, so I just had to see if I could apply that idea. Here is my first attempt

#!/bin/sh
# getdroid.sh

site="https://developer.android.com/sdk/index.html"
os="linux"

filename=`curl -s --insecure ${site} | \
awk '/id=\"'${os}'-tools\"/{
    gsub(/.*href=\"/,"")
    gsub(/\">.*/,"");
    if ($0) print
}'`

curl -O ${filename}

The short version of what I am doing:

1. for now we only care about a linux SDK, hence the os="linux". That said, you can see i am leaving it open to look for the SDK for another OS; you just need to know which pattern to look for.
   
2. Get a copy of the page source.
   
3. Look for the line that contains the link to the linux package.
   
4. Once that is found, grab the link and fetch the file.

The awk line looks rather busy because the search (the id= thingie) part of the statement escapes out to grab ${os}. You could make it easier to understand by using the normal garden-veriety search -- grep -- command to get the line that matches the id="linux-tools pattern, which then is piped into awk:

#!/bin/sh
# getdroid.sh

site="https://developer.android.com/sdk/index.html"
os="linux"

filename=`curl -s --insecure ${site} | \
grep "id=\"${os}-tools" | \
awk '{
    gsub(/.*href=\"/,"")
    gsub(/\">.*/,"");
    if ($0) print
}'`

curl -O ${filename}

Of course, as I mentioned I want to also use the same script on OSX; as it it, it will run on a Mac, but I will leave fetching the OSX version of the SDK as an exercise to the reader.

How about the NDK? It is found at https://developer.android.com/tools/sdk/ndk/index.html. You can do the same thing as I did above, but have to account for having a 32 and a 64bit version; just provide that to the script or make it aware of the OS you are running. Personally I would not do the later because I might want to, say, download the Windows 64bit files from a Mac. I will post my final script on github as soon as I have time; for now, what I put here should be enough to get you started.