Capturing the output of a command sent through ssh within script without it asking to verify host key

You have noticed when you connect to a new server, if its key is not in ~/.ssh/known_hosts ssh will ask you to verify the key:

raub@desktop:~$ ssh -i $SSHKEY_FILE cc@"$server_ip"
The authenticity of host 'headless.example.com (10.0.1.160)' can't be established.
ECDSA key fingerprint is SHA256:AgwYevnTsG2m9hQLu/ROp+Rjj5At2HU0HVoGZ+5Ug58.
Are you sure you want to continue connecting (yes/no)? 

That is a bit of a drag if you want to run a script to connect to said server. Fortunately ssh has an option just for that, StrictHostKeyChecking, as mentioned in the man page:

ssh automatically maintains and checks a database containing identification for all hosts it has ever been used with. Host keys are stored in ~/.ssh/known_hosts in the user's home directory. Additionally, the file /etc/ssh/ssh_known_hosts is automatically checked for known hosts. Any new hosts are automatically added to the user's file. If a host's identification ever changes, ssh warns about this and disables password authentication to prevent server spoofing or man-in-the-middle attacks, which could otherwise be used to circumvent the encryption. The StrictHostKeyChecking option can be used to control logins to machines whose host key is not known or has changed.

Now we can rewrite the ssh command as

ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip"

and it will login without waiting for us to verify the key. Of course this should only be done when the balance between security and automation meets your comfort level. However, if we run the above from a script, it will connect and just stay there with the session open, not accept any commands. i.e. if we try to send a command, pwd in this case,

ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip"
pwd

it will never see the pwd. Only way to get to the pwd is to kill the ssh session, when it will then go to the next command but running it in the local host, not the remote one. If we want to run pwd in the host we ssh into, we need to pass it as an argument, i.e.

raub@desktop:/tmp$ pwd
/tmpraub@desktop:/tmp$ ssh -o "StrictHostKeyChecking no" -i $SSHKEY_FILE cc@"$server_ip" pwd
/home/raub
raub@desktop:/tmp$

which means it will connect, run said command, and then exit. But, how to do that from a script? The answer is to use eval or bash -c (or other available shell):

raub@desktop:/tmp$ moo="ssh -o \"StrictHostKeyChecking no\" -i $SSHKEY_FILE cc@$server_ip"
raub@desktop:/tmp$ dirlist=$(eval "$moo pwd")
raub@desktop:/tmp$ echo $dirlist
/home/raub
raub@desktop:/tmp$ dirlist2=$(sh -c "$moo pwd")
raub@desktop:/tmp$ echo $dirlist2
/home/raub
raub@desktop:/tmp$

Now, there are subtle differences between eval and bash -c. There is a good thread in stackexchange which explains them better than I can do.

Now why someone would want to do that?

How about doing some poor-man ansible, i.e. when host on the other side does not have python or has some other idiosyncrasy. Like my home switch, but that is another story...

Creating a RSA key for rsync for Android using docker

I use Rsync4Android to backup my phone. It is convenient and you can even run a cronjob so it does its thing without your intervention (think of it backing up while you are having dinner somewhere) and it backs up to wherever you want, so you are in control. Now, one day it stopped working. Not knowing what was going on, I did what one usually does when dealing with ssh issues: run sshd in debug mode:

raub@desktop:~$ sudo /usr/sbin/sshd -D
[...]
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: inetd sockets after dupping: 3, 3
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user raub service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "raub"
debug1: PAM: setting PAM_RHOST to "10.0.0.129"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user raub service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]
Connection closed by 10.0.0.129 port 39739 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 10239
debug1: audit_event: unhandled event 12
raub@desktop:~$

The line that tells what is going on is

userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]

When creating a key pair, Rsync4Android uses DSA algorithm. As we know, DSA has been considered insecure for a while and the current releases of openssh do not support it by default. So, if I want to keep on using rsync4android, I either configure my ssh server to accept DSA keys or find a way to convince it to use a RSA key. I chose the RSA route, but how to do it?

Create Key

Nothing special here.

ssh-keygen -t rsa -b 4096 -C "Phone_backup" -f ~/.ssh/phonebackup

Don't like 4096 bits? Double it; Rsync4Android does not care.

Convert private key to something Rsync4Android can use

The public key will go to the Linux host, which runs openssh and can handle those keys just fine. Rsync4Android, on the other hand, needs dropbear style keys. As mentioned in Rsync4Android docs, the best way to convert is to use the dropbearconvert command, which for ubuntu comes in the dropbear package. As I did not want to install it in desktop, I quickly created a docker container, copied the private key, and then installed the package. And then ran it (note the path) telling I am feeding it an openssh format key and want a dropbear style key:

root@b3f7ed8c4f24:/home# apt-get install dropbear
[...]
root@b3f7ed8c4f24:/home# /usr/lib/dropbear/dropbearconvert openssh dropbear phonebackup phonebackup.dropbear
Key is a ssh-rsa key
Wrote key to 'phonebackup.dropbear'
root@b3f7ed8c4f24:/home#

Now I have the dropbear-formated private key, phonebackup.dropbear, I can finally set the circus up.

Copy public and private keys to the proper locations

Public key goes to the computer we are backing the phone to (in my case desktop): added to the account's ~/.ssh/athorized_keys file.

The private key to the android. I put it in the same directory Rsync4Android placed the original (DSA) key it created, /sdcard. Then it was a matter of renaming the key in the Rsync4Android config and running it again.

Test it

Rsync4Android has a dry run mode so you can see if it works. When testing, I also ran sshd in debug mode. Then I ran the backup in "production mode"; to the left is a screen capture of my phone. The reason you see a lot of output is because I have --partial --progress as rsync options; you should configure it to fit your needs. I suggest to also check on the --exclude option; I think you will find it quite useful.

Now we know it works, we can worry about running it as a cron job (the Rsync4Android docs have a link for how to do that) and then make it work from an external network. But that is for another article.