Showing posts with label container. Show all posts
Showing posts with label container. Show all posts

Thursday, March 09, 2023

Docker container user cannot write to shared volume

Something that has been baffling me for a while was how to mount a volume in a docker container such that a specific user inside said container would have the same group ID (gid) as a given user outside the container. But, before we delve into that, let's simplify the problem a bit with a few assumptions:

Assumptions

  • The "user outside the container" is a user in the docker server. This user can run docker (is in the docker group). We shall call this user outbob, with uid = gid = 1500
  • The "user inside the container" is a user in the docker container, which is created by the Dockerfile. This user will be known as inbob, with uid = gid = 1995
  • This is a bare example, the smallest proof of concept I could come up with. I may update this article later with a link to a practical application as soon as I shove it in my github account.

The Problem

Let's create a really simple Dockerfile that shows the problem in /tmp/bob for no reason whatsoever.

outbob@dockerbox:/tmp/bob$ cat Dockerfile
# Set the base image to Ubuntu
FROM ubuntu

ENV DEVUSER inbob
ENV DEVID 1995

# Create user
RUN useradd -m --shell /bin/bash -u $DEVID $DEVUSER

USER $DEVUSER
ENV WD /home/${DEVUSER}
WORKDIR ${WD}
outbob@dockerbox:/tmp/bob$

We build the image from that Dockerfile as usual:

outbob@dockerbox:/tmp/bob$ docker build -t bob .
Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM ubuntu
 ---> 27941809078c
Step 2/7 : ENV DEVUSER inbob
 ---> Running in e8501dbe8398
[...]
 ---> Running in 963671a09a5f
Removing intermediate container 963671a09a5f
 ---> a6049000bfda
Successfully built a6049000bfda
Successfully tagged bob:latest
outbob@dockerbox:/tmp/bob$

And run it, passing the same directory we used because I am not in the mood of being original. Yes, I could pass the command I wanted to execute directly from the docker run statement instead of starting bash and then running the command. Deal with it.

outbob@dockerbox:/tmp/bob$ docker run -i --rm -v /tmp/bob:/bob -t bob bash
inbob@3d7c097a38d1:~$ id
uid=1995(inbob) gid=1995(inbob) groups=1995(inbob)
inbob@3d7c097a38d1:~$ exit
exit
outbob@dockerbox:/tmp/bob$ 

So far so good. Now what happens if I try to become an user with the same uid and gid as outbob?

outbob@dockerbox:/tmp/bob$ docker run -i --rm  -u $(id -u):$(id -g) -v /tmp/bob:/bob -t bob bash
groups: cannot find name for group ID 1500
I have no name!@c4355f38747a:/home/inbob$ id
uid=1500 gid=1500 groups=1500
I have no name!@c4355f38747a:/home/inbob$ cd /bob
I have no name!@c4355f38747a:/bob$ touch nose
I have no name!@c4355f38747a:/bob$ ls -lh
total 4.0K
-rw-r--r-- 1 1500 1500 200 Feb 14 01:01 Dockerfile
-rw-r--r-- 1 1500 1500   0 Feb 14 18:31 nose
I have no name!@c4355f38747a:/bob$ id inbob
uid=1995(inbob) gid=1995(inbob) groups=1995(inbob)
I have no name!@c4355f38747a:/bob$

It works in that I can write to the volume but I am now a completely different user; a user with no name. Since I am not Clint Eastwood, I would rather be inbob. Is there a solution?

The Solution (so far)

Note: if you do not want to cut-n-paste the following excerpts, I also put this code in a repo.

We stablished I do not want to find out inbob created something in /tmb/bob as its default gid. The cleanest solution I found so far is to add inbob to the same group outbob used to create /tmb/bob, and then ensure anyone belonging to that group can write to this directory as a member of that group. So, let's do whatever I just said!

Set the directory in question to inherit the gid

We will set the setgid attribute for the directory:

outbob@dockerbox:/tmp/bob$ ls -ld /tmp/bob
drwxr----- 2 outbob outbob 4096 Mar  9 08:55 /tmp/bob/
outbob@dockerbox:/tmp/bob$
outbob@dockerbox:/tmp/bob$ chmod 2770 /tmp/bob
outbob@dockerbox:/tmp/bob$
outbob@dockerbox:/tmp/bob$ ls -ld /tmp/bob
drwxrws--- 2 outbob outbob 4096 Mar  9 08:55 /tmp/bob/
outbob@dockerbox:/tmp/bob$

The little s after the group permissions indicate we have been successful.

Configure the docker build to add user to proper group at runtime

We need to change the Dockerfile a bit and then create a docker-entrypoint.sh file:

outbob@dockerbox:/tmp/bob$ cat Dockerfile
# Set the base image to Ubuntu
FROM ubuntu

ENV DEVUSER inbob
ENV DEVUID 1995
ENV DEVGID 1995
ENV EXTGID 1999

# Create user
RUN useradd -m --shell /bin/bash -u $DEVUID $DEVUSER

# USER $DEVUSER
ENV WD /home/${DEVUSER}
WORKDIR ${WD}

# Put the entrypoint script somewhere we can find
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod 0700 /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

outbob@dockerbox:/tmp/bob$ cat docker-entrypoint.sh
#!/bin/sh
set -e

groupadd -g $EXTGID extgroup
adduser $DEVUSER extgroup

su - $DEVUSER

# And we are done here
exec "$@"
outbob@dockerbox:/tmp/bob$

Test time!

After we buld the new image we run it passing the default gid for outbob. Nothing stopping us from passing a different gid if it is what we need; docker does not care.

outbob@dockerbox:/tmp/bob$ docker run -i --rm  -e EXTGID=$(id -g) -v /tmp/bob:/bob -t bob bash
Adding user `inbob' to group `extgroup' ...
Adding user inbob to group extgroup
Done.
inbob@2a7339336f3e:~$ cd /bob
inbob@2a7339336f3e:/bob$ ls -l
total 8
-rw-r--r-- 1 5000 extgroup 389 Mar  9 13:42 Dockerfile
-rw-r--r-- 1 5000 extgroup 122 Mar  9 13:48 docker-entrypoint.sh
inbob@2a7339336f3e:/bob$ touch nose
inbob@2a7339336f3e:/bob$ ls -l
total 8
-rw-r--r-- 1    5000 extgroup 389 Mar  9 13:42 Dockerfile
-rw-r--r-- 1    5000 extgroup 122 Mar  9 13:48 docker-entrypoint.sh
-rw-rw-r-- 1 inbob extgroup   0 Mar  9 14:00 nose
inbob@2a7339336f3e:/bob$ exit
logout
root@2a7339336f3e:/home/inbob# exit
exit
outbob@dockerbox:/tmp/bob$ ls -l
total 8
-rw-r--r-- 1 outbob outbob 122 Mar  9 08:48 docker-entrypoint.sh
-rw-r--r-- 1 outbob outbob 389 Mar  9 08:42 Dockerfile
-rw-rw-r-- 1 1995 outbob   0 Mar  9 09:00 nose
outbob@dockerbox:/tmp/bob$ rm nose
outbob@dockerbox:/tmp/bob$

Other than getting out is now a two-step process, which may not be important when running a container in the background, the only telltale we were up to not good is that the uid for the file we created does not match any in our docker server. But, outbob can still delete the file.

And that is how we bring harmony in the divided worlds of bob.

Saturday, November 09, 2019

Creating a RSA key for rsync for Android using docker

I use Rsync4Android to backup my phone. It is convenient and you can even run a cronjob so it does its thing without your intervention (think of it backing up while you are having dinner somewhere) and it backs up to wherever you want, so you are in control. Now, one day it stopped working. Not knowing what was going on, I did what one usually does when dealing with ssh issues: run sshd in debug mode:

raub@desktop:~$ sudo /usr/sbin/sshd -D
[...]
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: inetd sockets after dupping: 3, 3
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user raub service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "raub"
debug1: PAM: setting PAM_RHOST to "10.0.0.129"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user raub service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]
Connection closed by 10.0.0.129 port 39739 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 10239
debug1: audit_event: unhandled event 12
raub@desktop:~$

The line that tells what is going on is

userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]

When creating a key pair, Rsync4Android uses DSA algorithm. As we know, DSA has been considered insecure for a while and the current releases of openssh do not support it by default. So, if I want to keep on using rsync4android, I either configure my ssh server to accept DSA keys or find a way to convince it to use a RSA key. I chose the RSA route, but how to do it?

Create Key

Nothing special here.

ssh-keygen -t rsa -b 4096 -C "Phone_backup" -f ~/.ssh/phonebackup

Don't like 4096 bits? Double it; Rsync4Android does not care.

Convert private key to something Rsync4Android can use

The public key will go to the Linux host, which runs openssh and can handle those keys just fine. Rsync4Android, on the other hand, needs dropbear style keys. As mentioned in Rsync4Android docs, the best way to convert is to use the dropbearconvert command, which for ubuntu comes in the dropbear package. As I did not want to install it in desktop, I quickly created a docker container, copied the private key, and then installed the package. And then ran it (note the path) telling I am feeding it an openssh format key and want a dropbear style key:

root@b3f7ed8c4f24:/home# apt-get install dropbear
[...]
root@b3f7ed8c4f24:/home# /usr/lib/dropbear/dropbearconvert openssh dropbear phonebackup phonebackup.dropbear
Key is a ssh-rsa key
Wrote key to 'phonebackup.dropbear'
root@b3f7ed8c4f24:/home#

Now I have the dropbear-formated private key, phonebackup.dropbear, I can finally set the circus up.

Copy public and private keys to the proper locations

Public key goes to the computer we are backing the phone to (in my case desktop): added to the account's ~/.ssh/athorized_keys file.

The private key to the android. I put it in the same directory Rsync4Android placed the original (DSA) key it created, /sdcard. Then it was a matter of renaming the key in the Rsync4Android config and running it again.

Test it

Rsync4Android has a dry run mode so you can see if it works. When testing, I also ran sshd in debug mode. Then I ran the backup in "production mode"; to the left is a screen capture of my phone. The reason you see a lot of output is because I have --partial --progress as rsync options; you should configure it to fit your needs. I suggest to also check on the --exclude option; I think you will find it quite useful.

Now we know it works, we can worry about running it as a cron job (the Rsync4Android docs have a link for how to do that) and then make it work from an external network. But that is for another article.

Saturday, January 30, 2016

Mounting user fileshare on boot2docker boot

For docker container development, and light using, I found boot2docker to be quite convenient. I have it to boot off its ISO and then mount a permanent drive for the containers (and to store config files).

/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
Whenever there is a new version, I shut the vm down, swap the ISOs (really just point the alias to the new file), and reboot. Nice and brainless.

I do not like to use the default account, docker, to do container development and running. Also, in a nice production environment you want to have other users running their containers. So, I created a user called ducker, which is a quick play on the default username. I also would prefer not having the user homedir in the drive where the containers are, which has been suggested before. You see, the way I see containers they are by design not important; blow them up if you feel like or wonder if they have been compromised. What matters is the data and the dockerfile required to rebuild the container. As a result, ducker has an account in the fileserver, which does its RAID and backup thingie as any good fileserver should. Now, if we want to have containers created and running from ducker's account when the server boots up, we need to have said account available.

So, the plan is that whenever the boot2docker server reboots, ducker will be there. And we then have a few issues to deal with. First, docker2file's ISO can't do automount. And second, usually when we shut down any user is lost because we are running of an ISO.

boot2docker's ISO is built so that we can provide some permanent stuff, which is why it mounts /dev/sda. But, that is not the only place we can mount things, nor the only way. In /opt/bootscript.sh we have the following interesting lines:

# Allow local bootsync.sh customisation
if [ -e /var/lib/boot2docker/bootsync.sh ]; then
    /bin/sh /var/lib/boot2docker/bootsync.sh
    echo "------------------- ran /var/lib/boot2docker/bootsync.sh"
fi

# Launch Docker
/etc/rc.d/docker

# Allow local HD customisation
if [ -e /var/lib/boot2docker/bootlocal.sh ]; then
    /bin/sh /var/lib/boot2docker/bootlocal.sh > /var/log/bootlocal.log 2>&1 &
    echo "------------------- ran /var/lib/boot2docker/bootlocal.sh"
fi

The files in /var/lib/boot2docker are in /dev/sda. Don't know which one to pick, but the second one does have a comforting Allow local HD customization message. So I will pick /var/lib/boot2docker/bootlocal.sh and add the following lines:

# Create local user, also creating the homedir
adduser -D -u 1003 ducker
# add user to docker group
adduser ducker docker
# Mount homedir
mount.nfs  fileserver.example.com:/export/home/ducker /home/ducker

After we create the file and reboot, we get

docker@boot2docker:~$ id ducker
uid=1003(ducker) gid=1003(ducker) groups=1003(ducker),100(docker)
docker@boot2docker:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    123.8M    772.8M  14% /
tmpfs                   498.1M         0    498.1M   0% /dev/shm
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda
cgroup                  498.1M         0    498.1M   0% /sys/fs/cgroup
df: /mnt/hgfs: Protocol error
fileserver.example.com:/home/ducker
                        295.3G    285.1G     10.0G  97% /home/ducker
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
docker@boot2docker:~$

User ducker can login because there is a public ssh key already in place. However, we did not do anything for that user's password, but there are a few ways to take care of that such as using /var/lib/boot2docker/bootlocal.sh to copy the hash into /etc/passwd. And, you really want to take care of that otherwise you will not be able to login.

Tuesday, June 23, 2015

Ubuntu Docker container with multiple architectures

Here's yet another quickie: let's say you created a docker Ubuntu container to do some sort of cross compilation. You know, maybe your container is 64bit Intel and you need to spit out some static 32bit Intel stuff. Or PPC for that matter. Now your development environment needs the 32bit version of libncurses-dev, which for ubuntu 14.04 would be libncurses5-dev:i386. We do need to specify the architecture (:i386) if we are not using the default one..

So you scribble up a quick Dockerfile:

############################################################
# Dockerfile to build a ubuntu container image thingie
# Based on Ubuntu (Duh!)
############################################################

# Set the base image to Ubuntu
FROM ubuntu:14.04

################## BEGIN INSTALLATION ######################

RUN apt-get update && apt-get -y upgrade 

RUN apt-get install -y build-essential curl libc6-dev libncurses5-dev:i386

##################### INSTALLATION END #####################
And then you do some building:
ducker@boot2docker:~/docker/test$ docker build -t raubvogel/test.
Sending build context to Docker daemon 4.096 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
 ---> 6d4946999d4f
Step 1 : RUN apt-get update && apt-get -y upgrade
 ---> Running in 2d035053a431
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
[...]
Processing triggers for ureadahead (0.100.0-16) ...
Setting up initscripts (2.88dsf-41ubuntu6.2) ...
guest environment detected: Linking /run/shm to /dev/shm
 ---> c1434c42218e
Removing intermediate container 2d035053a431
Step 2 : RUN apt-get install -y build-essential zip bzr curl libc6-dev libncurses5-dev:i386
 ---> Running in 2ac39f9430c2
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libncurses5-dev
INFO[0017] The command [/bin/sh -c apt-get install -y build-essential zip bzr curl libc6-dev libncurses5-dev:i386] returned a non-zero code: 100
ducker@boot2docker:~/docker/test$
And then it barks at the libncurses5-dev entry. What's going on?

The Debian Multiarch HOWTO seems to have a bit of a clue. I was going to write a long winded explanation but I am bored. So here is the short version:

dpkg --add-architecture i386
apt-get update
apt-get install libncurses5-dev:i386
See the dpkg --add-architecture i386 line? It tells the host that we are can do 32bit Intell thingies. The next line is there just to feed us with the 32bit repository data, so when we look for a 32bit package we can get it. Let apply that to our Dockerfile:
############################################################
# Dockerfile to build a ubuntu container image thingie
# Based on Ubuntu (Duh!)
############################################################

# Set the base image to Ubuntu
FROM ubuntu:14.04

################## BEGIN INSTALLATION ######################

# We need i386 crap
RUN dpkg --add-architecture i386
# Business as usual
RUN apt-get update && apt-get -y upgrade 

RUN apt-get install -y build-essential curl libc6-dev libncurses5-dev:i386

##################### INSTALLATION END #####################
And build it again.
ducker@boot2docker:~/docker/test$ docker build -t raubvogel/test.
Sending build context to Docker daemon 4.608 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
 ---> 6d4946999d4f
Step 1 : RUN dpkg --add-architecture i386
 ---> Running in 89382c3c3469
 ---> f367d3357fc4
Removing intermediate container 89382c3c3469
Step 2 : RUN apt-get update && apt-get -y upgrade
 ---> Running in 6d0fc9519029
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
[...]
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 build-essential : Depends: gcc (>= 4:4.4.3) but it is not going to be installed
                   Depends: g++ (>= 4:4.4.3) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
INFO[0002] The command [/bin/sh -c apt-get install -y build-essential  zip bzr curl libc6-dev libncurses5-dev:i386] returned a non-zero code: 100
ducker@boot2docker:~/docker/test$
Hmmm, that didn't work. Crap. Let's massage Dockerfile a bit:
############################################################
# Dockerfile to build a ubuntu container image thingie
# Based on Ubuntu (Duh!)
############################################################

# Set the base image to Ubuntu
FROM ubuntu:14.04

################## BEGIN INSTALLATION ######################

# We need i386 crap
RUN dpkg --add-architecture i386
# Business as usual
RUN apt-get update && apt-get -y upgrade

RUN apt-get install -y build-essential  &&\
    apt-get install -y zip bzr curl libc6-dev libncurses5-dev:i386

##################### INSTALLATION END #####################
And this time it works fine:
ducker@boot2docker:~/docker/test$ docker build -t raubvogel/test.
Sending build context to Docker daemon 4.608 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
[...]
Setting up libpam-systemd:amd64 (204-5ubuntu20.12) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
invoke-rc.d: unknown initscript, /etc/init.d/systemd-logind not found.
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
Processing triggers for ca-certificates (20141019ubuntu0.14.04.1) ...
Updating certificates in /etc/ssl/certs... 173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
 ---> d86a5113c577
Removing intermediate container 32e0cc9a0b5a
Successfully built d86a5113c577
ducker@boot2docker:~/docker/test$ 

What I found out is that it does not like to have the build-essentials in the same install statement as libncurses5-dev:i386. The easiest solution is to have build-essentials on its own install statement and everyone else in the next one. And life is once again well.

Sunday, February 01, 2015

Entrypoint scripts and data persistency: mounting a NFS fileshare inside a docker container

This is a quick article; you have been warned.

The traditional way to have persistent data in a docker container is to feed it with a volume from the docker host. You begin by locating a directory, be it in the local docker host drive, network mounted to the docker host, or a volume in another docker container. Then feed it to the container using the VOLUME statement defined in the Dockerfile or to command line. We all know that. But what some (very few we hope) of you might not have been aware of is the volume is only mounted/attached to the container when you tell docker to run the container, which might make running/configuring something that require files in those volumes a bit challenging.

At this point I would expect one of you to shout "Aha! But you can then use an entrypoint script to do all that monkeying that needs to happen after volume is mounted and before whatever service this container provides starts!" And you would be quite right! Here's a quick example in case we lost somebody: if our Dockerfile ended up like this:

# Put the entrypoint script somewhere we can find
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

EXPOSE 22
# Start service
CMD ["/usr/sbin/sshd", "-D"]

we would have created a file called entrypoint.sh in the root path of the container, which is run just after the volume is created and before the, in this example, sshd service is started. Said entrypoint file could do, I don't know:

#!/bin/sh
set -e

# Do something in /home, which is the mounted volume
/do-something-at-home.sh

# And we  are done here
exec "$@"

point is that do-something-at-home.sh is called only after the persistent volume is mounted into the container.

What if you want to mount a network fileshare straight into the container? Before I have time to answer that, someone from the audience will jump and state "Why would you want to do that? You can mount the network fileshare in the docker host and then use VOLUME just like Wicked Witch of the West intended!" What if I don't want to do that? What if I can't for whatever reason mount a network fileshare on the docker host?

The answer is that it is completely doable; you just have to ask the entrypoint script to do the mounting for you. To show what I mean I will use the example of mounting a NFSv4 fileshare that is supposed to be used by some website. So we modify our little entrypoint script a bit

#!/bin/sh
set -e

# /export/www was created in the Dockerfile
mount.nfs4 fileserver.example.com:/export/www /www/some-website 

# And we are done here
exec "$@"

Since we are using NFSv4, chances are you might need to add something like

RUN sed -i -e '/^#Domain/a Domain = example.com' /etc/idmapd.conf

to your dockerfile in addition to telling it to get the nfs client packages, but the general idea should apply for, say, SMB or ZFS or whatever other network filesystem that you fancy: let your entrypoint script do the heavy lifting!