Debugging bash multiline commands in vi

The ancient vi (1976), and its enhanced and slightly more modern (1991) successor vim have become unnapreciated workhorses in recent years. They come with some interesting features you expect to find in modern IDEs. Case in point is they have built-in features that help debug scripts. In today's episode, let's start with following script:

cat > multiline.sh << 'EOF'
#!/bin/bash
# Test multiline commands
# 20230926

is_this_on=true

if [ is_this_on ]; then
        echo "yes"
        ls -l \ 
                --author \
                $(pwd)
fi
EOF

Realistically, it only shows 3 things:

• if statements respond to true/false statements (the tests we feed them return that in the end of the day)
• The --author option offerend in modern gnu ls.

iamdeving@desktop:~/dev/scripts/shell$ ./multiline.sh 
yes
total 92
-rwxrwxr-x 1 iamdeving iamdeving iamdeving 1092 Jan 23  2020 argtest.sh
drwxrwxr-x 3 iamdeving iamdeving iamdeving 4096 Jun 28  2021 ca-stuff
-rw-rw-r-- 1 iamdeving iamdeving iamdeving  168 Jan 30  2020 commandlinetest.sh
-rwxrwxr-x 1 iamdeving iamdeving iamdeving  153 Jun 16  2021 countest.sh
[...]
iamdeving@desktop:~/dev/scripts/shell$

What is the third thing the script shows? How to break a command into multiple lines, which is done adding \ to where we want to have a break. In this case, the ls statement was broken up into 3 lines for no reason whatsoever but to show \ in action:

        ls -l \ 
                --author \
                $(pwd)

There is a potential problem in the above: if you add a space after one of the \, it will break the statement because now the \ is making the space special, not the linefeed (a.ka.a \n) character. I intentionally did that and rerun the script, leading to the following error.

iamdeving@desktop:~/dev/scripts/shell$ ./multiline.sh 
yes
ls: cannot access ' ': No such file or directory
./multiline.sh: line 11: --author: command not found
iamdeving@desktop:~/dev/scripts/shell$ 

If you are using vim, and have it configured to highlight the syntax of programming languages (it also does a great job with html, Python, github markup language, Jekyll), and others) and open the file, you will notice one of the \ is red. That indicates something is fishy around it. Moving the cursor will show there is a space after the slash:

That picture is great if

• You have setup vi to be in "helpful editing mode." But, it is not so great to show you there is an extra space there until you manually move your cursor there (I myself highlight the entire area and see if something that should not exist is hightighted). My trick is really not that helpful unless you know to look for it.
• You do not have vision impairements; it relies on you being able to see the colours (yes you can configure that), or being able to see at all.

Can we do something for those who do not meed the above requirements? Actually, yes. vi was created in a time where all text was one colour, usually green. So, it has aids that always work no matter the language you are editing or how fancy your terminal session is. In this case, while we have that file open, type :set list. You will not see the same file with some new characters added.

#!/bin/bash$
# Test multiline commands$
# 20230926$
$
is_this_on=true$
$
if [ is_this_on ]; then$
^Iecho "yes"$
^Ils -l \ $
^I^I--author \$ 
^I^I$(pwd)$     
fi$

The ^I represents a tab while the $ stands in for a linefeed character (we are editing a file written in Linux/UNIX/MacOS here, so it does not have carriage return characters Windows likes so much). If you look at the two lines that have a \ in the end, one of them have a space between the \ and the $ characters: we found the culprit.

If you do not want to use vi another humble option is cat:

iamdeving@desktop:~/dev/scripts/shell$ cat -A multiline.sh 
#!/bin/bash$
# Test multiline commands$
# 20230926$
$
is_this_on=true$
$
if [ is_this_on ]; then$
        echo "yes"$
        ls -l \ $
                --author \$
                $(pwd)$
fi$
iamdeving@desktop:~/dev/scripts/shell$ 

The uses for this feature do not end there. You can view special characters (besides tabs) and ensure there is nothing hidden that should not be there, which may not be as easy with helpful GUI-based IDEs.

Docker container user cannot write to shared volume

Something that has been baffling me for a while was how to mount a volume in a docker container such that a specific user inside said container would have the same group ID (gid) as a given user outside the container. But, before we delve into that, let's simplify the problem a bit with a few assumptions:

Assumptions

• The "user outside the container" is a user in the docker server. This user can run docker (is in the docker group). We shall call this user outbob, with uid = gid = 1500
• The "user inside the container" is a user in the docker container, which is created by the Dockerfile. This user will be known as inbob, with uid = gid = 1995
• This is a bare example, the smallest proof of concept I could come up with. I may update this article later with a link to a practical application as soon as I shove it in my github account.

The Problem

Let's create a really simple Dockerfile that shows the problem in /tmp/bob for no reason whatsoever.

outbob@dockerbox:/tmp/bob$ cat Dockerfile
# Set the base image to Ubuntu
FROM ubuntu

ENV DEVUSER inbob
ENV DEVID 1995

# Create user
RUN useradd -m --shell /bin/bash -u $DEVID $DEVUSER

USER $DEVUSER
ENV WD /home/${DEVUSER}
WORKDIR ${WD}
outbob@dockerbox:/tmp/bob$

We build the image from that Dockerfile as usual:

outbob@dockerbox:/tmp/bob$ docker build -t bob .
Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM ubuntu
 ---> 27941809078c
Step 2/7 : ENV DEVUSER inbob
 ---> Running in e8501dbe8398
[...]
 ---> Running in 963671a09a5f
Removing intermediate container 963671a09a5f
 ---> a6049000bfda
Successfully built a6049000bfda
Successfully tagged bob:latest
outbob@dockerbox:/tmp/bob$

And run it, passing the same directory we used because I am not in the mood of being original. Yes, I could pass the command I wanted to execute directly from the docker run statement instead of starting bash and then running the command. Deal with it.

outbob@dockerbox:/tmp/bob$ docker run -i --rm -v /tmp/bob:/bob -t bob bash
inbob@3d7c097a38d1:~$ id
uid=1995(inbob) gid=1995(inbob) groups=1995(inbob)
inbob@3d7c097a38d1:~$ exit
exit
outbob@dockerbox:/tmp/bob$ 

So far so good. Now what happens if I try to become an user with the same uid and gid as outbob?

outbob@dockerbox:/tmp/bob$ docker run -i --rm  -u $(id -u):$(id -g) -v /tmp/bob:/bob -t bob bash
groups: cannot find name for group ID 1500
I have no name!@c4355f38747a:/home/inbob$ id
uid=1500 gid=1500 groups=1500
I have no name!@c4355f38747a:/home/inbob$ cd /bob
I have no name!@c4355f38747a:/bob$ touch nose
I have no name!@c4355f38747a:/bob$ ls -lh
total 4.0K
-rw-r--r-- 1 1500 1500 200 Feb 14 01:01 Dockerfile
-rw-r--r-- 1 1500 1500   0 Feb 14 18:31 nose
I have no name!@c4355f38747a:/bob$ id inbob
uid=1995(inbob) gid=1995(inbob) groups=1995(inbob)
I have no name!@c4355f38747a:/bob$

It works in that I can write to the volume but I am now a completely different user; a user with no name. Since I am not Clint Eastwood, I would rather be inbob. Is there a solution?

The Solution (so far)

Note: if you do not want to cut-n-paste the following excerpts, I also put this code in a repo.

We stablished I do not want to find out inbob created something in /tmb/bob as its default gid. The cleanest solution I found so far is to add inbob to the same group outbob used to create /tmb/bob, and then ensure anyone belonging to that group can write to this directory as a member of that group. So, let's do whatever I just said!

Set the directory in question to inherit the gid

We will set the setgid attribute for the directory:

outbob@dockerbox:/tmp/bob$ ls -ld /tmp/bob
drwxr----- 2 outbob outbob 4096 Mar  9 08:55 /tmp/bob/
outbob@dockerbox:/tmp/bob$
outbob@dockerbox:/tmp/bob$ chmod 2770 /tmp/bob
outbob@dockerbox:/tmp/bob$
outbob@dockerbox:/tmp/bob$ ls -ld /tmp/bob
drwxrws--- 2 outbob outbob 4096 Mar  9 08:55 /tmp/bob/
outbob@dockerbox:/tmp/bob$

The little s after the group permissions indicate we have been successful.

Configure the docker build to add user to proper group at runtime

We need to change the Dockerfile a bit and then create a docker-entrypoint.sh file:

outbob@dockerbox:/tmp/bob$ cat Dockerfile
# Set the base image to Ubuntu
FROM ubuntu

ENV DEVUSER inbob
ENV DEVUID 1995
ENV DEVGID 1995
ENV EXTGID 1999

# Create user
RUN useradd -m --shell /bin/bash -u $DEVUID $DEVUSER

# USER $DEVUSER
ENV WD /home/${DEVUSER}
WORKDIR ${WD}

# Put the entrypoint script somewhere we can find
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod 0700 /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

outbob@dockerbox:/tmp/bob$ cat docker-entrypoint.sh
#!/bin/sh
set -e

groupadd -g $EXTGID extgroup
adduser $DEVUSER extgroup

su - $DEVUSER

# And we are done here
exec "$@"
outbob@dockerbox:/tmp/bob$

Test time!

After we buld the new image we run it passing the default gid for outbob. Nothing stopping us from passing a different gid if it is what we need; docker does not care.

outbob@dockerbox:/tmp/bob$ docker run -i --rm  -e EXTGID=$(id -g) -v /tmp/bob:/bob -t bob bash
Adding user `inbob' to group `extgroup' ...
Adding user inbob to group extgroup
Done.
inbob@2a7339336f3e:~$ cd /bob
inbob@2a7339336f3e:/bob$ ls -l
total 8
-rw-r--r-- 1 5000 extgroup 389 Mar  9 13:42 Dockerfile
-rw-r--r-- 1 5000 extgroup 122 Mar  9 13:48 docker-entrypoint.sh
inbob@2a7339336f3e:/bob$ touch nose
inbob@2a7339336f3e:/bob$ ls -l
total 8
-rw-r--r-- 1    5000 extgroup 389 Mar  9 13:42 Dockerfile
-rw-r--r-- 1    5000 extgroup 122 Mar  9 13:48 docker-entrypoint.sh
-rw-rw-r-- 1 inbob extgroup   0 Mar  9 14:00 nose
inbob@2a7339336f3e:/bob$ exit
logout
root@2a7339336f3e:/home/inbob# exit
exit
outbob@dockerbox:/tmp/bob$ ls -l
total 8
-rw-r--r-- 1 outbob outbob 122 Mar  9 08:48 docker-entrypoint.sh
-rw-r--r-- 1 outbob outbob 389 Mar  9 08:42 Dockerfile
-rw-rw-r-- 1 1995 outbob   0 Mar  9 09:00 nose
outbob@dockerbox:/tmp/bob$ rm nose
outbob@dockerbox:/tmp/bob$

Other than getting out is now a two-step process, which may not be important when running a container in the background, the only telltale we were up to not good is that the uid for the file we created does not match any in our docker server. But, outbob can still delete the file.

And that is how we bring harmony in the divided worlds of bob.