Showing posts with label nfs. Show all posts
Showing posts with label nfs. Show all posts

Monday, December 09, 2019

Configuring autofs in a generic way on redhat/debian derived distros in ansible

Let's say you want to install and configure autofs so to have network fileshares mounted on demand. That sounds like a good task for Ansible.

Installing AutoFS

The installing part as you know is fairly easy. In both RedHat and Debian derivatives the package name is autofs. Since we are not doing anything special we can create a rather generic Ansible task using the Ansible package module. What that means is instead of having to worry about having a version for, say, Ubuntu and RedHat, the package module uses whatever package manager is the default for the distro in question:

- name: setup autofs
  package:
    name: autofs
    state: latest
NOTE: I know package works in the redhat/debian derived distros, but I am not sure if it will work in other Linux flavours. I also would check the autofs package name for these other distros.

Configuring AutoFS

Next thing we want to do is ensure we are using the right NFS version. That is done finding the line beginning with mount_nfs_default_protocol and editing its value. In my case I want to make sure it is using NFS v4, which is the default anyway in a modern autofs package. So, why bother? Well, call me paranoid: I want to have exactly what I want. Or call it a simple example of using the Ansible lineinfile module. Or maybe someone is using NFS v3 and want to see how to change it.

- name: Ensure nfs v4
  lineinfile:
    path: /etc/autofs.conf
    regexp: '^mount_nfs_default_protocol '
    line: 'mount_nfs_default_protocol = 4'

Let's use this as an excuse to talk about lineinfile: the regexp here is looking for a line that starts with the string 'mount_nfs_default_protocol '; I wrote it in quotes because it includes the blank space after mount_nfs_default_protocol. Note that the search pattern also means "and anything else to the end of the line," so a line looking like this

mount_nfs_default_protocol could be something you do not want to touch
is fair game. I know usually in regexp you would end the query statement with (.*)$ to include everything to the end of the page, but just nod a lot and move on. Now if the looked like this:
#mount_nfs_default_protocol could be something you do not want to touch
the regexp would not work because it expects the line to begin with m. line defines what we want the line to look like. If it matches that, no changes made.

The next step is to define what we want to use autofs for. In my case, I want to mount user home directories off the fileserver. That means creating a /etc/auto.home file which describes which fileserver we are using. For this I suggest using the template module since we can define the name of the fileserver somewhere earlier in the playbook or in a config file (I am thinking here of a file in host_vars/ or group_vars/ associated with the host in question.). In my task file I use something like

- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs
which
  1. Grabs the template templates/auto.home.j2 and puts it in /etc/auto.home
  2. Sets the permission, ownership, and selinux parameters for /etc/auto.home. The _default means that if there is a default selinux setting for that file/directory we will use it.

And roles/common/templates/auto.home.j2 looks like this:

/etc/auto.home
#
# File: /etc/auto.home
#
*   -fstype=nfs4,hard,intr,rsize=8192,wsize=8192 {{ nfs_server }}:/home/&
where nfs_server is the name of the nfs server defined somewhere else.

Now, there are two ways to deal with it, the old autofs and the new one.

  • Old autofs: In the old days, you would edit the /etc/auto.master file, adding a line underneath +auto.master that would tell us how to mount user home directory. In the following example,
    +auto.master
    /home   /etc/auto.home --timeout=300
    
    the bottom line is saying "if you notice someone trying to access a file/directory in /home, go to /etc/auto.home to see how to mount it. But, if there is no activity after 300 seconds, unmount that." I use timeout of 300 seconds; change it to fit your needs. Now we need to add that to /etc/auto.master, which we will do using the lineinfile module once more:

    - name: Enable auto.home in auto.master
      lineinfile:
        path: /etc/auto.master
        regexp: '^\/home'
        insertafter: '^\+auto.master'
        line: /home   /etc/auto.home --timeout=300

    As you can see, it is a little more complex than the previous task:

    • The regexp statement has to escape the /
    • insertafter is used to look for a line which matches that pattern and then insert/change the line we want below this pattern. This is useful when you have more than one line matching the regexp pattern or you want the line we are creating/replacing to be on a specific location. You see, without that if line does not exist, it is appended on the end of the file.
  • New autofs: The more modern /etc/auto.master file has the following lines in it:
    #
    # Include /etc/auto.master.d/*.autofs
    # The included files must conform to the format of this file.
    #
    +dir:/etc/auto.master.d
    #

    Instead of editing the /etc/auto.master file, which might be overwritten by an upgrade, we simply throw a file inside the /etc/auto.master.d directory which is then loaded into the /etc/auto.master file. This file, say /etc/auto.master.d/home.autofs, looks very much like what we did in the old autofs example, main difference is that it is a file

    raub@desktop:~/dev/ansible$ cat roles/common/files/home.autofs
    /home   /etc/auto.home --timeout=300
    raub@desktop:~/dev/ansible$
    that needs to be uploaded using the Ansible file copy module:

    - name: Enable auto.home in auto.master.d
      copy:
        src: home.autofs
        dest: /etc/auto.master.d/home.autofs
        owner: root
        group: root
        serole: _default
        setype: _default
        seuser: _default
        mode: 0644

Which one to pick? You know your setup, so pick the one that fits your needs.

(Re)Starting autofs

The final step we do need to do is (re)start autofs service after all this configuring. We do that using handlers:

- name: start autofs
  service:
    name: autofs
    state: started
    enabled: yes

- name: restart autofs
  service:
    name: autofs
    state: restarted
    enabled: yes

The way I use them is to start autofs when package is installed (see the notify statement),

- name: setup autofs
  package:
    name: autofs
    state: latest
  notify: start autofs
and then restart it after finishing with auto.home
- name: configure auto.home
  template:
    src: auto.home.j2
    dest: /etc/auto.home
    mode: 0644
    owner: root
    group: root
    serole: _default
    setype: _default
    seuser: _default
  notify: restart autofs

After I unleash ansible, I then ssh as the non-root user which is allowed to login to vmhost2 and then see if fileshare was mounted:

[raub@vmhost2 ~]$ df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                               16G     0   16G   0% /dev
tmpfs                                  16G     0   16G   0% /dev/shm
tmpfs                                  16G  8.9M   16G   1% /run
tmpfs                                  16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/vmhost-root               2.0G   71M  2.0G   4% /
/dev/mapper/vmhost-usr                4.0G  1.8G  2.2G  46% /usr
/dev/sda2                             976M  179M  731M  20% /boot
/dev/sda1                             200M  6.8M  194M   4% /boot/efi
/dev/mapper/vmhost-var                4.0G  376M  3.7G  10% /var
/dev/mapper/vmhost-vg_backup           10G  104M  9.9G   2% /var/lib/libvirt/qemu/save
fileserver.example.com:/home/raub     690G  629G   61G  92% /home/raub
tmpfs                                 3.2G     0  3.2G   0% /run/user/1001
[raub@vmhost2 ~]$
[raub@vmhost2 ~]$ systemctl status autofs
● autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor pres>
   Active: active (running) since Mon 2019-12-09 14:17:26 EST; 2min 39s ago
 Main PID: 28387 (automount)
    Tasks: 6 (limit: 26213)
   Memory: 3.3M
   CGroup: /system.slice/autofs.service
           └─28387 /usr/sbin/automount --foreground --dont-check-daemon
[raub@vmhost2 ~]$

I do not know about you but it seems we have a winner. I will put a cleaner version of this playbook and supporting files in my github account later on.

Thursday, December 05, 2019

Replacing a VMWare ESXi host with a KVM one

Why?

Ok, I think I did my due diligence. With all the entries I have in this blog as proof, I think I put up long enough with the ESXi box. It was not as bad as Xen but I got tired of not being able to get it to behave as it should. And when I could not do PCI passthrough -- I am not even saying the Netronome card, but every single PCI or PCIe card I had available and had no problems passing to a vm guest using KVM as the hypervisor -- it was time to move on. The writing on the wall came after almost a year I could not get an answer from VMWare.

The Plan

  1. While the ESXi server, vmhost2, is still running, export the guests in .ovf format to a safe location. Just to be on the safe side, write down somewhere the specs for each guest (memory, cpu, OS, which network it is using, etc).
  2. Build the new vmhost2 using Debian or CentOS as the base OS and kvm as the hypervisor. Some things to watch out for:
    • Setup Network trunk and bridges to replicate old setup.
    • Use the same IP as before since we are keeping the same hostname.
    • Setup the logical volume manager so I can move things around later on.
    • Configure ntp to use our internal server
    • Configure DNS to use our internal server
    • Accounts of users who need to access the vm host itself will be mounted through autofs. If that fails, can login to root using ssh keypair authentication. If that is down (say, network issues, switch to console).
    • Like in the old vmhost2, ISOs for the install images are available through NSF.
    • Add whatever kernel options we might need. Remember we are building this from scratch, not just dropping a prebuilt system like xenserver. Or ESXi.
  3. Import enough vm guests to validate the system. Might take the opportunity to do some guest housecleaning.
  4. Add any PCI/PCIe cards we want to passthrough.
  5. Import the rest of the vm guests.
  6. (Future:) set it up so it can move/load balance vm guests with vmhost, the other KVM host.
Note: I did not wipe the original hard drive; instead I just bought a 111.8GB (128GB in fake numbers) SSD to run the OS in. I did not get a second drive and make it a RAID for now since I plan on running the OS in that disk, which will be configured using Ansible so I can rebuilt it quickly. Any vm guest running in this vm host will either run from the fileserver (iSCSI) or in a local RAID setup of sorts. Or I might simply deploy ZFS and be done with it. With that said, I might run a few vm gusts from that drive to validate the system.
Note: This project will be broken down into many articles otherwise it will be long and boring to read on a single sitting (some of the steps are applicable to other applications besides going from ESXi to KVM). I will try to come back and add the links of those articles, treating this post as the index.

Monday, May 14, 2018

Converting a .ovf file to work on an older/different VMWare ESXi (maybe also player) setup

I will be using ESXi because that is what I have; I do not see why it would not work in Player or Workstation.

As you know, the way vmware likes to export/import vm guests is using a ovf format. So, let's say we are supposed to add a guest called strangeguest. We get it as a directory called strangeguest, which contains the disk (strangeguest-disk1.vmdk in our case), the config file strangeguest.ovf and a mysterious file called strangeguest.mf (.mf extension for Mysterious File?). When we try to import it we get an error message that complains we cannot import the OVF. A quick look indicates that strangeguest expects to be of SystemType vmx-12 or better:

admin@fileserver:/export/public/ISOs/strangeguest$ fgrep vmx- strangeguest.ovf         vmx-12
admin@fileserver:/export/public/ISOs/strangeguest$

Thing is our ESXi setup does not support vmx-12 guest in our ESXi as is a bit old and needs to be upgraded (which will be subject of another article). However, right now we need to make this work.

So we cheat.

We know the latest systemtype our ESXi support is vmx-10 by looking at the properties of the guests currently in place. So, how about if we tell strangeguest that it is vmx-10?

admin@fileserver:/export/public/ISOs/strangeguest$ sed -i -e 's/vmx-12/vmx-10/' strangeguest.ovf
admin@fileserver:/export/public/ISOs/strangeguest$ fgrep vmx- strangeguest.ovf         vmx-10
admin@fileserver:/export/public/ISOs/strangeguest$

So we try again and we get a different error (note to myself: get that error message). What did we do wrong? Well, do you remember the mysterious file? Let's see what is inside it:

admin@fileserver:/export/public/ISOs/strangeguest$ cat strangeguest.mf
SHA1(strangeguest.ovf)= 7b11b4aacead791f8aaf76e5ed3c2354349b3b20
SHA1(strangeguest-disk1.vmdk)= 9ccd4817ac2f943f7f29be970b76461850460d18
admin@fileserver:/export/public/ISOs/strangeguest$

So it has the checksum (as SHA1, which is a step about MD5 but still not to be used as it can be lied to, but I digress). Remember we edited strangeguest.ovf!

admin@fileserver:/export/public/ISOs/strangeguest$ sha1sum strangeguest.ovf
f14befc1e790b0043dd5f8e22fd8d601637997bd  strangeguest.ovf
admin@fileserver:/export/public/ISOs/strangeguest$

So, we need to update strangeguest.mf:

sed: -e expression #1, char 83: unterminated `s' command
admin@fileserver:/export/public/ISOs/strangeguest$ sed -i -e \
's/7b11b4aacead791f8aaf76e5ed3c2354349b3b20/f14befc1e790b0043dd5f8e22fd8d601637997bd/' \
strangeguest.mf
admin@fileserver:/export/public/ISOs/strangeguest$ !cat
cat strangeguest.mf
SHA1(strangeguest.ovf)= f14befc1e790b0043dd5f8e22fd8d601637997bd
SHA1(strangeguest-disk1.vmdk)= 9ccd4817ac2f943f7f29be970b76461850460d18
admin@fileserver:/export/public/ISOs/strangeguest$

And we should be rewarded with strangeguest being properly imported.

Saturday, January 30, 2016

Mounting user fileshare on boot2docker boot

For docker container development, and light using, I found boot2docker to be quite convenient. I have it to boot off its ISO and then mount a permanent drive for the containers (and to store config files).

/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
Whenever there is a new version, I shut the vm down, swap the ISOs (really just point the alias to the new file), and reboot. Nice and brainless.

I do not like to use the default account, docker, to do container development and running. Also, in a nice production environment you want to have other users running their containers. So, I created a user called ducker, which is a quick play on the default username. I also would prefer not having the user homedir in the drive where the containers are, which has been suggested before. You see, the way I see containers they are by design not important; blow them up if you feel like or wonder if they have been compromised. What matters is the data and the dockerfile required to rebuild the container. As a result, ducker has an account in the fileserver, which does its RAID and backup thingie as any good fileserver should. Now, if we want to have containers created and running from ducker's account when the server boots up, we need to have said account available.

So, the plan is that whenever the boot2docker server reboots, ducker will be there. And we then have a few issues to deal with. First, docker2file's ISO can't do automount. And second, usually when we shut down any user is lost because we are running of an ISO.

boot2docker's ISO is built so that we can provide some permanent stuff, which is why it mounts /dev/sda. But, that is not the only place we can mount things, nor the only way. In /opt/bootscript.sh we have the following interesting lines:

# Allow local bootsync.sh customisation
if [ -e /var/lib/boot2docker/bootsync.sh ]; then
    /bin/sh /var/lib/boot2docker/bootsync.sh
    echo "------------------- ran /var/lib/boot2docker/bootsync.sh"
fi

# Launch Docker
/etc/rc.d/docker

# Allow local HD customisation
if [ -e /var/lib/boot2docker/bootlocal.sh ]; then
    /bin/sh /var/lib/boot2docker/bootlocal.sh > /var/log/bootlocal.log 2>&1 &
    echo "------------------- ran /var/lib/boot2docker/bootlocal.sh"
fi

The files in /var/lib/boot2docker are in /dev/sda. Don't know which one to pick, but the second one does have a comforting Allow local HD customization message. So I will pick /var/lib/boot2docker/bootlocal.sh and add the following lines:

# Create local user, also creating the homedir
adduser -D -u 1003 ducker
# add user to docker group
adduser ducker docker
# Mount homedir
mount.nfs  fileserver.example.com:/export/home/ducker /home/ducker

After we create the file and reboot, we get

docker@boot2docker:~$ id ducker
uid=1003(ducker) gid=1003(ducker) groups=1003(ducker),100(docker)
docker@boot2docker:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    123.8M    772.8M  14% /
tmpfs                   498.1M         0    498.1M   0% /dev/shm
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda
cgroup                  498.1M         0    498.1M   0% /sys/fs/cgroup
df: /mnt/hgfs: Protocol error
fileserver.example.com:/home/ducker
                        295.3G    285.1G     10.0G  97% /home/ducker
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
docker@boot2docker:~$

User ducker can login because there is a public ssh key already in place. However, we did not do anything for that user's password, but there are a few ways to take care of that such as using /var/lib/boot2docker/bootlocal.sh to copy the hash into /etc/passwd. And, you really want to take care of that otherwise you will not be able to login.

Wednesday, February 04, 2015

Creating a git server in docker... with NFS and custom port and ssh key pair

I usually like to start my post describing what we will try to accomplish here, but I think I can't do any better than what the title states. So, let's say if I can come up with a convincing excuse. Well, all I can come up with right now is that I think it is wasteful to create an entire VM to run a distributed version control system. At least one that does not have helpful paperclips with eyes and other features requiring you to download crap. And, it is nice to know if the docker host (or cloud) takes a dump, we can bring this service back rather quickly. For this article we will use git; some other time we can talk about svn.

The git server I will be using is gitolite because it is reasonably simple and quick to manage and get going. What I really like on it is that the accounts for the users using git are not accounts in the host itself, so they cannot login to the machine hosting git. By default the git users login using

Since I am lazy, I will store the repositories in a NFS fileshare that is mounted into the container at runtime. We talked about how to do the mounting in a previous article.

Assumptions

  1. gitolite running off /home/git
  2. We will connect to the git server on port 2022. I chose that because I need port 22 for to ssh into the docker host. Yes, they are in different IPs (in my case completely different VLANs), but I am weird like that.
  3. We will use ssh key pair authentication. And will use a different key than the default one. Note you can authenticate against LDAP, but that would go against what I wrote in the name of this article.
  4. gitolite being run as user git
  5. I am running this on centos6.

Install

  1. I created a CNAME for the docker host, gitserver.example.com, so it looks pretty.
  2. In the NFS server, create a fileshare owned by the user git, which in this example has uid=1201.
  3. We will need to create a ssh key pair for the gitadmin. I created my pair by doing something like
    ssh-keygen -t rsa -C gitadmin -f ~/.ssh/gitadmin
    You will need to copy ~/.ssh/gitadmin.pub into the docker host by whatever means you desire.
  4. I create a directory in the docker host to put all the files (docker-entrypoint.sh and Dockerfile) related to this container. Here is the Dockerfile
    ############################################################
    # Dockerfile to build a gitolite git container image
    # Based on CentOS
    ############################################################
    
    # Set the base image to CentOS
    FROM centos:centos6
    
    # File Author / Maintainer
    MAINTAINER Mauricio Tavares "raubvogel@gmail.com"
    
    ################## BEGIN INSTALLATION ######################
    # We need epel
    RUN rpm -Uvhi http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.no
    arch.rpm && \
        sed -i -e 's/^enabled=1/enabled=0/' /etc/yum.repos.d/epel.repo
    
    # We need NFS, openssh, and git
    # And ssmtp (from Epel)
    RUN yum update -y && yum install -y \
            git \
            nfs-utils \
            openssh-server && \
        yum install -y ssmtp --enablerepo=epel
    
    # Configure NFS
    RUN sed -i -e '/^#Domain/a Domain = example.com' /etc/idmapd.conf
    
    ##################### INSTALLATION END #####################
    
    # Create git user
    RUN adduser -m -u 1201 git
    
    # Configure ssmtp
    
    # Configure sshd
    RUN sed -i -e 's/^#Port .*$/Port 2022/' \
               -e 's/^#PermitRootLogin .*$/PermitRootLogin no/' \
               /etc/ssh/sshd_config && \
        sed -i -e \
            's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.s
    o@g' \
             /etc/pam.d/sshd && \
        ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa && \
        ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
    
    # And a mountpoint for repositories
    # Note: can't NFS mount from dockerfile, so will do it in an entrypoint script
    RUN su - git -c 'mkdir repositories'
    
    
    # And now the git server
    # Gitolite admin: gitadmin (it is based on the name of the pub key file)
    RUN su - git -c 'mkdir bin' && \
        su - git -c 'git clone git://github.com/sitaramc/gitolite' && \
        su - git -c 'mkdir -m 0700 .ssh' && \
        su - git -c 'echo "ssh-rsa AAAAB3NzaC1yc2EAASLDAQCOOKIEQDehf5hxGq9//34yrsL
    [...]
    7CfSpbiP gitadmin" > .ssh/gitadmin.pub'
    # The rest will be configured in the entrypoint script
    
    # Put the entrypoint script somewhere we can find
    COPY docker-entrypoint.sh /entrypoint.sh
    ENTRYPOINT ["/entrypoint.sh"]
    
    EXPOSE 2022
    # Start service
    CMD ["/usr/sbin/sshd", "-D"]
    

    Where

    1. You will need to put the public key gitadmin.pub between the double quotes in the line beginning with su - git -c 'echo "ssh-rsa.
    2. I am running a lot of things in the Dockerfile as user git.
    3. The NFS setup was mentioned before, so I will not bother with it right now.
    4. I forgot to add the setup for ssmtp. I will think about that sometime later.
  5. The docker-entrypoint.sh file looks vaguely like this:
    #!/bin/sh
    set -e
    
    # Mount git's repositories
    mount.nfs4 fileserver.example.com:/git /home/git/repositories
    
    su - git -c 'gitolite/install -to $HOME/bin'
    # setup gitolite with yourself as the administrator
    su - git -c 'gitolite setup -pk .ssh/gitadmin.pub'
    
    # And we are out of here
    exec "$@"
  6. So far so good? Ok, so let's build the image. I will call it git.
    docker build -t git .
    If the last build message looks like
    Successfully built 6fb1ac15b47a
    chances are the build was successful and you can go to the next step. Otherwise, figure out what went boink.
  7. Now start the service. Remember you need to run in priviledge mode because of NFS. Since this is a test, I am calling the contained test-git.
    docker run --privileged=true -d -P -p 3306:3306 --name test-git git

Setup and Testing

  1. Let's start testing by seeing what repositories we can see as an admin:
    $ /usr/bin/ssh -i /home/raub/.ssh/gitadmin git@gitserver.example.com info
    hello gitadmin, this is git@docker running gitolite3 v3.6.2-12-g1c61d57 on git 1.7.1
    
     R W    gitolite-admin
     R W    testing
    
    
    FYI, testing is a repo everyone allowed to use the git server can play with. Think of it as a, as its name implies, test repo. Since that worked we can proceed to the next step.
  2. Now we should edit your .ssh/config file to access the gitadmin repository.
    cat >> ~/.ssh/config << EOF
    Host gitadmin
            Hostname        gitserver.example.com 
            User            git
            Port            2022
            identityfile    /home/raub/.ssh/gitadmin
            protocol        2
            compression     yes
    EOF
    Yes, you can ask about what to do if you have a script that needs to pull stuff out of a repo, and I will tell you to wait for the next installment. This article deals with getting it to work.
  3. Retrieving the admin repository is now much easier. So, instead of having to do something like
    git clone ssh://git@gitserver.example.com:[port]/gitolite-admin
    which would also require us to feed the key (or rename it as the default which IMHO is a bad idea), thanks to the previous step we can now lazily do
    git clone gituser:/gitolite-admin
  4. Repository config file is in gitolite-admin/conf/gitolite.conf
  5. Adding a new user (and perhaps a repository)
    1. Get user's ssh public key, say raub.pub. How did the user created the ssh key pair? Don't know, don't care.
    2. Copy raub.pub to gitolite-admin/keydir. NOTE:file must be named after username user will use to connect to the git server; it does not need to have anything to do with the user's normal/real username.
    3. Create a repository for the user. Let's give it a nice and snuggly name, like somethingawful
      cat >> conf/gitolite.conf << EOF
      
      repo somethingawful
          RW      = raub
      EOF
    4. Commit changes
      git add conf keydir
      git commit -m 'Added new user and config a repo'
      git push origin master
  6. Now, let's pretend we are the user (i.e. what the user should do/see). Which repos can you/user see? If this looks like a step we did before, it is. Just using a new user.
    $ ssh -i /home/raub/.ssh/raub git@gitserver.example.com info 
    hello raub, this is git@docker running gitolite3 v3.6.2-12-g1c61d57 on git 1.7.1
    
    
     R W    somethingawful
     R W    testing
    
    There is somethingswful!
  7. Let's grab somethingawful
    $ git clone gituser:/somethingawful
    Cloning into 'somethingawful'...
    warning: You appear to have cloned an empty repository.
    Checking connectivity... done.
  8. Edit, commit, and off you go

I will put all those guys in my github account later on. And shall sneakly update this post afterwards.

Sunday, February 01, 2015

Entrypoint scripts and data persistency: mounting a NFS fileshare inside a docker container

This is a quick article; you have been warned.

The traditional way to have persistent data in a docker container is to feed it with a volume from the docker host. You begin by locating a directory, be it in the local docker host drive, network mounted to the docker host, or a volume in another docker container. Then feed it to the container using the VOLUME statement defined in the Dockerfile or to command line. We all know that. But what some (very few we hope) of you might not have been aware of is the volume is only mounted/attached to the container when you tell docker to run the container, which might make running/configuring something that require files in those volumes a bit challenging.

At this point I would expect one of you to shout "Aha! But you can then use an entrypoint script to do all that monkeying that needs to happen after volume is mounted and before whatever service this container provides starts!" And you would be quite right! Here's a quick example in case we lost somebody: if our Dockerfile ended up like this:

# Put the entrypoint script somewhere we can find
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

EXPOSE 22
# Start service
CMD ["/usr/sbin/sshd", "-D"]

we would have created a file called entrypoint.sh in the root path of the container, which is run just after the volume is created and before the, in this example, sshd service is started. Said entrypoint file could do, I don't know:

#!/bin/sh
set -e

# Do something in /home, which is the mounted volume
/do-something-at-home.sh

# And we  are done here
exec "$@"

point is that do-something-at-home.sh is called only after the persistent volume is mounted into the container.

What if you want to mount a network fileshare straight into the container? Before I have time to answer that, someone from the audience will jump and state "Why would you want to do that? You can mount the network fileshare in the docker host and then use VOLUME just like Wicked Witch of the West intended!" What if I don't want to do that? What if I can't for whatever reason mount a network fileshare on the docker host?

The answer is that it is completely doable; you just have to ask the entrypoint script to do the mounting for you. To show what I mean I will use the example of mounting a NFSv4 fileshare that is supposed to be used by some website. So we modify our little entrypoint script a bit

#!/bin/sh
set -e

# /export/www was created in the Dockerfile
mount.nfs4 fileserver.example.com:/export/www /www/some-website 

# And we are done here
exec "$@"

Since we are using NFSv4, chances are you might need to add something like

RUN sed -i -e '/^#Domain/a Domain = example.com' /etc/idmapd.conf

to your dockerfile in addition to telling it to get the nfs client packages, but the general idea should apply for, say, SMB or ZFS or whatever other network filesystem that you fancy: let your entrypoint script do the heavy lifting!

Monday, April 28, 2014

Booting a ESXi VM from a .iso in a NFS share

First time I created a VM in Vmware's ESXi, I placed the .iso containing the install image of the operating system in the host I run vsphere client from. And then told the VM where to get that. Some info on this procedure can be fount at http://community.spiceworks.com/topic/332810-create-vm-from-iso-without-uploading-to-datastore. At first glance, it seemed to be quick and easy; I would even add for a small deployment it is quite convenient. However, but it was just cumbersome (too many hoops to make that work) and slow (it expects the vsphere box to be in a fast and reliable connection to the ESXi one, which might not be the case). It really makes sense to have the images either in the server itself or mounted on the server (fileshare). I was not looking forward to having the images in the server itself because:

  • It would probably require me to download them somewhere else and then upload to the server. That does not sound like a dealbreaker to most since if I can ssh into my ESXi, which I do, I can scp the files into it. Well, I do not believe on having a single device/program/app/thingie that does it all; you see, I do like the Rule of Simplicity from the Unix Philosophy. And that dovetails to the next item:
  • I have a perfectly good fileserver (ok, a Synology NAS box) thank you verymuch. Many of my VMs already NFS mount shares from it, or just have their entire disk in an iSCSI LUN from said NAS.
  • I might want to use those images with my other vm host, which runs KVM.

What I want is maybe a NFS share that can be mounted somewhere where I can download the .iso files to and then have it available read-only to the vm hosts (ESXi, KVM/libvirt, Vbox). Let's see if we can make that happen, shall we?

Setup

We first need to begin with the fileshare itself. It is being exported read-only as a NFSv4 fileshare from the NAS; we won't go over how to do that in this discussion. Since I could not find showmount or even mount in ESXi, let's assume I know what I am doing and believe the share I want is fileserver.example.com:/public (I cheated and verified in a Linux box). We can add that to vmhost2 using the vsphere client:

Using the ISO

So now we did all this boring work, let's see if we can boot using a, well, boot .iso from the NFS share. So, On the vsphere client

  • Select the vmhost. In my case that is vmhost2.
    1. Select Configuration->Storage in the Hardware panel.
    2. Click Datastores and click Add Storage.

    3. Select Network File System and click Next.


    4. Enter

      • server name: fileserver.example.com
      • mount point folder name: /export/public (yep NFS3)
      • [x] Mount NFS read only
      • datastore name: public

    And that should result in a new datastore entry called public.
  • Select the vm client, which is called devcentos.
    1. select the vm in question, devcentos:
    2. Edit virtual machine properties->CD/DVD->Device Type->Datastore ISO File
    3. Hit Browse
    4. Datastores->public->ISOs->CentOS.iso
    5. Boot to BIOS. For some reason I have to manually select which device to boot. Even though most of the time the virtual hard drive is completely virgin, the bios does not failover to the ISO, Maybe that has been solved by now, but just learn this step... just in case.
    6. Turn CD player on on boot

References

https://communities.vmware.com/thread/456682

Tuesday, June 18, 2013

Resizing a shared partition in a Synology DiskStation

I bought one of those devices, specifically the DS212j, to use as network storage (NAS for you alphabet soup lovers) for my home. I slapped two green 2TB Western Digital drives in it, set them up as a raid 1, created a 100GB (which probably means using the fake gigabyte, not the power-of-two one) NFS share partition for users, and off I went. Now, since I used its default clickety-click interface (it's pronounced web-based), when I ssh into the device (I am a bit of a command-line (CLI; I did not forget you) kinda bloke), I found it is using the standard Linux lvm (ok, this one I actually use) and named the logical volume I created as volume1, formatted as ext4 mounted as /volume_1. Even though I personally like to call my volumes after their function and try to avoid mounting stuff on the root, I can live with that. But, the point is ext4 and lvm involved. i.e. sane stuff. I like that. It also means that even though a lot of those devices use a scaled down version of linux, this one is not as scaled down as you would be led to believe.

This morning I received a email from the device. Since I want to make this post look long and important, I will post it here in glorious quadrovision:

Dear user,

The available space of volume 1 on spindizzy is running out; please delete some files to free space.

Total capacity: 98.43 GB
Available capacity: 0.98 GB (1.00%)

Sincerely,
Synology DiskStation

Hmmm, that sounds kinda bad. What should I do? Well, I am lazy. Do you remember when I mentioned the sane stuff Synology is using in this device? Let's do some exploring since I still need to fill more space:

spindizzy> pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/md2   vg1  lvm2 a-   1.81T 1.72T
spindizzy> vgs
  VG   #PV #LV #SN Attr   VSize VFree
  vg1    1   2   0 wz--n- 1.81T 1.72T
spindizzy> lvs
  LV                    VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  syno_vg_reserved_area vg1  -wi-a-  12.00M                                      
  volume_1              vg1  -wi-ao 100.00G                                      
spindizzy> 

So, the entire raid (minus whatever the device needs to do its thing) is a single physical volume which is allocated as a single volume group (cleverly called vg1, inside which is our logical volume. And, as the email said and df -h can show,

/dev/vg1/volume_1        98.4G     97.4G    916.5M  99% /volume1

rather full. Well, how about if we take care of that lvm-style?

spindizzy> lvextend -L +100G /dev/vg1/volume1
  Logical volume volume1 not found in volume group vg1
spindizzy> lvextend -L +100G /dev/vg1/volume_1
  Extending logical volume volume_1 to 200.00 GB
  Logical volume volume_1 successfully resized
spindizzy> resize2fs /dev/vg1/volume_1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg1/volume_1 is mounted on /volume1; on-line resizing required
old desc_blocks = 7, new_desc_blocks = 13
Performing an on-line resize of /dev/vg1/volume_1 to 52428800 (4k) blocks.
The filesystem on /dev/vg1/volume_1 is now 52428800 blocks long.

spindizzy> df -h              
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  2.3G    425.6M      1.8G  19% /
/tmp                    121.8M    264.0K    121.5M   0% /tmp
/dev/vg1/volume_1       196.9G     97.4G     99.2G  50% /volume1
spindizzy> 

What I did was to add an extra 100G, effectively doubling its size, to the logical volume volume_1. And all that was done live. Exciting huh? For those of you who do not dabble with lvm a lot, one of its nicest features is that you can increase the size of a logical volume life, without needing to unmount it first. All you need is to have some free space in the volume group (the VFree column). Going the other way around is a bit more challenging, for you need to umount the volume first, but can be done. I will later write an article on monkeying with lvm, I promise (remind me!).

Some of you might be like big deal, you could probably have done that using the web interface, just like in many other equivalent devices. What's so special about the a Linux-based network storage thingie? Well, the fact they can (either from factory or by adding the required packages) use lvm means I do not need to recreate a partition whenever I need more space, which was a problem with other devices I had. And, I can not only take care of that through the command line instead of needing a web browser, but also I could write a script to do that for me. Are they the only ones doing it? I doubt, but it reminds me why when shopping for a NAS I look for one that runs Linux in some shape or form.

Thursday, October 05, 2006

Terrible tales of NIS, NFS, and automounting - II

Maps, Matey!

On the last installment we began to setup the NIS server for Cannelloni Inc, a performance kitcar manufacturing company. Now that we have the domain name defined and a home for the NIS maps we are going to use, how about creating some maps? We will go over that by first working on the NIS server, by creating and exporting the maps, and then reading them in the client.

Server setup

Ok, we need to create the NIS maps, but what are those maps anyway? Well, maps are the files NIS uses to keep the information it needs and passes around. Think of them as plain text databases where each entry is a pair (as in first column is defined by the remaining columns). I guess the best way to explain them is to show how they compare to some of the files used by Linux/Unix:

MapsEquivalent unix fileComments
hosts.byname, hosts.byaddr/etc/hostsMaps IP addresses to host names
passwd.byname, passwd.byuid/etc/passwdMaps UIDs to usernames (and passwords)
group.byname, group.bygid/etc/groupMaps Group IDs to group names

So, our /etc/src/auto.home would look something like this:

bob            -nosuid,intr    obelix.cannelloni.com:/export/home/bob
thetick        -nosuid,intr    obelix.cannelloni.com:/export/home/thetick
heathcliff     -nosuid,intr    obelix.cannelloni.com:/export/home/heathcliff
mccoy          -nosuid,intr    obelix.cannelloni.com:/export/home/mccoy

and so on.

The netgroup map, which we chose (when we edited the /var/yp/Makefile, remember?) to be stored in /var/yp/, is like the groups file but can be used to group not only users but also any combination of users, domains, and hosts. We have two printers, falbala and bonemine, so we create a group for them which will be called printers. So far, our /var/yp/netgroup file looks like this:

openwheel (assurancetourix,,) (alambix,,) (caiousbonus,,) (petisuix,,)
printers  (falbala,,) (bonemine,,)

Remember that once you ad a Linux box as a NIS client you should run /usr/sbin/gdm-restart so the login window knows of the changes and maps. For some reason, ssh and the text-based login screen have no problems being updated, but gdm does. Perhaps it is buffering the user data.

For Linux, create an /etc/exports file that looks like this:

/home/sunpci/linux 192.168.0.11(rw,no_root_squash)

For Solaris, set up your /etc/dfs/dfstab this way:

share -F nfs -o rw=@192.168.0.11/32,root=@192.168.0.11/32 /home/sunpci/linux

Once you have finished with /etc/exportfs, you need to make the changes take place by typing

Linux:

# exportfs -a

Solaris:

# share all

Tuesday, September 19, 2006

Terrible tales of NIS, NFS, and automounting

NFS and NIS have been around for a while, way before someone decided to network two Windows boxes. They have a lot of neat features.

The Network Information Service (NIS) is a directory service protocol created by Sun. It is not as elegant as, say, LDAP with Kerberos, but can get the work done if due care is taken to keep it as safe as possible.

I am going to present the steps necessary to setup a NFS/NIS system that would server a bunch of users and the unix boxes they connect to. Originally I was going to make this fit one single post but I realized that it (hopefully) would be easier if I broke down into sessions and dealt with just one aspect at a time. I will also create a fake company so it has that professional look to it. Sounds like a plan? Great! Let's get busy then. In this example, we work at Cannelloni LLC, a performance kitcar manufacturing company. It is primarily a Linux shop all the way to the desktops. Recently it has grown enough to need a centralized directory service and file sharing systems. Since we are talking about NIS, Cannelloni chose to use NIS for now. Later on the show we will talk about NIS limitations.

Layout

I would start this mess with the main/master NIS server because I want to have the authentication side of the business out of the way. First of all, we need a NIS domain name. This really needs not to have anything to do with the DNS domain, but should make sense to you. Think of it as a logical group or unit. You talked to your boss and after a few beers it was decided to divide the current network mess into the following groups:

  • Management
  • Office
  • Accounting
  • HR
  • Development
  • Production

Probably you could have come up with better names, but that is what you get by trying to work drunk. You can always change them later. If nothing else, just to piss off accounting. Since doing every single group would bore me to death, we will assume that development decides to take the lead. If it works there, the same concept will be generalized across the entire company. So, development choses idefix (it is not very powerful but that really does not matter) as its NIS server; another machine, obelix, which has a nice hotswappable RAID 5 will be the fileserver which will export fileshares through NFS.

First, we start by finding out a bit about the company's network and which part of it belongs to development. Careful research indicates the entire company is behind a router, so it has only a handful of public IPs (webserver, mail, and so on) while the LAN uses the private network 192.168.10.0 (it is a small company). The IPs assigned to development are 192.168.10.100 to 192.168.10.120, and all the IPs have not been assigned yet. This is very important to know because we can limit which machines can see the NIS maps. How do we do that? Well, we are getting a bit ahead; let's first create a place to save all the configuration files we will be creating.

We Need a Home

NIS stores a lot of important stuff in general in /var/yp Go take a look at it; it should look kinda like this:

dalek@idefix-> ls /var/yp
binding  Makefile  nicknames
dalek@idefix->

Kinda boring I know but we are just starting with it. The Makefile you see there is used to generate the NIS maps. By default it will use /etc/passwd, /etc/shadow, /etc/group and a lot of other files that are in /etc. I honestly do not like that. /etc for me is kinda of an important directory and I would rather have its contents not being passwd all over the universe. Instead, I prefer to feed NIS my own passwd, group, and any other map I want to share. Not only that makes it a bit safer but also easier to manage/move around as everything is contained in a single location you can simply tar and move to the next machine. So, we need to do some editing in Makefile. So, I create two directories: /var/yp/src and /var/yp/src/pw. Then, I edit the Makefile as follows (you will need to search within that file for those definitions):

  • Linux
    # YPSRCDIR = /etc
    YPSRCDIR = /var/yp/src
    # YPPWDDIR = /etc
    YPPWDDIR = /var/yp/src/pwd
    
  • Solaris
    # DIR =/etc
    DIR =/var/yp/src
    # PWDIR =/etc
    PWDIR =/var/yp/src/pwd

The next step is to create those directories and make sure they can only be read/accessed by root, specially /var/yp/src/pwd as it will host the password file that will be shared through NIS. Next we will create a file called securenets in /var/yp which will tell which machines can see these maps:

dalek@idefix-> cat /var/yp/securenets
# /var/yp/securenets
# Restrict access to the NIS maps to the machines defined in this file
#
# allow connections from local host -- necessary
host 127.0.0.1
# same as 255.255.255.255 127.0.0.1
#
# allow connections from any host on the development network
host 192.168.10.100  # asterix
host 192.168.10.101  # obelix
host 192.168.10.102  # idefix
host 192.168.10.103  # panoramix
host 192.168.10.104  # abraracourcix
host 192.168.10.105  # bonemine
host 192.168.10.106  # agecanonix
host 192.168.10.107  # assurancetourix
host 192.168.10.108  # cetautomatix
host 192.168.10.109  # ordralfabetix
host 192.168.10.110  # lelosubmarine
host 192.168.10.111  # falbala
host 192.168.10.112  # aplusbegalix
host 192.168.10.113  # amerix
host 192.168.10.114  # caiusbonus
host 192.168.10.115  # caiusmalosinus
host 192.168.10.115  # tragicomix
host 192.168.10.116  # alambix
host 192.168.10.117  # petisuix
host 192.168.10.118  # jolitorax
host 192.168.10.119  # beaufix
host 192.168.10.120  # barberouge
dalek@idefix->

Do notice we chose to specify each host in /var/yp/securenets. Since the number of machines in this list is not a power of 2, we could not use IP of the first machine and a carefully chosen subnet mask to cover them all. Also, spelling out every machine we plan on using allows us to later on comment out the ones we do not need.

A Domain by Any Name

Now that we have that taken care of, we need to come up with a name for our NIS domain. Since this name does not need to be remotely related to our DNS domain, we will call it development as it is the NIS domain for the development group. I know, I know, I am very original...

Ok, you ask, now how to make NIS know we chose a NIS domain name? Well, the domain name is stored in /etc/defaultdomain (for solaris) or /etc/domainname (for Linux). If you write the domain name you want to use in that file (i.e. there is just one single line in the file and all it has is your domain name, in this case, development) and reboot, idefix will then know the name of the domain. You can check it by using the command domainname as follows:

dalek@idefix>cat /etc/defaultdomain
development
dalek@idefix>

Now, before you go rebooting the machine, let's see if we can change its domain name without rebooting, shall we? In both Solaris and Linux, you can set the runtime domain name to development by saying

dalek@idefix>domainname development
dalek@idefix>

How about checking it?

dalek@idefix>domainname
development
dalek@idefix>

Of course, since we had already defined the domain name in /etc/defaultdomain (Solaris) or /etc/domainname, we could have said

  • Solaris
    dalek@idefix>domainname `cat /etc/defaultdomain`
  • Linux
    dalek@idefix>domainname `cat /etc/domainname`

Do note the back quotes; they are rather important.

Ok, I am going to take a break for now. Next time we will talk about the wonderful world of maps. Stay tuned!