Relative paths and roles in Ansible

Here is something I finally understood in Ansible that has been annoying the hell out of me for a while: relative paths. The idea of relative paths is so if you have to tell Ansible to grab a file -- like some config file or even an image or certificate -- and put it somehwere else you do not have to give the entire source path.

Imagine if you wrote your playbook in your Linux desktop which has a task that grabs the file /home/bob/bobs-development-stuff/server.stuff/ansible/roles/webservers/files/main-logo.png and put it somewhere in the target machine. That is a mouthfull of a path. But then since you are clever you have this playbook in an internal git repository and then check it out in your Mac laptop to do some work. Thing is the path now is /Users/bobremote/ansible/roles/webservers/files/main-logo.png and now your task ain't working no more. And that is the short version for why absolute paths in your ansible box are not a great idea.

So we use relative paths instead and the main question of the day is what are those paths relative to?

Do you remember when we talked about creating and uploading ssh key pairs? Well, there was a bit of a fudge right in the key: line

- name: Copy the public key to the user's authorized_keys
    # It also creates the dir as needed
    authorized_key:
      user: "{{user_name}}"
      key: '{{ lookup("file", "roles/files/" ~ user_name ~ ".ssh.pub") }}'
      state: present

On a second thought, it is really not a fudge but it hints on the answer to our question.All we are asking is for the task to grab a file inside the roles/files directory. The original layout for this playbook looks like this

ansible
 hosts
 ansible.cfg
 site.yml
 roles
  defaults
  files
   moose.ssh
   moose.ssh.pub
  handlers 
   main.yml
  meta
  tasks 
   main.yml
  template
  vars

So we really only have just one playbook inside the ansible directory. When we write in the roles/tasks/main.yml tasklist file to grab the .ssh.pub< file, it knows exactly how to get it. With time, we became clever and would like to plan ahead so decided to break the roles into the ones common to every server and the ones that build up on the common one and then do the specialization stuff, say website or database server. So, what we really want is to have a layout like this (inside the same old ansible dir)

site.yml
web.yml
group_vars
roles
 common
   defaults
   files
      epel.repo
      iptables-save
      moose.ssh
      moose.ssh.pub
   handlers 
      main.yml
   meta
   tasks 
     main.yml
   template
   vars
 website
   defaults
   files
   handlers
   meta
   tasks
     main.yml
   template
   vars
 db
   defaults
   files
   handlers
   meta
   tasks
     main.yml
   template
   vars

The playbook that copies the ssh key is in the common role, specifically roles/common/tasks/main.yml (yes I could break it out some more but let's not confuse things just yet and hurt my brain). And we expect if we write our function as

- name: Copy the public key to the user's authorized_keys
    # It also creates the dir as needed
    authorized_key:
      user: "{{user_name}}"
      key: '{{ lookup("file", "./files/" ~ user_name ~ ".ssh.pub") }}'
      state: present

our playbook to create roles/common/files/moose.ssh.pub and then upload to whatever machine we are building. Instead it gives us the following friendly message:

fatal: [ansibletest -> localhost]: FAILED! => {"changed": true, "cmd": "ssh-
keygen -f ./files/\"moose\".ssh -t rsa -b 4096 -N \"\" -C \"moose\"", "delta": "0:
00:01.056746", "end": "2016-11-30 15:55:19.618226", "failed": true, "rc": 1, "s
tart": "2016-11-30 15:55:18.561480", "stderr": "Saving key \"./files/moose.ssh\
" failed: No such file or directory", "stdout": "Generating public/private rsa
 key pair.", "stdout_lines": ["Generating public/private rsa key pair."], "warni
ngs": []}

Clearly we have a bit of a path problem. Don't believe me? Let's ask where the playbook thinks it is running from

- name: Find out where I am locally speaking
    local_action: shell pwd > /tmp/here

All we are doing here is getting the path this is being executed from and saving in a file so we can see it. And see it we shall:

raub@desktop:~/dev/ansible$ cat /tmp/here
/home/raub/dev/ansible
raub@desktop:~/dev/ansible$ 

My guess is that if we use relative paths, they are all from where we ran the playbook from, /home/raub/dev/ansible in my case. Which really sucks! Now, there is an environmental variable in Ansible called role_path, which I found being mentioned in the docs which should show the path to the role we are currently running. Let's try it out

- name: Find out where I am locally speaking
    debug: msg="Role path is supposed to be {{ role_path }} "

which gives

TASK [website : Find out where I am locally speaking] ******************************
ok: [ansibletest] => {
    "msg": "Role path is supposed to be /home/raub/dev/ansible/roles/website "
}

So, the solution is to rewrite our little function as

- name: Copy the public key to the user's authorized_keys
    # It also creates the dir as needed
    authorized_key:
      user: "{{user_name}}"
      key: '{{ lookup("file", "{{ role_path }}/files/" ~ user_name ~ ".ssh.pub") }}'
      state: present

so the path becomes /home/raub/dev/ansible/roles/website/files/moose.ssh.pub

Creating a fixed length blank(spaces) string and padding others with spaces in powershell

Powershell has clever ways to create an empty string or string array that can then be increased in size. And then I need a fixed length empty string. Why? I need to create a file where each line is composed by fields which are position dependent. So I need to pad between the fields with blank spaces.

After searching for quite a while I found Ethan's reply on the bottom of thread on limiting a string to N characters. And it was so obvious I had to smack me on the head: if you want to create a string called moo that contains nothing but 24 spaces, you can do it by typing

PS C:\> $moo = " "*24
PS C:\> $moo.length
PS C:\> 24

So let's do something useful with that then!

So the plan is to write a function that given the string that will be used in a given field and the field size, it will return a properly padded string. Now, I think we should also create a second function whose only goal in life is to create a string with N blank spaces. And then the other function can call this. Reason I propose this is in case we need to provide some extra blank spaces because the format of this file might be, well, rather silly. Planning ahead is always a smart move. So, we start with our blank string generating function, which will look very similar to the $moo example above:

function BlankString($size)
{
   return " "*$size
}

If we want to test that we can modify the moo example

$moo = BlankString(24)
$moo.length

Which should return 24 as before. So far so good. Now let's create a function that will return a properly padded field. Something like

function PadString($name, $stringSize)
{
   if($name.length -lt $stringSize)
   {
      $name += BlankString ($stringSize - $name.length)
   }
   return $name
}

looks like a good starting place. Let's try it out:

$noo = PadString "My Head hurts" 48
DebugLine "[$noo]"
$noo.length

Results in

[My Head hurts                                   ]
48

If you are curious, my DebugLine function looks like this

function DebugLine($OutputMessage)
{
   Write-Host $OutputMessage -foregroundcolor red -backgroundcolor yellow
}

The [ and ] are there just so we can see where the string, blank spaces included, begins and ends. If you do not like those characters, change them to something more remarkable, say ---> and <---

Now, what if the string we are trying to put in this new string is bigger than it can handle? This would be the opposite of padding the new string: we need to chop it. So, we modify our PadString function a bit:

function PadString($name, $stringSize)
{
   if($name.length -lt $stringSize)
   {
      $name += BlankString ($stringSize - $name.length)
   }
   else
   {
      $name = $name.Substring(0, $stringSize)
   }
   return $name
}

So, if we redo the same test we did before, but this time we make our resulting string shorter,

$noo = PadString "My Head hurts" 8
DebugLine "[$noo]"
$noo.length

the result is now truncated

[My Head ]
8

And we can still create weird blank spaces using BlankString as needed to meet the file format requirments.

Creating a user with random password using Ansible

So I need to create an user in a machine so I can then have a script that will log into this machine and backup its database. Which database is that, you might ask? For this discussion it does not matter besides that it is running in a Linux box. But, if you want a more specific example, we could be backing up a sqlite database since we talked about how to do the deed before.

Anyway, the plan is to have the script ssh into the database server and grab the backup. Since we are using ssh in the script, we might as well use key pair authentication. Now I have learned that by default if you create a user and do not assign it a password, you will not be able to login as said user using key pair authentication. You can turn that off but I would rather not. Instead, since I am creating said user programmatically, I can give it a long random password.

Now that we have a plan, let's see how to do it in an ansible playbook. I will present only the relevant bits since I do not know how you do your playbooks. So, we could create a user using something like

- name: Create user_name
    user:
      name={{user_name}}
      groups={{user_groups}}
      shell=/bin/bash
      state=present
      append=yes

which would create user user_name who will also belong to the groups defined in user_groups, where user_name and user_groups are variables defined somewhere earlier in the show. And this would create a user without a password, which would do us no good. Nor would us make the playbook stop and ask us to enter a password. We said earlier we are going to create a random password, so let's see if we can make something random enough for our needs.

I plan on generating this random password in the machine we are running ansible on, not the target machine. One of the reasons is that I want to use the Linux command mkpassword to create the password hash (note it is being called using the shell command. So, I will use a local_action to do the deed. For instance, let's say I want the password to be pickles and encoded using SHA-512 hash (mild encryption). I could accomplish it by writing

- name: generate random password for new user
    local_action: shell  mkpasswd --method=SHA-512 pickles
    register: user_pw

This would create a hash, say

$6$rIcep9bGJTUpE$s8pka1dX6gWfyeNfi8YLaqrg/85tgtpJv809AUmO2jHhMQbSUnuNSloJSa6EmOQS02Ek4mvpiIu2DAvA9W0UL0

and assign it to the variable user_pw. This of course has to be done before the user is created. To use it with our new user, we can then modify our little user creation function to something like this:

- name: Create user_name
    user:
      name={{user_name}}
      groups={{user_groups}}
      shell=/bin/bash
      state=present
      append=yes
      update_password=on_create
      password="{{user_pw.stdout}}"

In the last line we are feeding the value of user_pw, user_pw.stdout, to password. But why can't I just feed user_pw? Here's an exercise to you: tell your playbook just print user_pw. Doesn't it look very object-like?

If you run your playbook and all went well, go to the target machine and check if there is a password associated with the user in /etc/shadow. If the user was already created, you will need to delete user and let ansible recreate it.

So we have so far created a way to create a password hash and then create a new user with that password. The last step we need is to make the password random. Here is what I am proposing: how about if we use date since epoch in seconds as our password and then mangle it a bit? Here is a simple mangling example:

raub@desktop:~$ date +%s
1479095572
raub@desktop:~$ date +%s | sha384sum
3d47137f4cd6bfb638deecc661b4e0ae9545ab2454b20eac51b6992807b3f518a49be9ef3b72a5abdc9167e32acc2473  -
raub@desktop:~$ date +%s | sha384sum | md5sum
65db8ecf178a505ea17e9b53cc5adf31  -
raub@desktop:~$ date +%s | sha384sum | md5sum | cut -f 1 -d ' '
b8a49ccaa4721877cf39e510c7ac3622
raub@desktop:~$

Which gives b8a49ccaa4721877cf39e510c7ac3622 as the output, which should be long enough to fulfill our needs. Of course if you run it again, it will spit out a different result, which is what we want? Perfectly random? Not by a long shot, but it is long enough for our needs. Remember: there is nothing saying you have to use the above. Hav efun creating your own function!

So, let's apply that to our little password generating function:

- name: generate random password for new user
    local_action: shell  mkpasswd --method=SHA-512 $(date +%s | sha384sum | md5sum | cut -f 1 -d ' ')
    register: user_pw

And we should be good to go. Here is how the final version should look like in a playbook:

- name: generate random password for new user
    local_action: shell  mkpasswd --method=SHA-512 $(date +%s | sha384sum | md5sum | cut -f 1 -d ' ')
    register: user_pw
# Something might happen here before we create user
- name: Create user_name
    user:
      name={{user_name}}
      groups={{user_groups}}
      shell=/bin/bash
      state=present
      append=yes
      update_password=on_create
      password="{{user_pw.stdout}}"

Now we have an user, we can then create the ssh key pair we talked about in an earlier article. Of course we might edit the ./ssh/authorized_keys file to restrict what that key can do.

Yum Manually and multiple repos

Quick post (I hope) about something I learned today that has a bit of a Captain Obvious taste to it. But I thought some people might find that amusing... even if it is at my expense.

Like many who use Red Hat products (CentOS comes to mind) and derived distros, I use repos outside the official ones. Because those repos tend to have newer versions of some packages, I try to be careful about only upgrading the packages that are required by the package I added said repo to my list. Short answer is the Law of Unintended Consequences. Long version is that I expect that people building the official packages being quite careful about compatibility and security. So, I should only reach out to the different repos after I found out the official packages do not do what I need.

What I have been doing, and I do not claim it is the best solution, is to install and then disable the non-official repos so if I want something from them I have to specifically ask for it. So, if I want to use the remi repo, I would first disable it

sed -i -e 's/^enabled=1/enabled=0/' /etc/yum.repos.d/remi.repo

and then specifically ask for it to install, say, php

yum install php --enablerepo=remi

which would install the latest PHP version that remi has. On a side note, if you had to install php 5.6 from remi, you would use remi-php56. But, what about upgrading them? After all, yum check-update and yum update by default will not check the disabled repos even if you have packages installed from them. So, you have to use --enablerepo. Now, what I learned today is that you can just list all the repos you are using by separating them with commas.

Let me show you in action: at first we think there are no updates:

raub@pickles ~]$ sudo yum check-update
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-
              : manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
[raub@pickles ~]$

Now, let's list all the repos we have been using with this machine and see what it tells us:

raub@pickles ~]$ sudo yum check-update --enablerepo=remi-php56,epel,secu
rity_shibboleth
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-
              : manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
security_shibboleth                                      | 1.2 kB     00:00
security_shibboleth/primary                                |  15 kB   00:00
security_shibboleth                                                       96/96

libcurl-openssl.x86_64                7.51.0-2.1             security_shibboleth
opensaml-schemas.x86_64               2.6.0-1.1              security_shibboleth
shibboleth.x86_64                     2.6.0-2.1              security_shibboleth
xmltooling-schemas.x86_64             1.6.0-1.1              security_shibboleth
[raub@pickles ~]$

As you can see, we do need to update shibboleth and its dependencies! And update we shall:

[raub@pickles ~]$ sudo yum update --enablerepo=remi-php56,epel,security_shibboleth
[...]
[raub@pickles ~]$ 

Kinda neat, eh?