Sunday, March 29, 2015

Building a single/simple-minded website in Python

A lot of scripts I end up writing begin as solutions to a problem or a way for me to be lazy and maybe learn something in the process; today's topic is no different. As some of you know, when you are deploying new Linux boxes from scratch, you can be lazy and use stuff like kickstart (centos/RedHat) and preseed (ubuntu, Debian) to do the initial install before handing over to something like Puppet, Ansible, or Salt (to name a few). Thing is you need to feed the preseed/kickstart file somehow.

One way to do it is to build a .iso or setup network install (you know the drill: PXE boot, DHCP, and so on). A very nice example (which I myself have used before) is shown in the CentOS docs. Ubuntu/Debian have something very similar. Now, the step that is relevant to this blog entry is the one in which the preseed/kickstart file is passed to the new host you are building. We can make it available in a web server, and tell the new machine where it is. Just to let you know, if you are using docker, the concept is the same. I know I am going really quickly through this because all I am doing right now is explaining the need that caused me to write this.

So, we established we need a web server to feed the preseed/kickstart file. But, there are times we do not need a full fledged website, all we want it to do is to offer one single file. And once the file is provided and the host is created, the website can go away. I imagine you are smelling some kind of automated host building script that automagically creates the web server it needs in the process. And you are right, which is why I wanted something with as little footprint as I can get away with. In other words, I would love to have a web server that completely runs off a single script.

To do the deed, I chose to use Python. Besides the fact I suck at ruby, I bumped into an example of a simple python-based webserver using something called BaseHttpServer. I modified it a bit and came up with the following script to serve a preseed.cfg file:

#! /usr/bin/env python
'''
Simple dumb webserver to serve a file.
It will try to serve a file called preseed.cfg, located in the directory
program was called, on localhost:8000

The idea is you can ask for any file you want, and will get what we
give to you.

Shamelessly based on https://wiki.python.org/moin/BaseHttpServer
'''
import time
import BaseHTTPServer

HOST_NAME = '' # Accept requests on all interfaces
PORT_NUMBER = 8000
FILE = 'preseed.cfg'

def read_file(filename):
    '''
    Read in (text) file and return it as a string
    '''
    file = open(filename, "r")
    return file.read()

class MyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_GET(self):
        file = read_file(FILE)
        self.send_response(200)
        self.send_header()
        self.end_headers()
        self.wfile.write(file)

if __name__ == '__main__':
    server_class = BaseHTTPServer.HTTPServer
    httpd = server_class((HOST_NAME, PORT_NUMBER), MyHandler)
    print time.asctime(), "Server Starts - %s:%s" % (HOST_NAME, PORT_NUMBER)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    print time.asctime(), "Server Stops - %s:%s" % (HOST_NAME, PORT_NUMBER)

Here's a quick rundown on what it does:

  1. Import the two libraries we need. Note they are default libraries every python install should have. This is not supposed to be a fancy or remotely clever script.
  2. Define some constants.
    HOST_NAME = '' # Accept requests on all interfaces
    PORT_NUMBER = 8000
    FILE = 'preseed.cfg'
    In the script I showed how to take the lazy way and make the service listen on all interfaces. In fact, when you run the above script, it should show in netstat as
    raub@desktop:~$ netstat -apn|grep 8000
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    tcp   0   0 0.0.0.0:8000    0.0.0.0:*     LISTEN     11539/python
    raub@desktop:~$
    As you know, setting the IP to 0.0.0.0 means everyone + the cat, which is why you can see it is listening on every interface in this machine, localhost included, on port 8000.
  3. The MyHandler class, which handles all the http requests, only cares about processing GET events. And, when it sees one, all it does is spits out the file preseed.cfg as a Content-type: text/plain.
  4. When you run the script, it should show something like
    Mon Mar 23 09:17:50 2015 Server Starts - :8000
    when it starts. And then when someone actually does hit the server, it would show a message like
    192.168.5.10 - - [23/Mar/2015 09:28:13] "GET / HTTP/1.0" 200 -
    which would indicate that 192.168.5.10 connected to our little webserver and, as a result, got the preseed file. If you use wget,
    wget the-server:8000
    it will create a index.html file with the contents of preseed.cfg

I will be the first to say this Python script is very small and dumbed down from the script shown in the wiki. I did that for a reason: since it ignores any real request from user (no matter what you ask, it only sends the config file), it is very simple minded in a good way. Asking it to show the list of files somewhere or upload something might be a bit challenging. Now, you might want instead of offering this config file to serve some kind of simple webpage that is created on the fly, like some status page. You could use a script like the above to do the deed.

I guess where I am really getting to is that if you need, say, a webserver to only server one simple stupid page changes are you do not need a full Apache install. In my own case, why I would even want to have a full fledged webserver running 24/7 just to server a page (or many pages) that only need to be available for a few minutes? I know this concept is not hip anymore, but there is something to be said about having a simple tool that does one single thing well and can cooperate with the other tools to build a complex task.

As I mentioned above, the script is pretty hopelessly dumb. And I bet you can improve on it. I mean, even I decided to improve on it a bit. Specifically, I wanted to be able to provide the filename, IP address (so it is only running on the network interface using that IP), and port from the command line. That would make it easier to use the script without having to modify its code.

And I found that BaseHttpServer really did not want to do that. In fact, what it really want is to read the request from a client and do something based on that. Since that is not what I wanted, I had to learn how to, well, hack my way around that by overriding the __init__ constructor. I am not going to waste time here posting the modified code; I placed the script on github where I hope one day to prettify/improve it.

Notes

  • If you do not want to use python, you can run a webserver in one line of bash or Powershell

Monday, March 23, 2015

environment variables, date, and string concatenation in powershell

This is another of those quick posts. Sometimes in Linux/OSX I want (or even need) to rename or copy a file filename to path/filename_date. For instance, let's say the file is called cli64.log. I then can do something like

bash-3.2$ cp cli64.log cli64_`date +%Y%M%d-%H%M.log`
bash-3.2$ ls -lh cli64.log*
-rw-r--r--  1 dalek  staff   2.8K Feb 21  2013 cli64.log
-rw-r--r--  1 dalek  staff   2.8K Mar 23 16:29 cli64_20152923-1629.log
bash-3.2$
to append the date (as YearMonthDay which in this case turns out to be 20150323) and the time (HourMinute which when I did the above was 1629 or 4:29PM for those who cannot count past 12) to the name. So far so good.

As some of you have guessed -- maybe the title of this article was a dead giveaway -- I sometimes need to deal with Windows. And I do my best to make it behave as close to Linux (using Linux as placeholder for Linux/OSX/whatever since they behave the same in this case. In fact, the machine I ran the above command is a Mac Mini running OSX) as I can, which is why I use Powershell. So, how do I do the same copy command in Powershell?

Date

To get the date, the command we need is Get-Date. When you run it by itself, it gives something like

PS C:\Users\raub> get-date

Monday, March 23, 2015 4:41:13 PM


PS C:\Users\raub>
which is not useful for us; we want to make the date be part of the filename in the format we want. We will work this two part problem starting at the format and then worrying about the concatenation part.

According to the docs, and to http://ss64.com/ps/get-date.html (which has a convenient list of the time formats we can use), we can use the -format option. Let's try and see if we can replicate the output of date +%Y%M%d-%H%M:

PS C:\Users\raub> get-date -format 'yyyyMMdd-HHmm'
20150323-1643
PS C:\Users\raub>
That looks very similar to what we did in Linux. How abut adding that to the filename?

Concatenating

So, concatenating strings in Powershell is a bit interesting. Let's say we have $theTestFile=C:\Users\raub\monkey\testfile.txt and we want to append tmp to it. Now we can try a few things and see what we can come up with:

PS C:\Users\raub> echo "$theTestFile" + tmp
C:\Users\raub\monkey\testfile.txt
+
tmp
PS C:\Users\raub> echo "$theTestFile" += tmp
C:\Users\raub\monkey\testfile.txt
+=
tmp
PS C:\Users\raub> echo "$theTestFile += tmp"
C:\Users\raub\monkey\testfile.txt += tmp
PS C:\Users\raub> echo "$theTestFiletmp"

PS C:\Users\raub> echo "$theTestFile tmp"
C:\Users\raub\monkey\testfile.txt tmp
PS C:\Users\raub> Write-host "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub> echo "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub>
So, the parenthesis thingie seems to be what we want to do.

Putting it all together

Now we know how to get the date and concatenate, if we put the date thingie inside the parenthesis thingie, the equivalent of

cp cli64.log cli64_`date +%Y%M%d-%H%M.log`
in powershell is
copy cli64.log "cli64_$(Get-Date -format 'yyyyMMdd-HHmm').log"

Er, we are not done yet

But, you will point out, the title of this article mentions environment variables! Fair enough. So we will expand the original problem. Say, you also want to name the copy of the log file to not only included when it was copied but also the hostname. Reason here is that you or someone else who will get this file might need to know where this file came from. In Linux, that can be done with $HOSTNAME, as in
cp cli64.log cli64_$HOSTNAME-`date +%Y%M%d-%H%M.log`
but what about Windows and powershell? Enter the environment variables we talked about. We would hope the computer knows what it is called, amongst other things, right? Let's see what it knows
PS C:\Users\raub> ls env:

Name                           Value
----                           -----
ALLUSERSPROFILE                C:\ProgramData
APPDATA                        C:\Users\raub\AppData\Roaming
CommonProgramFiles             C:\Program Files\Common Files
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files
CommonProgramW6432             C:\Program Files\Common Files
COMPUTERNAME                   VBOX01
ComSpec                        C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK               NO
HOMEDRIVE                      C:
HOMEPATH                       \Users\raub
LOCALAPPDATA                   C:\Users\raub\AppData\Local
LOGONSERVER                    \\ZOOL
NUMBER_OF_PROCESSORS           1
OS                             Windows_NT
Path                           %SystemRoot%\system32\WindowsPowerShell\v1.0\;C:\ProgramData\Oracle\Java\javapath;C:\...
PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE         AMD64
PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
PROCESSOR_LEVEL                6
PROCESSOR_REVISION             3c03
ProgramData                    C:\ProgramData
ProgramFiles                   C:\Program Files
ProgramFiles(x86)              C:\Program Files (x86)
ProgramW6432                   C:\Program Files
PSModulePath                   C:\Users\raub\Documents\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowe...
PUBLIC                         C:\Users\Public
SESSIONNAME                    Console
SystemDrive                    C:
SystemRoot                     C:\Windows
TEMP                           C:\Users\raub\AppData\Local\Temp
TMP                            C:\Users\raub\AppData\Local\Temp
UATDATA                        C:\Windows\CCM\UATData\D9FFC898-CBB8-491d-D8CA-173A9FF1B077
USERDNSDOMAIN                  EXAMPLE.COM
USERDOMAIN                     EXAMPLE
USERNAME                       raub
USERPROFILE                    C:\Users\raub
windir                         C:\Windows


PS C:\Users\raub>
That looks very impressive. How do we get that info in a way we can use when renaming out file? Here is how you get the hostname:
PS C:\Users\raub>$env:computername
VBOX01
PS C:\Users\raub>

So, if we wanted to copy poor cli4.log to some directory in some drive, adding to the new filename the hostname and date the file was copied, we could do a lot worse than

copy cli64.log "J:\kitchen\fridge\cli64_$env:computername-$(Get-Date -format 'yyyyMMdd-HHmm').log"

So I think we might have made managing Windows be a bit saner than before.

Friday, March 20, 2015

Quick notes on using git (gitolite) + NetBeans + ssh keys

Most of my posts (hopefully) talk about something that might be useful/helpful to others. Others are to help me not repeat a mistake. This is one of the latter ones; don't expect it to be very impressive. Really. Just look the other way, or at least have the decency of waiting until to laugh at my expense. Deal?

If you remember, I setup a git server using gitolite and docker not long ago. And then I found out one of the future users wanted to access it using NetBeans in Windows. As I have never used that IDE before, I looked in its website and found some info on how to make it talk to github. Well that sounded promising. First place I got stuck was creating the ssh key pair. You see, I know how to use ssh-keygen in Linux/Unix/OSX, but did not know how to do that in Windows. I could install cygwin and do it command line, and that would be great for me but not as nice for a typical Windows user. And, I would be installing a lot of crap this user did not need.

Searching around the net, I found a quick article about how to generate a ssh key on windows using putty (that is exactly how it is called). Now we are making progress; we just need to go and get puttygen, whcih can be obtained by itself, and create the keys. Following the last link, I created a key pair -- 4092 SSH2 RSA without passphrase since this is a test -- as shown in the picture on the left (you can click on it to make it bigger or something). Note the field I highlighted; I will refer to it later. For now, let's ignore that.

Here is the second place I screwed up. At first I thought the buttons to export the public and private keys were what I needed. So, I clicked on them and saved the files. The fact it wanted to give the private key the extension .ppk should have woken me up, but I dozed through that. Completely.

But, as I completely ignored that warning sign, I went back to the NetBeans Instructions and put the private key where it should be. And then I put the public key in my gitolite server. And then tried.

And it did not work.

So I decided to take a look at the keys. Here is the private one (trimmed out a bit to show the format while keeping this article small):

PuTTY-User-Key-File-2: ssh-rsa
Encryption: none
Comment: rsa-key-20150320
Public-Lines: 12
AAAAB3NzaC1yc2EAAAABJQAAAgEA1pA7YEyqbDyVAjjK+VCJpCGxfOwf8WGJ6J4Q
PaE1KMs0TBt5rEPlOpIdsJwOPMBvIUGG8hIHTyxrmOhxj221GYLPNSku7BdLCrt8
[....]
Q/G9k8opoE/0UPtFC0ykkGJg3Vhjq2XRGn3nYJaps7hP8ZUnvkiBuOmBR5SETnFi
3PrYFZ3lgYf1Nz2FgPMjg1uufIxWWv3MUb2Nya44x3JYQUkNfXCjUVmKiK/gE5sM
enufKa0=
Private-Lines: 28
AAACAFb8M79BrN/FiIRcNpxswi3IeGMTnj9DN+iuyFBWHHhSYU9JaK2e/BDTc9H6
E5wWqZfcS3bkttrofqXGBIZBO5S4fYRBINxAy2U3QG8mkBRWoBX/FuyztWBXxKug
pPIDWliV0oiTP0Q+PSXQ4LKTNcZm4UYlKg0Qk+ejR7FTpELqbv0nJQvMulvnEqVk
[...]
a6KahmtK1FFcFUtnLUFW+t7d59TYB/aB6HByzf5JDZPE9dtVpdBF33NSHe0Z4i2e
9XybtlNfYqKWEfaFzaeJqkyhRH5woG7le8GPU67BKVg/mKHYnG3Tk93NvhcOXuyx
7I2AKDeR1M0qMl0aH0ympymKoNxJHD2CBweLyJSoG3QdXY5DxoRvdn/gMeGRdLp0
mR80D0IN/tUduNT/69lB+I1oTU4N+WB/NcHCicOZyzHQRIKnmHC/raBM5ln2Q/ih
qtM=
Private-MAC: 56aa3a9bcb05a65a89110a5de990d5021cfb9273

It sure does not look like the ones I created using ssh-keygen. The public key also looked a bit different. Then it hit me: because gitolite uses openssh, it expects the key to be puttygen is exporting the keys in a different format. So, how do we make this work? Well, do you remember the field I highlighted in the first picture? That is the public key already in the ssh format; that is what gitolite needs. So, cut-n-paste that to, say, smurf.pub inside the keydir directory in the gitolite config file.

Next is to export the private key in the right format. That is done by clickign on Conversions->Exporting OpenSSH key, naming it as something helpful; I named mine smurf_rsa to remind me I happened to have created a RSA key.

Time to go back to Netbeans. The picture on the left shows the setup I used. I was in a hurry so I used the testing repo, which is a bit of a village bicycle in my server: everyone who can connect to localgit can access, read, and write to and in general monkey with it. The key was fed and then I told it to finish. This time it worked or seemed to: proceeded to want me to create a project (with all the little files and directories the IDE creates). I let it do the deed and later on was able to check out what was created.

The moral of this tale is make sure you use the right key format or things will get very interesting. Either that or only drink warm beer if the fridge was built by Lucas, the Prince of Darkness. Or something like that. On the bright side, you managed to reach the end of this tale! You now may put your seat and tray in the upright position and start to laugh.

I did.