Downloading a single file from github (using ansible perhaps)

I was going to setup Ansible to talk to one of my Windows servers. According to the Ansible page on Windows, the easiest way to install the windows side is to download the powershell script ConfigureRemotingForAnsible.ps1 and blindly run it on the windows server. It supposedly does all the magic to install and set things up. Should I run it? I don't know, but let's try to get it. We can inspect its code once we have the file.

NOTE: Setting up Ansible on windows is not the topic of this thread. All I want to show is a way to download a single file from a github repo.

Since I do not know how to do it, let's do some searching. And then we try each method out and see what's what.

Attempt I

There is a thread in stackoverflow called How to pull a single file from a server repository in Git? which
suggested using the git clone command as in

git clone https://github.com/igniterealtime/Openfire.git \
Openfire/src/java/org/apache/mina/management/MINAStatCollector.java

Let's try it out:

raub@desktop:/tmp/rmoo$ git clone https://github.com/igniterealtime/Openfire.git \
Openfire/src/java/org/apache/mina/management/MINAStatCollector.java
Cloning into 'Openfire/src/java/org/apache/mina/management/MINAStatCollector.java'...
remote: Counting objects: 107450, done.
remote: Compressing objects: 100% (53/53), done.
Receiving objects:  14% (15868/107450), 61.11 MiB | 209.00 KiB/s
[...]
remote: Total 107450 (delta 32), reused 31 (delta 16), pack-reused 107380
Receiving objects: 100% (107450/107450), 802.60 MiB | 8.23 MiB/s, done.
Resolving deltas: 100% (63893/63893), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo$ ls Openfire/src/java/org/apache/mina/management/MINAStatCollector.java
build/          i18n/     webadmin/     LICENSE.txt  README.md
dbutil/         src/      webadmintld/  Makefile
documentation/  starter/  xmppserver/   pom.xml
raub@desktop:/tmp/rmoo$

Er, does that look like it grabbed the right thing? For some reason I thought the .java file was, well, a file and not a bunch of files and directories. At least I could swear writing .java files in vi so they were text files. Maybe I am wrong, so let's see if we can get the file I really want:

raub@desktop:/tmp/rmoo$ git clone https://github.com/ansible/ansible.git ansible
/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps1
Cloning into 'ansible/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps
1'...
remote: Counting objects: 236787, done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 236787 (delta 33), reused 25 (delta 6), pack-reused 236712
Receiving objects: 100% (236787/236787), 73.53 MiB | 8.23 MiB/s, done.
Resolving deltas: 100% (152234/152234), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo$ ls
ansible/
raub@desktop:/tmp/rmoo$ ls ansible/blob/devel/examples/scripts/ConfigureRemotin$ForAnsible.ps1/
ansible-core-sitemap.xml  .gitattributes            RELEASES.txt
bin/                      .github/                  requirements.txt
CHANGELOG.md              .gitignore                ROADMAP.rst
CODING_GUIDELINES.md      .gitmodules               setup.py
contrib/                  hacking/                  shippable.yml
CONTRIBUTING.md           lib/                      test/
COPYING                   .mailmap                  ticket_stubs/
.coveragerc               Makefile                  tox.ini
docs/                     MANIFEST.in               VERSION
docsite_requirements.txt  MODULE_GUIDELINES.md      .yamllint
examples/                 packaging/
.git/                     README.md
raub@desktop:/tmp/rmoo$ ls ansible/blob/devel/examples/scripts/ConfigureRemotin$ForAnsible.ps1/
bin/        test/                     docsite_requirements.txt  ROADMAP.rst
contrib/    ticket_stubs/             Makefile                  setup.py
docs/       ansible-core-sitemap.xml  MANIFEST.in               shippable.yml
examples/   CHANGELOG.md              MODULE_GUIDELINES.md      tox.ini
hacking/    CODING_GUIDELINES.md      README.md                 VERSION
lib/        CONTRIBUTING.md           RELEASES.txt
packaging/  COPYING                   requirements.txt
raub@desktop:/tmp/rmoo$

I do not know about you, but that not look like what I really wanted: a single file. On a second thought, that sure looks like the root for the ansible git repo:

I don't know about you but the files and directories sure look familiar.

I guess it is time to try something else.

Attempt II

Let's try something else: in StackOverflow there is a thread called Retrieve a single file from a repository, which suggests

git clone --no-checkout --depth 1 git@github.com:foo/bar.git && cd bar && git show HEAD:path/to/file.txt

For this attempt, we will try to get the file ConfigureRemotingForAnsible.ps1 I want:

git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && cd ansible && \
git show HEAD:examples/scripts/ConfigureRemotingForAnsible.ps1

Thing is that will just spit the file to the screen, literally:

raub@desktop:/tmp/rmoo$ git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && \
cd ansible && git show HEAD:examples/scripts/ConfigureRemotingForAnsible.ps1
Cloning into 'ansible'...
remote: Counting objects: 5873, done.
remote: Compressing objects: 100% (4282/4282), done.
remote: Total 5873 (delta 962), reused 3652 (delta 660), pack-reused 0
Receiving objects: 100% (5873/5873), 7.13 MiB | 8.09 MiB/s, done.
Resolving deltas: 100% (962/962), done.
Checking connectivity... done.
#Requires -Version 3.0

# Configure a Windows host for remote management with Ansible
# -----------------------------------------------------------
#
# This script checks the current WinRM (PS Remoting) configuration and makes
# the necessary changes to allow Ansible to connect, authenticate and
# execute PowerShell commands.
#
[...]

We could improve that by saving the file into a file. But how? The quickest solution I can think of is to pipe it to a file:

file="ConfigureRemotingForAnsible.ps1" ; git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && cd ansible && $(git show HEAD:examples/scripts/$file > $file)

will put $file inside the ansible dir:

raub@desktop:/tmp/rmoo/ansible$ ls
ConfigureRemotingForAnsible.ps1
raub@desktop:/tmp/rmoo/ansible$

Of course we can do better, like placing it on the original pwd and delete the (now temporary) ansible dir. Something like

file="ConfigureRemotingForAnsible.ps1" ; git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && cd ansible && $(git show HEAD:examples/scripts/$file > ../$file) && cd .. && rm -rf ansible

should do just fine. But you do not have to believe on me; here's it in action:

raub@desktop:/tmp/rmoo$ file="ConfigureRemotingForAnsible.ps1" ; \
git clone --no-checkout --depth 1 https://github.com/ansible/ansible.git  && \
cd ansible && $(git show HEAD:examples/scripts/$file > ../$file) && cd .. && rm -rf ansible
Cloning into 'ansible'...
remote: Counting objects: 5873, done.
remote: Compressing objects: 100% (4282/4282), done.
remote: Total 5873 (delta 962), reused 3652 (delta 660), pack-reused 0
Receiving objects: 100% (5873/5873), 7.13 MiB | 758.00 KiB/s, done.
Resolving deltas: 100% (962/962), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo$ ls
ConfigureRemotingForAnsible.ps1
raub@desktop:/tmp/rmoo$

Just to show this is not an accident, let's validate it by applying it to get the java source file we tried to get earlier.

raub@desktop:/tmp/rmoo$ file="MINAStatCollector.java" ; git clone --no-checkout --depth 1 https://github.com/igniterealtime/Openfire.git  && cd Openfire && $(git show HEAD:src/java/org/apache/mina/management/$file > $file)
Cloning into 'Openfire'...
remote: Counting objects: 5291, done.
remote: Compressing objects: 100% (4433/4433), done.
remote: Total 5291 (delta 807), reused 3394 (delta 479), pack-reused 0
Receiving objects: 100% (5291/5291), 92.80 MiB | 8.24 MiB/s, done.
Resolving deltas: 100% (807/807), done.
Checking connectivity... done.
raub@desktop:/tmp/rmoo/Openfire/Openfire$ head MINAStatCollector.java
package org.apache.mina.management;

import static org.jivesoftware.openfire.spi.ConnectionManagerImpl.EXECUTOR_FILTER_NAME;

import org.apache.mina.core.service.IoService;
import org.apache.mina.core.service.IoServiceListener;
import org.apache.mina.core.session.IdleStatus;
import org.apache.mina.core.session.IoSession;
import org.apache.mina.filter.executor.ExecutorFilter;
import org.apache.mina.filter.executor.OrderedThreadPoolExecutor;
raub@desktop:/tmp/rmoo/Openfire$

I think we can improve it by making it more generic, which might be the subject of another post. Or something; I've been dragging on finishing this article for a few weeks now so I just want it gone.

Now what about something completely Ansible-ly?

Er, this Article is getting way longer than I originally planned. I will put the ansible side on another. How about that for raising your expectations and then expertly crushing them?

Create output filename based on input filename using powershell

Here's a situation that happened to me many times in Linux: let's say we create a script which expects the user to enter the input and output filenames. Now, what if the user forgets to enter the output filename? Should we bark or come up with a filename based on the input filename? This of course depends on the situation, but when I decided to create the output filename I would tack today's date to the input filename so they would be different.

But that was Linux and bash and python and this is Windows with powershell. And, yes, we could keep on writing in Bash using cygwin, but that owuld be cheating. Give the constraint of only running what came in Windows 7 and above (I am dating myself), let's see what we can do:

1. Date. The date formats I like are (using Linux here, so focus on the output not the command)

   raub@desktop:~$ date +%F
   2017-01-30
   raub@desktop:~$ 

   and

   raub@desktop:~$ date +%Y%m%d
   20170130
   raub@desktop:~$ 

   Both write year using 4 digits followed by 2 digits for the month and two for the day. I know some people will cry and moan and demand the traditional US format, day/month/year, but the format I like makes sorting in a directory much easier as we are putting what changes the fastest on the end of the filename. But we are talking about powershell, not Bash or Bourne shell. That is true but now we know what we want to accomplish.

   To make it easier, I will pick one of the two formats -- YYYYMMDD -- and run with it; you can later modify the code to use the other one as exercise. The equivalent in powershell is:

   PS C:\Users\raub> get-date -format "yyyyMMdd"
   20170130
   PS C:\Users\raub> 

   Looks just like what we did above in Linux.

2. Create filename if not given. We are reading the input filename into the script in some way or fashion. How we are doing it depends on the script and whether we should be passing options with or without flags to identify them. For now, we are going to be lazy and do the simplest thing possible: using param() on the beginning of the script.

   param($inputFile, $outputFile)

   If we have just one argument, it shall be the inputfile. If two, the second one is the outputfile. What if no arguments are passed? We can just send an error message and get out.

   function Usage
   {
      Write-Host "Usage:", $MyInvocation.MyCommand.Name, "inputfile [outputfile]"
      exit
   }
   
   if (!$inputfile)
   {
      Usage
   }

   The $MyInvocation.MyCommand.Name is a lazy way to for the script to get its own name by itself.

3. Do something if there is no $outputFile. This is a variation of the same test we did to see if we had an $inputFile:

   function LazyOutputFilename($ifile)
   {
      $ofile = (Get-Item $ifile ).DirectoryName + '\' +  `
               (Get-Item $ifile ).BaseName + `
               '_' + (get-date -format "yyyyMMdd") + `
               (Get-Item $ifile ).Extension
      return $ofile
   }
   
   function GetOutputFilename($ifile, $ofile)
   {
      # $ofile cannot be $ifile
      # Create a $ofile if one was not given
      if (( [string]::IsNullOrEmpty($ofile) ) -or ( $ofile -eq $ifile ))
      {  
         $ofile = LazyOutputFilename $ifile
      }
   
      return $ofile
   }
   
   $outputFile = GetOutputFilename $inputFile $outputFile

   • In LazyOutputFilename() we are creating the output filename. We are putting it in the same directory as the input filename and then adding the formatted date right before the file extension.

   • The ( [string]::IsNullOrEmpty($ofile) ) checks is the output file, called $ofile inside this function, is empty. The reason we also wants to make sure the output file is not the input file is because we might be reading the input file a chunk at a time (line by line if text) so the script can handle large files without using up all the memory. If we are reading it line by line and then write right back to it, bad things might happen.

   • And, yes, we are overwriting the output filename if it gets changed in GetOutputFilename().

4. Put everyone together.
   

   param($inputFile, $outputFile)
   
   function Usage
   {
      Write-Host "Usage:", $MyInvocation.MyCommand.Name, "inputfile [outputfile]"
      exit
   }
   
   <#
    Create output filename based on the full path of the input filename +
    today's date appended somewhere
    #>
   function LazyOutputFilename($ifile)
   {
      $ofile = (Get-Item $ifile ).DirectoryName + '\' +  `
               (Get-Item $ifile ).BaseName + `
               '_' + (get-date -format "yyyyMMdd") + `
               (Get-Item $ifile ).Extension
      return $ofile
   }
   
   function GetOutputFilename($ifile, $ofile)
   {
      # $ofile cannot be $ifile
      # Create a $ofile if one was not given
      if (( [string]::IsNullOrEmpty($ofile) ) -or ( $ofile -eq $ifile ))
      {  
         $ofile = LazyOutputFilename $ifile
      }
   
      return $ofile
   }
   
   if (!$inputfile)
   {
      Usage
   }
   
   $outputFile = GetOutputFilename $inputFile $outputFile

Detecting text file format using hexdump

another quick one: so we had a text file that had text with accented words and we had to figure out which format they were. You see, for a while the "standard" text format for computers was ASCII, more precisely 7bit ASCII (characters 0 to 127 in decimal) which was created in the 1960s and whose character set aassumed English language only. Before some of you get all excited please note that ASCII stands for American Standard Code for Information Interchange, so it stands to reason they picked English. As this standard became adopted by other countries, it became clear that some of them used characters that were not representable with only those characters, and that let to many attempts to solve that. One of the earliest was to extend the original ASCII table, where another 128 possible characters were added, which after a few adventures evolved into ISO-8859-1 a.k.a. ISO-Latin-1, and UTF-8. There are other character sets, but the principle is the same.

Thanks for the history lesson, but how about getting to the point? How to identify the text format given a file? Let's answer that by using a couple of examples.

Example 1

Let's say we got a text file that has a name, say Luis de La Peña in it somewhere. Depending how you look -- how helpful your text viewer is -- at the file, it might either show "ñ" or some garbled character; the later happens if the text viewer only knows 7bit ASCII. For instance, cat would spit out something like this in my Ubuntu laptop:

bash-3.2$ cat text_test1 
Luis de La Pe�a 
bash-3.2$

Don't know about you, but that "�" does nothing to me; it's just cat's way of telling us it cannot represent the character so it is putting a placeholder. Let's try something else; since the title of this article mentions hexdump, I propose to look at it through that program (I am telling it to print the value of each character and then the ascii representation of those characters):

bash-3.2$ hexdump -Cv text_test1 
00000000  4c 75 69 73 20 64 65 20  4c 61 20 50 65 f1 61 20  |Luis de La Pe.a |
00000010  0a                                                |.|
00000011
bash-3.2$ 

first thing we notice is that it too does not know how to show "ñ", so it is using "." as placeholder. That is a different character than what cat in my ubuntu box used; just deal with it. What we really care about is the hex side tells us that

0xF1 = ñ

That is very important: it tells us that "ñ" is represented by one 8-Bit character, so UTF8 is out. So, we need to look for an 8-bit charset. After hours of agonizing search and heavy drinking, we find that the extended ASCII and/or the ISO-8859-1 tables match all characters (don't believe me? Check the other characters in the text including space). Not bad at all, so we can read the text and convert it to a different char set.

Example 2

So we feel all good about ourselves, and we need another example. This time, I will steal a real life example from a previous article, where we had a text containing Italienisches Olivenöl which would cause DKIM email body authentication failures. Yeah, something as seemly harmless as character set can create some annoying problems.

As before, we begin by asking cat what it thinks about the text:

bash-3.2$ cat text_test2
Italienisches Olivenöl
bash-3.2$

Hold on right there. Why is cat able to print that "ö" but could not print that "ñ" earlier? Now you begin to see some of the limitations of cat compared to hexdump for these probulations: depending on how cat was compiled, it will handle some character sets but not others. hexdump knows nothing about character sets: it only knows of ASCII; anything else becomes a ".". Of course, it would suck to use hexdump all the time, so you need to know your tools and when to use each one. Since we talked about hexdump, let's see what it sees:

bash-3.2$ hexdump -Cv text_test2
00000000  49 74 61 6c 69 65 6e 69  73 63 68 65 73 20 4f 6c  |Italienisches Ol|
00000010  69 76 65 6e c3 b6 6c 0a                           |iven..l.|
00000018
bash-3.2$

Some kind of funny business happening in the second row:

1. all the English-looking characters not only seem to be represented by one 8bit value but also the same ones we saw earlier in the ASCII example:
   

   0x69 = i
   0x76 = v
   0x65 = e
   0x6e = n
   0x6c = l

2. There are two "." characters (0xc3 and 0xb6) where "ö" should be.
   
3. There is a 0x0a after the "l".
   

What's going on here? Hold onto that question and let's check the "ö". According to the above, the two "." are there to tell us "ö" is represented as two 8bit values:

0xc3b6

If we look at any UTF8 conversion table such as this one (picked at random), we will see that is the UTF8 HEX for "ö" (Unicode code would be U+00F6).

Ok, smartypants, what about the 0x0a after the "l"? Yes that. You might have not noticed it was also on text_test1 on the first example. That is the line feed character, which in Linux means end of line.

Insert Boring Conclusion Here

I hope this was useful to you; I thought it was fun and even learned a few things while writing this. the thought process here is similar to what, say you would do when you are examining an encrypted document: try to find known patterns to work with before going after the really unknown stuff.

Finding the file format of Microsoft Outlook attachment.

Couple of days ago I was asked by one of my customers to help her with her email. Over there they use Exchange (they are a big Microsoft shop) so her mail client is Outlook. She received an email regarding some appointment she had, and it included an attachment and she could not open it.

I have to say I froze. Usually they are careful about not opening suspicious attachments; I hope this was legit and her computer was not infected.
When I got to her office, she showed me the email, which looks proper. And then showed that the attachment was being presented in Outlook as a file called

Appointment Info Jul2015.save

which Outlook thought was an Adobe Acrobat document(?). Maybe Outlook was setup not to show extensions, like Windows does by default. So, I decided that we should contact the sender; he did say he sent the file and even read out the filename of the attachment. Next step was to save the file and see what's what. The quickest way I had to find the extension extension of the saved file was checking its properties. It did show it to me, but it was not .pdf. It was

Appointment Info Jul2015.save.copy

Ookay. As you know I am not a Windows guru; I just am someone who understand linux and Unix in general who happens to dabble with Windows at times. So, I tried to treat it as a linux problem.

Well, we do not have the command file in Windows by default as far as I know. So, how about if we open the file in Notepad and look at the header? Here is what I saw:

PK^C^D
^@^@^@^@^@^Nc▒F^@^@^@^@^

I guess we need to find which file uses that header. A bit of search indicates it is probably a .zip file. I renamed it as

Appointment Info Jul2015.save.zip

and unzipped it. It then created a new directory with a file inside it. And that file was a pdf:

%PDF-1.4
%▒▒▒▒
1 0 obj
<%lt/Type/XObject/Subtype/Image/Width 1602/Height 1037/Mask [252 252 ]/Length 65127/ColorSpace[/Indexed/DeviceRGB 255(^@^@^@^@^@

Thoughts

1. Notepad is not like one of those helpful programs that try to detect and process and file you leave too close to them. And that is good. All I wanted is to see the top of the file without something running without my consent. Kinda like running vi
   
2. Don't let outlook open attachments using any program by default. If you do not know what it is, it should not try to open it.
   
3. I really do not like how Windows by default hides file extensions. Really hate that. Great way to camouflage a bad program from your average user.
   
4. Sometimes you might not have a program that you can click on and will do something. You need to be creative. Hackers do that all day and twice on Sunday.
   
5. Finding out the contents of a mysterious file inside another and so on is a classic event in hacking competition. Sometimes those tasks become very Rudy Goldbergish ina completely new and frustrating level.
   

environment variables, date, and string concatenation in powershell

This is another of those quick posts. Sometimes in Linux/OSX I want (or even need) to rename or copy a file filename to path/filename_date. For instance, let's say the file is called cli64.log. I then can do something like

bash-3.2$ cp cli64.log cli64_`date +%Y%M%d-%H%M.log`
bash-3.2$ ls -lh cli64.log*
-rw-r--r--  1 dalek  staff   2.8K Feb 21  2013 cli64.log
-rw-r--r--  1 dalek  staff   2.8K Mar 23 16:29 cli64_20152923-1629.log
bash-3.2$

to append the date (as YearMonthDay which in this case turns out to be 20150323) and the time (HourMinute which when I did the above was 1629 or 4:29PM for those who cannot count past 12) to the name. So far so good.

As some of you have guessed -- maybe the title of this article was a dead giveaway -- I sometimes need to deal with Windows. And I do my best to make it behave as close to Linux (using Linux as placeholder for Linux/OSX/whatever since they behave the same in this case. In fact, the machine I ran the above command is a Mac Mini running OSX) as I can, which is why I use Powershell. So, how do I do the same copy command in Powershell?

Date

To get the date, the command we need is Get-Date. When you run it by itself, it gives something like

PS C:\Users\raub> get-date

Monday, March 23, 2015 4:41:13 PM


PS C:\Users\raub>

which is not useful for us; we want to make the date be part of the filename in the format we want. We will work this two part problem starting at the format and then worrying about the concatenation part.

According to the docs, and to http://ss64.com/ps/get-date.html (which has a convenient list of the time formats we can use), we can use the -format option. Let's try and see if we can replicate the output of date +%Y%M%d-%H%M:

PS C:\Users\raub> get-date -format 'yyyyMMdd-HHmm'
20150323-1643
PS C:\Users\raub>

That looks very similar to what we did in Linux. How abut adding that to the filename?

Concatenating

So, concatenating strings in Powershell is a bit interesting. Let's say we have $theTestFile=C:\Users\raub\monkey\testfile.txt and we want to append tmp to it. Now we can try a few things and see what we can come up with:

PS C:\Users\raub> echo "$theTestFile" + tmp
C:\Users\raub\monkey\testfile.txt
+
tmp
PS C:\Users\raub> echo "$theTestFile" += tmp
C:\Users\raub\monkey\testfile.txt
+=
tmp
PS C:\Users\raub> echo "$theTestFile += tmp"
C:\Users\raub\monkey\testfile.txt += tmp
PS C:\Users\raub> echo "$theTestFiletmp"

PS C:\Users\raub> echo "$theTestFile tmp"
C:\Users\raub\monkey\testfile.txt tmp
PS C:\Users\raub> Write-host "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub> echo "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub>

So, the parenthesis thingie seems to be what we want to do.

Putting it all together

Now we know how to get the date and concatenate, if we put the date thingie inside the parenthesis thingie, the equivalent of

cp cli64.log cli64_`date +%Y%M%d-%H%M.log`

in powershell is

copy cli64.log "cli64_$(Get-Date -format 'yyyyMMdd-HHmm').log"

Er, we are not done yet

But, you will point out, the title of this article mentions environment variables! Fair enough. So we will expand the original problem. Say, you also want to name the copy of the log file to not only included when it was copied but also the hostname. Reason here is that you or someone else who will get this file might need to know where this file came from. In Linux, that can be done with $HOSTNAME, as in

cp cli64.log cli64_$HOSTNAME-`date +%Y%M%d-%H%M.log`

but what about Windows and powershell? Enter the environment variables we talked about. We would hope the computer knows what it is called, amongst other things, right? Let's see what it knows

PS C:\Users\raub> ls env:

Name                           Value
----                           -----
ALLUSERSPROFILE                C:\ProgramData
APPDATA                        C:\Users\raub\AppData\Roaming
CommonProgramFiles             C:\Program Files\Common Files
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files
CommonProgramW6432             C:\Program Files\Common Files
COMPUTERNAME                   VBOX01
ComSpec                        C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK               NO
HOMEDRIVE                      C:
HOMEPATH                       \Users\raub
LOCALAPPDATA                   C:\Users\raub\AppData\Local
LOGONSERVER                    \\ZOOL
NUMBER_OF_PROCESSORS           1
OS                             Windows_NT
Path                           %SystemRoot%\system32\WindowsPowerShell\v1.0\;C:\ProgramData\Oracle\Java\javapath;C:\...
PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE         AMD64
PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
PROCESSOR_LEVEL                6
PROCESSOR_REVISION             3c03
ProgramData                    C:\ProgramData
ProgramFiles                   C:\Program Files
ProgramFiles(x86)              C:\Program Files (x86)
ProgramW6432                   C:\Program Files
PSModulePath                   C:\Users\raub\Documents\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowe...
PUBLIC                         C:\Users\Public
SESSIONNAME                    Console
SystemDrive                    C:
SystemRoot                     C:\Windows
TEMP                           C:\Users\raub\AppData\Local\Temp
TMP                            C:\Users\raub\AppData\Local\Temp
UATDATA                        C:\Windows\CCM\UATData\D9FFC898-CBB8-491d-D8CA-173A9FF1B077
USERDNSDOMAIN                  EXAMPLE.COM
USERDOMAIN                     EXAMPLE
USERNAME                       raub
USERPROFILE                    C:\Users\raub
windir                         C:\Windows


PS C:\Users\raub>

That looks very impressive. How do we get that info in a way we can use when renaming out file? Here is how you get the hostname:

PS C:\Users\raub>$env:computername
VBOX01
PS C:\Users\raub>

So, if we wanted to copy poor cli4.log to some directory in some drive, adding to the new filename the hostname and date the file was copied, we could do a lot worse than

copy cli64.log "J:\kitchen\fridge\cli64_$env:computername-$(Get-Date -format 'yyyyMMdd-HHmm').log"

So I think we might have made managing Windows be a bit saner than before.

Terrible tales of NIS, NFS, and automounting - II

Maps, Matey!

On the last installment we began to setup the NIS server for Cannelloni Inc, a performance kitcar manufacturing company. Now that we have the domain name defined and a home for the NIS maps we are going to use, how about creating some maps? We will go over that by first working on the NIS server, by creating and exporting the maps, and then reading them in the client.

Server setup

Ok, we need to create the NIS maps, but what are those maps anyway? Well, maps are the files NIS uses to keep the information it needs and passes around. Think of them as plain text databases where each entry is a pair (as in first column is defined by the remaining columns). I guess the best way to explain them is to show how they compare to some of the files used by Linux/Unix:

Maps	Equivalent unix file	Comments
hosts.byname, hosts.byaddr	/etc/hosts	Maps IP addresses to host names
passwd.byname, passwd.byuid	/etc/passwd	Maps UIDs to usernames (and passwords)
group.byname, group.bygid	/etc/group	Maps Group IDs to group names

So, our /etc/src/auto.home would look something like this:

bob            -nosuid,intr    obelix.cannelloni.com:/export/home/bob
thetick        -nosuid,intr    obelix.cannelloni.com:/export/home/thetick
heathcliff     -nosuid,intr    obelix.cannelloni.com:/export/home/heathcliff
mccoy          -nosuid,intr    obelix.cannelloni.com:/export/home/mccoy

and so on.

The netgroup map, which we chose (when we edited the /var/yp/Makefile, remember?) to be stored in /var/yp/, is like the groups file but can be used to group not only users but also any combination of users, domains, and hosts. We have two printers, falbala and bonemine, so we create a group for them which will be called printers. So far, our /var/yp/netgroup file looks like this:

openwheel (assurancetourix,,) (alambix,,) (caiousbonus,,) (petisuix,,)
printers  (falbala,,) (bonemine,,)

Remember that once you ad a Linux box as a NIS client you should run /usr/sbin/gdm-restart so the login window knows of the changes and maps. For some reason, ssh and the text-based login screen have no problems being updated, but gdm does. Perhaps it is buffering the user data.

For Linux, create an /etc/exports file that looks like this:

/home/sunpci/linux 192.168.0.11(rw,no_root_squash)

For Solaris, set up your /etc/dfs/dfstab this way:

share -F nfs -o rw=@192.168.0.11/32,root=@192.168.0.11/32 /home/sunpci/linux

Once you have finished with /etc/exportfs, you need to make the changes take place by typing

Linux:

# exportfs -a

Solaris:

# share all

Terrible tales of NIS, NFS, and automounting

NFS and NIS have been around for a while, way before someone decided to network two Windows boxes. They have a lot of neat features.

The Network Information Service (NIS) is a directory service protocol created by Sun. It is not as elegant as, say, LDAP with Kerberos, but can get the work done if due care is taken to keep it as safe as possible.

I am going to present the steps necessary to setup a NFS/NIS system that would server a bunch of users and the unix boxes they connect to. Originally I was going to make this fit one single post but I realized that it (hopefully) would be easier if I broke down into sessions and dealt with just one aspect at a time. I will also create a fake company so it has that professional look to it. Sounds like a plan? Great! Let's get busy then. In this example, we work at Cannelloni LLC, a performance kitcar manufacturing company. It is primarily a Linux shop all the way to the desktops. Recently it has grown enough to need a centralized directory service and file sharing systems. Since we are talking about NIS, Cannelloni chose to use NIS for now. Later on the show we will talk about NIS limitations.

Layout

I would start this mess with the main/master NIS server because I want to have the authentication side of the business out of the way. First of all, we need a NIS domain name. This really needs not to have anything to do with the DNS domain, but should make sense to you. Think of it as a logical group or unit. You talked to your boss and after a few beers it was decided to divide the current network mess into the following groups:

• Management
• Office
• Accounting
• HR
• Development
• Production

Probably you could have come up with better names, but that is what you get by trying to work drunk. You can always change them later. If nothing else, just to piss off accounting. Since doing every single group would bore me to death, we will assume that development decides to take the lead. If it works there, the same concept will be generalized across the entire company. So, development choses idefix (it is not very powerful but that really does not matter) as its NIS server; another machine, obelix, which has a nice hotswappable RAID 5 will be the fileserver which will export fileshares through NFS.

First, we start by finding out a bit about the company's network and which part of it belongs to development. Careful research indicates the entire company is behind a router, so it has only a handful of public IPs (webserver, mail, and so on) while the LAN uses the private network 192.168.10.0 (it is a small company). The IPs assigned to development are 192.168.10.100 to 192.168.10.120, and all the IPs have not been assigned yet. This is very important to know because we can limit which machines can see the NIS maps. How do we do that? Well, we are getting a bit ahead; let's first create a place to save all the configuration files we will be creating.

We Need a Home

NIS stores a lot of important stuff in general in /var/yp Go take a look at it; it should look kinda like this:

dalek@idefix-> ls /var/yp
binding  Makefile  nicknames
dalek@idefix->

Kinda boring I know but we are just starting with it. The Makefile you see there is used to generate the NIS maps. By default it will use /etc/passwd, /etc/shadow, /etc/group and a lot of other files that are in /etc. I honestly do not like that. /etc for me is kinda of an important directory and I would rather have its contents not being passwd all over the universe. Instead, I prefer to feed NIS my own passwd, group, and any other map I want to share. Not only that makes it a bit safer but also easier to manage/move around as everything is contained in a single location you can simply tar and move to the next machine. So, we need to do some editing in Makefile. So, I create two directories: /var/yp/src and /var/yp/src/pw. Then, I edit the Makefile as follows (you will need to search within that file for those definitions):

• Linux

  # YPSRCDIR = /etc
  YPSRCDIR = /var/yp/src
  # YPPWDDIR = /etc
  YPPWDDIR = /var/yp/src/pwd
  

• Solaris

  # DIR =/etc
  DIR =/var/yp/src
  # PWDIR =/etc
  PWDIR =/var/yp/src/pwd

The next step is to create those directories and make sure they can only be read/accessed by root, specially /var/yp/src/pwd as it will host the password file that will be shared through NIS. Next we will create a file called securenets in /var/yp which will tell which machines can see these maps:

dalek@idefix-> cat /var/yp/securenets
# /var/yp/securenets
# Restrict access to the NIS maps to the machines defined in this file
#
# allow connections from local host -- necessary
host 127.0.0.1
# same as 255.255.255.255 127.0.0.1
#
# allow connections from any host on the development network
host 192.168.10.100  # asterix
host 192.168.10.101  # obelix
host 192.168.10.102  # idefix
host 192.168.10.103  # panoramix
host 192.168.10.104  # abraracourcix
host 192.168.10.105  # bonemine
host 192.168.10.106  # agecanonix
host 192.168.10.107  # assurancetourix
host 192.168.10.108  # cetautomatix
host 192.168.10.109  # ordralfabetix
host 192.168.10.110  # lelosubmarine
host 192.168.10.111  # falbala
host 192.168.10.112  # aplusbegalix
host 192.168.10.113  # amerix
host 192.168.10.114  # caiusbonus
host 192.168.10.115  # caiusmalosinus
host 192.168.10.115  # tragicomix
host 192.168.10.116  # alambix
host 192.168.10.117  # petisuix
host 192.168.10.118  # jolitorax
host 192.168.10.119  # beaufix
host 192.168.10.120  # barberouge
dalek@idefix->

Do notice we chose to specify each host in /var/yp/securenets. Since the number of machines in this list is not a power of 2, we could not use IP of the first machine and a carefully chosen subnet mask to cover them all. Also, spelling out every machine we plan on using allows us to later on comment out the ones we do not need.

A Domain by Any Name

Now that we have that taken care of, we need to come up with a name for our NIS domain. Since this name does not need to be remotely related to our DNS domain, we will call it development as it is the NIS domain for the development group. I know, I know, I am very original...

Ok, you ask, now how to make NIS know we chose a NIS domain name? Well, the domain name is stored in /etc/defaultdomain (for solaris) or /etc/domainname (for Linux). If you write the domain name you want to use in that file (i.e. there is just one single line in the file and all it has is your domain name, in this case, development) and reboot, idefix will then know the name of the domain. You can check it by using the command domainname as follows:

dalek@idefix>cat /etc/defaultdomain
development
dalek@idefix>

Now, before you go rebooting the machine, let's see if we can change its domain name without rebooting, shall we? In both Solaris and Linux, you can set the runtime domain name to development by saying

dalek@idefix>domainname development
dalek@idefix>

How about checking it?

dalek@idefix>domainname
development
dalek@idefix>

Of course, since we had already defined the domain name in /etc/defaultdomain (Solaris) or /etc/domainname, we could have said

• Solaris

  dalek@idefix>domainname `cat /etc/defaultdomain`

• Linux

  dalek@idefix>domainname `cat /etc/domainname`

Do note the back quotes; they are rather important.

Ok, I am going to take a break for now. Next time we will talk about the wonderful world of maps. Stay tuned!

Backing up

It has been said that a business will not survive if it does not have a good backup system. I personally agree with it. I also have noticed that many companies only consider backup solutions after the lack of it bit them in their behinds, resulting in data and productivity losses, credibility issues, and, above all, losses where it counts the most: the wallet. So, why would they avoid it like most of us, excluding characters like a certain guy from Little Shop of Horrors, would avoid a visit to the dentist? Well, sometimes it has to do with perceived cost. Since backup does not directly translates to making profit, it is seen as waste of good money which could be used in more productive manners. Kinda like boats.

Then, we have the old adage, do not change a winning team. In other words, if the system is working, do not mess with it by adding this newflanged and unproven backup thingamajig. That also implies that a properly configured system should not need backup. So, if the system has a problem -- something that is to be expected as no systems are perfect (yes, I know, there are systems that are less perfect than others) and hardware will go boink before tea-time -- someone did not do his/hers/its share. As a result, heads in the IT group will roll.

Anywhoo, the point is that files will be lost (either by malice or by mistake) and machines will crash and hard drives will fail and other things will happen to make your original copy of the data unusable. At that point, if you have backup you only lose a few hours worth of work if that. If you do not, well, you may have lost years of data. It happened to me and I can tell you it is not a nice feeling.

All this is nice and boring. How hard is to have a backup? And, how expensive it can be? It depends on what do you want to do, how much effort you are willing to put on it, and how nice you want it to be. You have price, speed, and quality, but can only pick two of them. That is fine. How complex a backup system do you need? Sometimes not that much if your needs are small. Let me present an example based on a real story. I was asked by a company I consult for to come up with a simple, hands-off backup of their most important application (in this case a program they use to keep track of what happens in their shop). One of the most important issues is that they want to be able to continue working in case their application server goes on a holiday. And, they run Windows XP. Yes, I know this blog is supposed to be about unix, but the principle is what matters. I promise I will show later an unix example.

My approach was first to understand the program. Because it is dos-based (that is a choice done by the developer, and they do go over great lengths to justify it. It really boils down to: it is works nicely and does not need much resources to run) program, it is placed on a directory which is then exported to the machines that are authorized to mount it (sounds familiar? Can you say NFS + NIS/LDAP? I bet you can). Interestingly enough, for it to work the entire system disk of the server machine that hosts the program must be exported with permissions set for everybody who access the program be able to read and write to the disk for it to work. Why? I do not know but do think that is a safety issue. But, I digress...

When the client said he wanted to be able to continue working in case of system crash, what he really meant was that if the server crashed, he would be able to go to another PC in his network and run off the backup copy. So, it was decided the most convenient backup media was an USB hard drive formated with one NTFS partition. To keep this quite Windows-centric, we chose to use robocopy, which is found in the Microsoft website, as the underlining program we create a little script (or batch file for those of you who are anal about terminology), that would be called to do out daily backups.

:: BACKUP.BAT
:: Backs up a single directory from one location to another.  Since we are using robocopy,
:: the locations can be in local or network drives.
@echo off

:: What to copy and where to copy to
set dirname=shopmanager
set sourcepath=z:set targetpath=f:set sourcedir=%sourcepath%%dirname%
set targetdir=%targetpath%%dirname%

:: Just to be on the safe side, I am defining where robocopy has been 
:: installed. BTW, this is the default path.
set COPYPATH=C:\Program Files\Windows Resource Kits\Tools\robocopy
set COPYARGS=/e /copyall

:: Add today's day number to targetdir
set today=%date:~7,2%
set targetdir=%targetpath%_%today%

echo %COPYPATH% %sourcedir% %targetdir% %COPYARGS%

And