Finding disk space hogs in a Windows server/workstation

If you have any doubts, we will be doing it from the command line. Just want to put that out before we start. Also, it turned out this is a long and boring article; deal with it.

So, where were we? Disk space and what is using it. That is a problem common to all OS: you have a partition without infinite disk space (sorry ZFS, it happens sometimes) and is running out of space:

• Sometimes it is a careless user; some OS allow you to tell non-system programs, like the ones run by a user, can only use up to 95% of the disk. This way we have some space to fix things.
  
• Sometimes it is actually a program being run as a system/root/admin account, which is trouble since it can use up the entire partition.

If the machine in question is a server, a Windows server since that is what we wrote in the name of the article, we might not be able to just ignore it; others will be affected by this. So, how to take care of this problem? The lazy fix is to throw more space at it and move on. The proper solution is to find out who is hoarding all the space and why. I would like to talk about doing the right thing.

The standard Windows approach would be to search for some app online, which must have a graphics interface and ideally from some site with a name like "finddiskusage.com" because such domain names do inspire confidence, right? Specially when the site's text is pretty much "You do not know what is using your disk space? Click here to download the solution!" Any relationship with a phishing email is merely coincidental.

So, after downloading this shady program from the suspicious website, we then install it and make sure to turn off the firewall and run it with admin rights. And after it does what we hope it is supposed to do, we then take a screenshot of the output and paste it to our documentation.

I do not know about you but I really do not like to install programs in any server, be it windows/linux/mac/solaris/aix/whatever. I think they should have only the bare minimum to do their job; you should see my Linux servers. Since I am the one writing this article, I will put my dictator hat and look for something that fits my style.

In Unix in general and Linux as a special case there is a program called du which allows you to check the disk usage at a given location. You can be short and only show, say, how much all the files and directories (folders in Windows) inside a given directory, or go recursively and show detailed views for every single directory inside the original one. Output is text, which means you can feed it to something else like sort or some program that will make a decision based on the data.

It would be really cool if there was something like that in Windows. One can dream...

Thing is, we do not have to wait for unicorns and fairies to come up with a solution. Nor we have to reinvent the wheel. You see,

1. There is something like that natively for windows. I do like cygwin but that requires installing yet another collection of packages that need to be patched and upgraded. Kinda wasteful if all you want is little du. I believe less is more.
   
2. It is called du just like the unix one.
   
3. You do not need to look for it in some shady or just compromised site. You can find it right at Microsoft as part of the sysinternals package(s).
   
4. You do not even need to install anything. Just put its directory somewhere you want to use it, including a USB or network drive, and run it from there.

Not bad if you ask me. Enough talk, let's use it.

Using du

The most common way I use du in Windows is like I use in Linux:

C:\Documents and Settings\raub>"\Documents and Settings\raub\My Documents\DU\du.exe" -l 1 \windows >> du.log

Du v1.5 - report directory disk usage
Copyright (C) 2005-2013 Mark Russinovich
Sysinternals - www.sysinternals.com


C:\Documents and Settings\raub>

Ok, it is old but it is not like it is getting more and more useless features to make it bloated. Like interfacing with your bluetooth-enabled IoT-based massage chair. Let me show it in action with a real (!) example: at work I have a Windows 10 vm for desktop. And as you can see, it ran out of disk disk space:

That's not much free space left! If you know Windows, it will get really slow when its boot/OS disk is that full. I am going to assume I should first check the Users directory. If I am wrong, I would then check the Windows one, First I would like to make a point I will be running du.exe off a network fileshare, strongbadia

PS C:\Users\raub> ls \\spacemoose\users\raub\bin


    Directory: \\strongbadia\users\raub\bin


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        5/21/2018   9:32 AM         169072 du.exe
-a----        5/21/2018   9:32 AM         191616 du64.exe
-a----        6/28/2018   4:06 PM            543 GetDirSize4DateRange.ps1
-a----        7/18/2017  10:12 AM         854072 putty.exe


PS C:\Users\raub>

You are now getting to learn a few secret things about me! Yes I have a bin dir. Inside it you can see the du.exe and du64.exe. Both are very tiny compared to the crazy GUI programs you can get off suspicious sites to do the very same thing. And, that is all you need: those two files. Well, I will be running du.exe even though my guest is a 64bit windows vm. Because I can. So, let's see what is in the root dir for the users dir:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

      31,480  c:\users\bob.adm
       2,665  c:\users\Default
  51,390,386  c:\users\raub
     629,717  c:\users\raub.adm
       2,727  c:\users\raub.tst
       2,768  c:\users\Public
     129,592  c:\users\windows-user
Files:        2429983
Directories:  9581
Size:         54,003,166,071 bytes
Size on disk: 59,917,407,544 bytes

PS C:\Users\raub>

Man! My homedir is full of junk. What is that and where is it? Let's check it:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\raub\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

  46,023,020  c:\users\raub\AppData
           0  c:\users\raub\Contacts
           1  c:\users\raub\Desktop
           3  c:\users\raub\dev
   1,836,295  c:\users\raub\Documents
   3,551,535  c:\users\raub\Downloads
           0  c:\users\raub\eqlgroupmgr
           0  c:\users\raub\Favorites
           1  c:\users\raub\Links
           0  c:\users\raub\Music
           0  c:\users\raub\OneDrive
       5,012  c:\users\raub\Pictures
           0  c:\users\raub\Saved Games
           3  c:\users\raub\Searches
           0  c:\users\raub\Videos
Files:        2425559
Directories:  3976
Size:         52,663,166,132 bytes
Size on disk: 58,521,224,472 bytes

PS C:\Users\raub>

AppData! A conveniently normally invisible source of many out of space drives. Not to du it is. Let's keep going in:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\raub\appdata\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

  45,780,916  c:\users\raub\appdata\Local
       2,588  c:\users\raub\appdata\LocalLow
     198,212  c:\users\raub\appdata\Roaming
Files:        2424970
Directories:  3898
Size:         47,085,278,621 bytes
Size on disk: 52,935,700,880 bytes

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 c:\users\raub\appdata\local\
[...]
PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 C:\users\raub\appdata\local\Microsoft\Windows\INetCac
he\\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

         239  c:\users\raub\appdata\local\microsoft\windows\inetcache\Content.MSO
         878  c:\users\raub\appdata\local\microsoft\windows\inetcache\Content.Outlook
       5,207  c:\users\raub\appdata\local\microsoft\windows\inetcache\Content.Word
      18,852  c:\users\raub\appdata\local\microsoft\windows\inetcache\IE
  39,825,165  c:\users\raub\appdata\local\microsoft\windows\inetcache\Low
           0  c:\users\raub\appdata\local\microsoft\windows\inetcache\Virtualized
           0  c:\users\raub\appdata\local\microsoft\windows\inetcache\WebTempDir
Files:        2410022
Directories:  55
Size:         40,806,752,248 bytes
Size on disk: 46,635,979,024 bytes

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 C:\users\raub\appdata\local\Microsoft\Windows\INetCache\low\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

  39,819,877  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\IE
Files:        2408915
Directories:  34
Size:         40,780,969,849 bytes
Size on disk: 46,606,749,696 bytes

PS C:\Users\raub>

That smells like that other nemesis of web browsers: Internet Explorer or Edge. What is inside that dir?

PS C:\Users\raub> ls C:\users\raub\appdata\local\Microsoft\Windows\INetCache\Low\IE\
PS C:\Users\raub> dir C:\users\raub\appdata\local\Microsoft\Windows\INetCache\Low\IE\
PS C:\Users\raub>

WTF? Why can't I see what is inside it? Let me du inside it:

PS C:\Users\raub> \\strongbadia\users\raub\bin\du.exe -l 1 C:\users\raub\appdata\local\Microsoft\Windows\INetCache\low\

DU v1.61 - Directory disk usage reporter
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

   1,237,378  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\10EAXOSV
   1,246,204  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\145KIAQM
   1,235,782  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\2F8FLUJD
   1,241,300  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\2VB3GL0K
   1,250,215  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\5BZQD406
   1,244,360  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\5IVFOAV9
[...]
   1,251,347  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\TQ26RPSG
   1,236,991  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\VQH18R7G
   1,240,357  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\VYHW6URF
   1,239,508  c:\users\raub\appdata\local\microsoft\windows\inetcache\low\ie\XYKKE45S
Files:        2408912
Directories:  33
Size:         40,775,554,697 bytes
Size on disk: 46,601,322,496 bytes

PS C:\Users\raub>

Before you ask, I am using a powershell window, where ls and dir behave the same. I came from unix so you can understand which one I prefer. So we have a ton (33) of stupid cache folders that the browser could not be bothered to delete after it quit. Thanks, Microsoft, for not cleaning after itself. Really. And, everything below c:\users\raub\appdata\local\microsoft\windows\inetcache\low is hidden (?). Alright then, off it goes! Command line cares not about hidden paths! Note: get-help can be seen as the powershell equivalent of the Unix/linux man.

PS C:\Users\raub> get-help rm

NAME
    Remove-Item

SYNOPSIS
    Deletes files and folders.


SYNTAX
    Remove-Item [-Confirm] [-Credential ] [-Exclude ] [-Filter ] [-Force] [-Include
    ] -LiteralPath  [-Recurse] [-Stream ] [-UseTransaction] [-WhatIf]
    []

    Remove-Item [-Path]  [-Confirm] [-Credential ] [-Exclude ] [-Filter ]
    [-Force] [-Include ] [-Recurse] [-Stream ] [-UseTransaction] [-WhatIf] []
   
    Remove-Item [-Stream ] []
   

DESCRIPTION
    The Remove-Item cmdlet deletes one or more items. Because it is supported by many providers, it can delete many
    different types of items, including files, folders, registry keys, variables, aliases, and functions.
    In file system drives, the Remove-Item cmdlet deletes files and folders.
   
    If you use the Stream dynamic parameter, it deletes the specified alternate data stream, but does not delete the
    file.
   
    Note: This custom cmdlet help file explains how the Remove-Item cmdlet works in a file system drive. For
    information about the Remove-Item cmdlet in all drives, type "Get-Help Remove-Item -Path $null" or see Remove-Item
    at http://go.microsoft.com/fwlink/?LinkID=113373.
   

RELATED LINKS 
    Online version: http://technet.microsoft.com/library/jj628241(v=wps.630).aspx
    Remove-Item (generic); http://go.microsoft.com/fwlink/?LinkID=113373
    FileSystem Provider
    Clear-Content
    Get-Content
    Get-ChildItem
    Get-Content
    Get-Item
    Remove-Item
    Set-Content
    Test-Path


REMARKS
    To see the examples, type: "get-help Remove-Item -examples".
    For more information, type: "get-help Remove-Item -detailed".
    For technical information, type: "get-help Remove-Item -full".
    For online help, type: "get-help Remove-Item -online"


PS C:\Users\raub> rm -force -recurse C:\users\raub\appdata\local\Microsoft\Windows\INetCache\low\IE

Note that since we are running this from the command line, I did not have to do the usual screenshot Windows blog and articles love so much. I could cut and paste the real output and put it here. I could also have piped it into some other script to use the output for its nefarious uses. FYI, the above command has been running for 38 minutes now and has not finished yet.

So, what I have done above in this real example can be also used with servers since

1. We are using Microsoft published program.
   
2. The program is self-contained and requires no installation.
   
3. The program fears no hidden directories.
   
4. The program has very small footprint.
   
5. The program can be run locally, off a USB, or from a network drive.

If you can put up with not having a cute window with some animation, I think sysinternals' version of du is a nice tiny add-on to a Windows server manager arsenal.

Testing for multiple strings without much clutter using powershell

Yes, this hopefully will be quick, and yes it is powershell, which does not make me feel as dirty as if it was Windows. So hear me out.

I wrote a script a while ago that I needed to look for a pattern inside a string and then do something. In its simplest form, the code could look like this (second line is there to show the entire string):

$the_string = "There are pickles in a jar"
$the_string

if ($the_string -match "pickles")
{
        "Found me pickles"
}
else
{
        "nothing to declare"
}

If you wonder why I am using -match instead of -contains, there is a nice discussion you want to check. At least I learned a lot from it. When we run the script, which is henceforth called switchtest.ps1, we get

PS C:\Users\dalek> powershell -file .\dev\switchtest.ps1
There are pickles in a jar
Found me pickles
PS C:\Users\dalek>

So far so good. But, what if we want to do things based on whether other substrings are in the string? After all, instead of one single string we might be looping over a list of them, or reading a file and doing things based on what we find on a line-by-line-basis. We could add more if statements but that gets nasty quickly:

$the_string = "There are pickles in a jar"
$the_string

if ($the_string -match "pickles")
{
        "Found me pickles"
}

elseif ($the_string -match "there")
{
        "There is here"
}
else
{
        "nothing to declare"
}

Notes:

• By default powershell is case insensitive; this is consistent with how DOS and Windows behaves, which is exactly the opposite of UNIX in general and Linux specifically.
• If you have used other programming languages like python or C, you might remember switch statements (switch, then case-this and case-that). And, they are also implemented in powershell without the case word being explicitly used but the behavior is there for all to see.
• -match looks for the substring anywhere in the string. So, it would find there in:
  • $the_string = "There are pickles in a jar"
  • $the_string = "Are there pickles in a jar?"
  • $the_string = "Pickles are in a jar over there"

Let's now redo switchtest.ps1 using case:

$the_string = "There are pickles in a jar"
$the_string

switch -wildcard ($the_string)
{
        "*pickles*" {"Found me pickles"}
        "there*" {"There is here"}
        default {"nothing to declare"}
}

and then run it

PS C:\Users\dalek> powershell -file .\dev\switchtest.ps1
There are pickles in a jar
Found me pickles
There is here
PS C:\Users\dalek>

The -wildcard option is where the main magic is. It allows me to enter a string as the search pattern, such as "pickles". The more astute of you probably noticed the asterisk (*) surrounding the search string. Without the first, it would only look for a string that starts with "pickles"; Since we only want to look for there when it begins the string we do not need the leading *. There is a thread in the Spiceworks forum showing another example of looking for strings that begin with a, in their case, specific letter using a case statement. The second one is so it will not look only for string ending with that substring.

There is nothing forcing each case statement to be a one-liner. In fact, it would be clearer to write the second one as

// Do something if the string begins with "there" 
        "there*" 
             {
                   "There is here"
             }

specially if we are going to do more than just write out a string.

Another thing you might also have noticed is the behaviour of the output does not match the one we did using the if statement. Reason is unless we specifically tell the switch statement to stop testing once it finds a match, it will keep going down to the list (the default is only done if nothing matches. Sometimes that is exactly what you want to do, but let's assume that is not the case. Just as in other languages, the command we need to give it is break. Let's add them

$the_string = "There are pickles in a jar"
$the_string

switch -wildcard ($the_string)
{
        "*pickles*" {"Found me pickles"; break}
        "there*" {"There is here"; break}
        default {"nothing to declare"}
}

and try it once more

PS C:\Users\dalek> powershell -file .\dev\switchtest.ps1
There are pickles in a jar
Found me pickles 
PS C:\Users\dalek>

Now the output looks just like the original script, but it is much cleaner and easier to expand.

Tivoli (TSM) Backup can't backup a drive in Windows

Over here we use IBM's TSM backup system. I do not want to go over its features and setup, but the bottom line is that I get an email listing the backup status for each machine (known as nodes in TSM lingo) I am backing up. And one day one of those nodes barked:

backup7x SERVER02.EXAMPLE           Failed***    12     2017-06-22 00:00:00 2017-06-22 
00:01:06 2017-06-22 00:01:07

If you are curious about the 12, here is what it means right out of that very same email (I copied that session including the wasteful blank lines):

Result:

0 - Success.

1 - See explanation for 'Missed'.

4 - The operation completed successfully, but some files were not
processed.

8 - The operation completed with at least one warning message.

12 - The operation completed with at least one error message
(except for error messages for skipped files).

That does not help me much. You see, I like to have access to logs and not sad face cryptic messages. So I went to C:\Program Files\Tivoli\TSM\baclient to look into dsmsched.log for any funny business. And funny business I found:

06/22/2017 00:01:09 --- SCHEDULEREC OBJECT BEGIN D-0000AM 06/22/2017 00:00:00
06/22/2017 00:01:10 Incremental backup of volume '\\server02\d$'
06/22/2017 00:01:11 ANS1228E Sending of object '\\server02\d$' failed.
06/22/2017 00:01:11 ANS1751E Error processing '\\server02\d$': The file system can not 
be accessed.
06/22/2017 00:01:11 --- SCHEDULEREC STATUS BEGIN
06/22/2017 00:01:11 --- SCHEDULEREC OBJECT END D-0000AM 06/22/2017 00:00:00
06/22/2017 00:01:11 ANS1512E Scheduled event 'D-0000AM' failed.  Return code = 12.
06/22/2017 00:01:11 Sending results for scheduled event 'D-0000AM'.
06/22/2017 00:01:11 Results sent to server for scheduled event 'D-0000AM'.

Ok, what's so special about the D drive? I looked at the config file, C:\Program Files\Tivoli\TSM\baclient\dsm.opt, and it seems to be right. If you do not believe me (I wouldn't and I have to live with me), here are its first few lines:

NODENAME SERVER02.EXAMPLE
TCPSERVERADDRESS backup7x.example.com

DOMAIN "\\server2\d$"
MANAGEDSERVICES WEBCLIENT SCHEDULE
webports 1501 1581

txnbytelimit 25600
schedmode prompted
schedlogretent 30,d
errorlogretent 30,d
passwordaccess generate
quiet
tapeprompt no


EXCLUDE.BACKUP "*:\Thumbs.db"
EXCLUDE.BACKUP "*:\desktop.ini"
EXCLUDE.BACKUP "*:\*.tmp"
EXCLUDE.BACKUP "*:\...\Scans\mpcache-*"
EXCLUDE.BACKUP "*:\microsoft uam volume\...\*"
EXCLUDE.BACKUP "*:\microsoft uam volume\...\*.*"
EXCLUDE.BACKUP "*:\...\EA DATA. SF"
EXCLUDE.BACKUP "*:\IBMBIO.COM"
EXCLUDE.BACKUP "*:\IBMDOS.COM"
EXCLUDE.BACKUP "*:\IO.SYS"
[...]

As you can see, I am telling it to only backup the D drive. Maybe we should take a look at this drive and see who can access it:

C:\Users\raub> icacls d:\
d:\ AD\EXAMPLE_Domain Admins:(OI)(CI)(F)
    AD\EXAMPLE_Users:(RX)

Successfully processed 1 files; Failed processing 0 files
C:\Users\raub>

Where:

• OI: Object inherit
  
• CI: Container inherit
  
• F: Full access
  
• RX: Read and execute

We can also do that through powershell:

PS C:\Users\raub> get-acl d:\ | fl


Path   : Microsoft.PowerShell.Core\FileSystem::D:\
Owner  : BUILTIN\Administrators
Group  : AD\Domain Users
Access : AD\EXAMPLE_Domain Admins Allow  FullControl
         AD\EXAMPLE_Users Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:BAG:DUD:PAI(A;OICI;0x1200a9;;;SY)(A;OICI;FA;;;S-1-5-21-344340502-4252695000-2390403120-1439459)(A;;0x1200a9;
         ;;S-1-5-21-344340502-4252695000-2390403120-1439468)(A;OICI;FA;;;S-1-5-21-344340502-4252695000-2390403120-14759
         66)



PS C:\Users\raub>

which as you can see is a more verbose way to say the same thing. But what is missing here? You see, by default Windows services are run by the system user (it's full name is NT AUTHORITY\SYSTEM. So let's add it. Does it need to write to the drive as far as TSM is concerned? We are backing up here. Maybe if we need to restore we might need to write but we will cross that bridge when we get to it (hopefully never).

You can add that user and setup the permissions (I did read-execute; but wonder if read only would suffice. Let me know if you find the answer) either using the windows explorer, icacls, or Set-Acl. Pick one; what really matters is that in the end of the day you should have something like this:

C:\Users\raub> icacls d:\
d:\ AD\EXAMPLE_Domain Admins:(OI)(CI)(F)
    AD\EXAMPLE_Users:(RX)
    NT AUTHORITY\SYSTEM:(OI)(CI)(RX)

Successfully processed 1 files; Failed processing 0 files
C:\Users\raub>

or in powershell,

PS C:\Users\raub> get-acl d:\ | fl


Path   : Microsoft.PowerShell.Core\FileSystem::D:\
Owner  : BUILTIN\Administrators
Group  : AD\Domain Users
Access : NT AUTHORITY\SYSTEM Allow  ReadAndExecute, Synchronize
         AD\EXAMPLE_Domain Admins Allow  FullControl
         AD\EXAMPLE_Users Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:BAG:DUD:PAI(A;OICI;0x1200a9;;;SY)(A;OICI;FA;;;S-1-5-21-344340502-4252695000-2390403120-1439459)(A;;0x1200a9;
         ;;S-1-5-21-344340502-4252695000-2390403120-1439468)(A;OICI;FA;;;S-1-5-21-344340502-4252695000-2390403120-14759
         66)



PS C:\Users\raub>

And now I get an email saying all is well:

backup7x SERVER02.EXAMPLE           Completed    0      2017-06-24 00:00:00 2017-06-24 
00:00:54 2017-06-24 01:07:57

Some of you noticed this status email is from 2 days later. The reason was that on the 23rd it was catching up and that took quite a while.

Create output filename based on input filename using powershell

Here's a situation that happened to me many times in Linux: let's say we create a script which expects the user to enter the input and output filenames. Now, what if the user forgets to enter the output filename? Should we bark or come up with a filename based on the input filename? This of course depends on the situation, but when I decided to create the output filename I would tack today's date to the input filename so they would be different.

But that was Linux and bash and python and this is Windows with powershell. And, yes, we could keep on writing in Bash using cygwin, but that owuld be cheating. Give the constraint of only running what came in Windows 7 and above (I am dating myself), let's see what we can do:

1. Date. The date formats I like are (using Linux here, so focus on the output not the command)

   raub@desktop:~$ date +%F
   2017-01-30
   raub@desktop:~$ 

   and

   raub@desktop:~$ date +%Y%m%d
   20170130
   raub@desktop:~$ 

   Both write year using 4 digits followed by 2 digits for the month and two for the day. I know some people will cry and moan and demand the traditional US format, day/month/year, but the format I like makes sorting in a directory much easier as we are putting what changes the fastest on the end of the filename. But we are talking about powershell, not Bash or Bourne shell. That is true but now we know what we want to accomplish.

   To make it easier, I will pick one of the two formats -- YYYYMMDD -- and run with it; you can later modify the code to use the other one as exercise. The equivalent in powershell is:

   PS C:\Users\raub> get-date -format "yyyyMMdd"
   20170130
   PS C:\Users\raub> 

   Looks just like what we did above in Linux.

2. Create filename if not given. We are reading the input filename into the script in some way or fashion. How we are doing it depends on the script and whether we should be passing options with or without flags to identify them. For now, we are going to be lazy and do the simplest thing possible: using param() on the beginning of the script.

   param($inputFile, $outputFile)

   If we have just one argument, it shall be the inputfile. If two, the second one is the outputfile. What if no arguments are passed? We can just send an error message and get out.

   function Usage
   {
      Write-Host "Usage:", $MyInvocation.MyCommand.Name, "inputfile [outputfile]"
      exit
   }
   
   if (!$inputfile)
   {
      Usage
   }

   The $MyInvocation.MyCommand.Name is a lazy way to for the script to get its own name by itself.

3. Do something if there is no $outputFile. This is a variation of the same test we did to see if we had an $inputFile:

   function LazyOutputFilename($ifile)
   {
      $ofile = (Get-Item $ifile ).DirectoryName + '\' +  `
               (Get-Item $ifile ).BaseName + `
               '_' + (get-date -format "yyyyMMdd") + `
               (Get-Item $ifile ).Extension
      return $ofile
   }
   
   function GetOutputFilename($ifile, $ofile)
   {
      # $ofile cannot be $ifile
      # Create a $ofile if one was not given
      if (( [string]::IsNullOrEmpty($ofile) ) -or ( $ofile -eq $ifile ))
      {  
         $ofile = LazyOutputFilename $ifile
      }
   
      return $ofile
   }
   
   $outputFile = GetOutputFilename $inputFile $outputFile

   • In LazyOutputFilename() we are creating the output filename. We are putting it in the same directory as the input filename and then adding the formatted date right before the file extension.

   • The ( [string]::IsNullOrEmpty($ofile) ) checks is the output file, called $ofile inside this function, is empty. The reason we also wants to make sure the output file is not the input file is because we might be reading the input file a chunk at a time (line by line if text) so the script can handle large files without using up all the memory. If we are reading it line by line and then write right back to it, bad things might happen.

   • And, yes, we are overwriting the output filename if it gets changed in GetOutputFilename().

4. Put everyone together.
   

   param($inputFile, $outputFile)
   
   function Usage
   {
      Write-Host "Usage:", $MyInvocation.MyCommand.Name, "inputfile [outputfile]"
      exit
   }
   
   <#
    Create output filename based on the full path of the input filename +
    today's date appended somewhere
    #>
   function LazyOutputFilename($ifile)
   {
      $ofile = (Get-Item $ifile ).DirectoryName + '\' +  `
               (Get-Item $ifile ).BaseName + `
               '_' + (get-date -format "yyyyMMdd") + `
               (Get-Item $ifile ).Extension
      return $ofile
   }
   
   function GetOutputFilename($ifile, $ofile)
   {
      # $ofile cannot be $ifile
      # Create a $ofile if one was not given
      if (( [string]::IsNullOrEmpty($ofile) ) -or ( $ofile -eq $ifile ))
      {  
         $ofile = LazyOutputFilename $ifile
      }
   
      return $ofile
   }
   
   if (!$inputfile)
   {
      Usage
   }
   
   $outputFile = GetOutputFilename $inputFile $outputFile

Creating extended ASCII file in bash and maybe powershell

Does anyone remember extended ASCII (as opposite to UTF-8)? If you never heard of them, we are not talking about a proper character list that supports Russian or Japanese languages. All we are dealing with here is iso-8859-1, whose table can be found here.

I have a document in that format I need to convert to something else; if this reminds you of some 8bitmime issues we talked about, well, let's just say we could have used this to create the test file. With that said, the current situation is that I wrote a script to manipulate it which is not preserving that format; that we can talk about in a future post.

Bottom line is I need to create a small test file that I can later throw it and the script output on hexdump.

My test file will have only 3 lines,

Olivenöl
Bayerstraße 22
München

Nothing fancy; just enough to use one extended ASCII character per line. Now let's try to create the little file. Just to be different, instead of starting on Linux we will do most of the attempts in OSX. Once we have a working system, we can see if it also works on Linux.

Attempt #1

How about if we do the lazy thing and jus cut-n-paste the 3 lines above into a text file we opened usng vim, notepad++, or some pico clone? Done. Now let's see how it looks like

bash-3.2$ cat /tmp/chartest 
Olivenöl
Bayerstraße 22
München
bash-3.2$ 

That looks very promising. In fact, this might end up being a very short article. Before I publish it, should we see what hexdump thinks of it?

bash-3.2$ hexdump -Cv /tmp/chartest 
00000000  4f 6c 69 76 65 6e c3 b6  6c 0d 0a 42 61 79 65 72  |Oliven..l..Bayer|
00000010  73 74 72 61 c3 9f 65 20  32 32 0d 0a 4d c3 bc 6e  |stra..e 22..M..n|
00000020  63 68 65 6e 0d 0a                                 |chen..|
00000026
bash-3.2$ 

Correct me if I am wrong but it seems each extended ASCII character is taking two characters to be represented instead of just one single character. For instance ö is being represented by two characters, 0xC3B6. That sounds more like UTF-8/Unicode/whatever (if you want to know what to look for, they all start with a 0xC3) but not extended ASCII. Also, it is using carriage return (CR, 0x0D in hexadecimal) and line feed (LF, 0x0A) characters to separate the lines. But this is very Windowsy, not OSX/Linux style, where lines are separated by the line feed (0x0A) character only.

Attempt #2

What if we paste the lines onto the terminal and use echo to write that to the test file? Well, let's make a single line test file and see what happens

bash-3.2$ echo "Olivenöl" > /tmp/chartest 
bash-3.2$ hexdump -Cv /tmp/chartest 
00000000  4f 6c 69 76 65 6e c3 b6  6c 0a                    |Oliven..l.|
0000000a
bash-3.2$ 

Still using two characters to represent ö; at least it did not add a CR. Now this is getting annoying; is there anything else we can try?

Attempt #3

It turns out there is and we do not need any extra stuff. You see, echo has this -e option that allows you to pass a character by its hexadecimal code. From the extended ASCII table we know that ö= 0xF6 and ß= 0xDF (just to pick two examples). We also know that CRFL = 0x0D0A. I know I whined about that before, but the reason is I want to be able to decide when I want to use those characters and when I do not want, as opposite to having some program or script making a choice for me.

Let's try again, this time passing the extended ASCII characters explicitly:

bash-3.2$ echo -e "Oliven\xf6l\x0d\x0aBayerstra\xdfe 22\x0d\x0aM\xfcnchen\x0d" > /tmp/chartest
bash-3.2$ 

bash-3.2$ cat /tmp/chartest 
Oliven�l
Bayerstra�e 22
M�nchen
bash-3.2$ hexdump -Cv /tmp/chartest 
00000000  4f 6c 69 76 65 6e f6 6c  0d 0a 42 61 79 65 72 73  |Oliven.l..Bayers|
00000010  74 72 61 df 65 20 32 32  0d 0a 4d fc 6e 63 68 65  |tra.e 22..M.nche|
00000020  6e 0d 0a                                          |n..|
00000023
bash-3.2$ 

That's more like it: only one character is used to represent each character in the file. Isn't it interesting when we cat the file it is replacing the extended characters with �? But, if hexdump says they are there that is good enough for me.

What about powershell?

Even though the name of this blog implied Unix, we use enough powershell we might as well see if we can do the same. But, we have to accept we start with a bit of a handicap: powershell really really wants to write UTF-8 or unicode instead of extended ascii/iso-8859-1. Let me show you what I mean by trying to create a small file with just one single word on it, Olivenöl. As we seen before, ö = 0xF6 = 246. And that should still be true in powershell; let's find out:

PS > 'Oliven' + [char]246 + 'l'
Olivenöl
PS >

Looks like we are getting somewhere, right? For our next trick, we will save that to a file (| out-file .\chartest.txt is equivalent to doing > .\chartest.txt.

PS > 'Oliven' + [char]246 + 'l' | out-file .\chartest.txt
PS > cat .\chartest.txt
Olivenöl
PS >

Hey chief! It seems to be working fine? Why are you make this huge drama about this? That is a very good question. I will let dear old hexdump do the talking:

$ hexdump -Cv chartest.txt
00000000  ff fe 4f 00 6c 00 69 00  76 00 65 00 6e 00 f6 00  |..O.l.i.v.e.n...|
00000010  6c 00 0d 00 0a 00                                 |l.....|
00000016

$

Each character is now represented by 2 characters. Smells like Unicode, right? ok, smart guy. Now just force it to save as ASCII then. Will do:

PS > 'Oliven' + [char]246 + 'l' | out-file -encoding ASCII .\chartest.txt
PS > cat .\chartest.txt
Oliven?l
PS >

And hexdump

$ hexdump -Cv chartest.txt
00000000  4f 6c 69 76 65 6e 3f 6c  0d 0a                    |Oliven?l..|
0000000a

$

It converted the characters into ?. Helpful, isn't it? The Microsoft Scripting Guys forum pretty much tells you should save file as unicode or UTF-8 and then convert it somehow. Far from me to disagree with them, at least in this article since it makes for a great cliffhanger. In a future article we will talk about how to get extended ASCII properly in powershell just like we did in bash. It will be a bit longer but doable.

Creating a fixed length blank(spaces) string and padding others with spaces in powershell

Powershell has clever ways to create an empty string or string array that can then be increased in size. And then I need a fixed length empty string. Why? I need to create a file where each line is composed by fields which are position dependent. So I need to pad between the fields with blank spaces.

After searching for quite a while I found Ethan's reply on the bottom of thread on limiting a string to N characters. And it was so obvious I had to smack me on the head: if you want to create a string called moo that contains nothing but 24 spaces, you can do it by typing

PS C:\> $moo = " "*24
PS C:\> $moo.length
PS C:\> 24

So let's do something useful with that then!

So the plan is to write a function that given the string that will be used in a given field and the field size, it will return a properly padded string. Now, I think we should also create a second function whose only goal in life is to create a string with N blank spaces. And then the other function can call this. Reason I propose this is in case we need to provide some extra blank spaces because the format of this file might be, well, rather silly. Planning ahead is always a smart move. So, we start with our blank string generating function, which will look very similar to the $moo example above:

function BlankString($size)
{
   return " "*$size
}

If we want to test that we can modify the moo example

$moo = BlankString(24)
$moo.length

Which should return 24 as before. So far so good. Now let's create a function that will return a properly padded field. Something like

function PadString($name, $stringSize)
{
   if($name.length -lt $stringSize)
   {
      $name += BlankString ($stringSize - $name.length)
   }
   return $name
}

looks like a good starting place. Let's try it out:

$noo = PadString "My Head hurts" 48
DebugLine "[$noo]"
$noo.length

Results in

[My Head hurts                                   ]
48

If you are curious, my DebugLine function looks like this

function DebugLine($OutputMessage)
{
   Write-Host $OutputMessage -foregroundcolor red -backgroundcolor yellow
}

The [ and ] are there just so we can see where the string, blank spaces included, begins and ends. If you do not like those characters, change them to something more remarkable, say ---> and <---

Now, what if the string we are trying to put in this new string is bigger than it can handle? This would be the opposite of padding the new string: we need to chop it. So, we modify our PadString function a bit:

function PadString($name, $stringSize)
{
   if($name.length -lt $stringSize)
   {
      $name += BlankString ($stringSize - $name.length)
   }
   else
   {
      $name = $name.Substring(0, $stringSize)
   }
   return $name
}

So, if we redo the same test we did before, but this time we make our resulting string shorter,

$noo = PadString "My Head hurts" 8
DebugLine "[$noo]"
$noo.length

the result is now truncated

[My Head ]
8

And we can still create weird blank spaces using BlankString as needed to meet the file format requirments.

Get info on installed packages using Powershell (and maybe also WMIC)

Warning: Expect this article to be rather vague

From what I understand, when you install a program in Windows using some kind of installing package, it writes some info about the program in the registry. And that allows us to go to the control panel and see the program being listed there.

You are probably waiting for me to ask the question I seem to ask a lot here: Can we do it from command line? Good question! First candidate as the command of choice is wmic; it really allows you to get (and set) a lot of info about the computer. Let's use some examples to see what we can do: we shall begin by asking it which CPU we have

C:\Users\raub\dev> wmic cpu get name
Name
Intel(R) Core(TM) i5-4570T CPU @ 2.90GHz

C:\Users\raub\dev>

Which bios version do we have?

C:\Users\raub\dev> wmic bios get version
Version
LENOVO - 14b0

C:\Users\raub\dev>

What about who made the motherboard?

C:\Users\raub\dev> wmic baseboard get manufacturer
Manufacturer
LENOVO

C:\Users\raub\dev>

From the previous command, we kinda expected this to be a Lenovo. So, let's call this verifying that we are in fact talking to a Lenovo. Now, would it tell us which motherboard is in the machine?

C:\Users\raub\dev> wmic baseboard get model
Model


C:\Users\raub\dev>

So it is hiding info from me. Bastard! Let's see what else we can find out about the motherboard then:

C:\Users\raub\dev> wmic baseboard get /?

Property get operations.
USAGE:

GET [<property list>] [<get switches>]
NOTE: <property list> ::= <property name> | <property name>,  <property list>

The following properties are available:
Property                                Type                    Operation
========                                ====                    =========
ConfigOptions                           N/A                     N/A
Depth                                   N/A                     N/A
Description                             N/A                     N/A
Height                                  N/A                     N/A
HostingBoard                            N/A                     N/A
HotSwappable                            N/A                     N/A
InstallDate                             N/A                     N/A
Manufacturer                            N/A                     N/A
Model                                   N/A                     N/A
Name                                    N/A                     N/A
OtherIdentifyingInfo                    N/A                     N/A
PartNumber                              N/A                     N/A
PoweredOn                               N/A                     N/A
Product                                 N/A                     N/A
Removable                               N/A                     N/A
Replaceable                             N/A                     N/A
RequirementsDescription                 N/A                     N/A
RequiresDaughterBoard                   N/A                     N/A
SKU                                     N/A                     N/A
SerialNumber                            N/A                     N/A
SlotLayout                              N/A                     N/A
SpecialRequirements                     N/A                     N/A
Status                                  N/A                     N/A
Tag                                     N/A                     N/A
Version                                 N/A                     N/A
Weight                                  N/A                     N/A
Width                                   N/A                     N/A

The following GET switches are available:

/VALUE                       - Return value.
/ALL(default)                - Return the data and metadata for the attribute.
/TRANSLATE:<table name>      - Translate output via values from <table name>.
/EVERY:<interval> [/REPEAT:<repeat count>] - Returns value every (X interval) seconds, If /REPEAT specified the command
is executed <repeat count> times.
/FORMAT:<format specifier>   - Keyword/XSL filename to process the XML results.

NOTE: Order of /TRANSLATE and /FORMAT switches influences the appearance of output.
Case1: If /TRANSLATE precedes /FORMAT, then translation of results will be followed by formatting.
Case2: If /TRANSLATE succeeds /FORMAT, then translation of the formatted results will be done.

PS C:\Users\mtavares\dev>

From the above, we could run

wmic baseboard get /all

to see all the properties and their current values. It does not look very pretty unless you format it (option /FORMAT), but you can see it has a lot of potential. In fact, here is a list of other interesting queries you can do in wmic.

All that is great, but what about packages since that is the subject of this article? Another very good question. Let's pick an example from the control panel and see if we can find it using wmic. The example I will use is the TightVNC, whose publisher is GlavSoft LLC:

C:\Users\raub> wmic product where "vendor like '%%glav%%'" get name
Name
TightVNC

C:\Users\raub>

Real life example

It is real life but I changed the name to protect the guilty

Let's try it with a program I am interested on. We use a program called Careless Data, which is produced by Cargo Cult Development. According to the control panel, it consists of the following packages:

• Careless Client: The program the users run to connect to the Careless Server.
  
• Careless Data: Data from Careless Server that is temporarily stored in an unencrypted flat file.
  
• Careless Upgrades: The upgrade package that brought it to version 12.3.4 r3. Well, the Careless Client binary was also upgraded.
  
• Careless Dependencies: Support libraries for the Careless Client. It is actually seen as two distinct packages, CarelessDependencies and CarelessDependenciesMSI

Let's see if we can find all those packages using wmic:

C:\Users\raub>  wmic product where "vendor like '%%cargo%%'" get name
Name
CarelessDependenciesMSI
CarelessData

C:\Users\raub>

Houston we have a problem: three out of five the packages are missing. What is going on here?

Enter Powershell

So I got annoyed and decided to go back to what I know more: Powershell. Short version (this article is getting long): I knew the information about installed packages was in HKLM:\Software, so I searched for a few installed packages and found some info in

• HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall
  
• HKLM:\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\U‌​ninstall

Here is an example

PS C:\Users\raub> get-itemproperty -path 'HKLM:\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{09DA5EE2-7E46-4DC4-96F9-BFEE50D40659}' 


PSPath       : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{09DA5EE2-7E46-4DC4-96F9-BFEE50D40659}
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall
PSChildName  : {09DA5EE2-7E46-4DC4-96F9-BFEE50D40659}
PSDrive      : HKLM
PSProvider   : Microsoft.PowerShell.Core\Registry
DisplayName  : Citrix Online Launcher
[...]


PS C:\Users\raub>

I know I did not show it above but one of the attributes we can get, besides PSPath and DisplayName, is UninstallString. so, I think we could write a script that would search those paths for programs and fetch for some attributes. Let's use the vendor name as the search criteria:

param($vendor)

$locations = ("software\microsoft\windows\currentversion\uninstall", "software\Wow6432Node\Microsoft\Windows\CurrentVer
sion\Uninstall")
$count = 0

foreach ($location in $locations)
{
   foreach ($obj in (get-childitem "hklm:\$location" | get-itemproperty | where {$_.publisher -match $vendor} ))
   {  
      write-host "Name: $($obj.displayname)"
      write-host "   PSChildName: $($obj.PSChildName)"
      write-host "   PSPath $($obj.PSPath)"
      write-host "   Publisher: $($obj.Publisher)"
      write-host "   Version: $($obj.DisplayVersion)"
      write-host "   BuildVersion: $($obj.Version)"
      write-host "   UninstallString: $($obj.UninstallString)"

      $global:count++
   }
}

write-host "Done"
write-host " $($count) entries found"

Now let's see what we can find about our careless program:

.\findinstalledprogram.ps1 cargo
Name: Careless Client 12.3.4
   PSChildName: Careless Client 12.3.4_r3
   PSPath Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\Careless Client 12.3.4_r3
   Publisher: CargoCultDevelopment LLC.
   Version:
   BuildVersion:
   UninstallString: "C:\Windows\unins000.exe"
Name: CarelessDependenciesMSI
   PSChildName: {45B39321-107B-4F40-A7DC-A4CB4BFC3051}
   PSPath Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{45B39321-107B-4F40-A7DC-A4CB4BFC3051}
   Publisher: CargoCultDevelopment
   Version: 1.00.0000
   BuildVersion: 16777216
   UninstallString: MsiExec.exe /I{45B39321-107B-4F40-A7DC-A4CB4BFC3051}
Name: CarelessDependencies
   PSChildName: {4509C469-6B21-4999-BE60-3449EE7B6EDF}
   PSPath Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{4509C469-6B21-4999-BE60-3449EE7B6EDF}
   Publisher: CargoCultDevelopment.com LLC
   Version: 1.00.0000
   BuildVersion: 16777216
   UninstallString: MsiExec.exe /I{4509C469-6B21-4999-BE60-3449EE7B6EDF}
Name: CarelessUpgrades
   PSChildName: {E20A19AD-27C1-4F59-AED2-A04636BD2575}
   PSPath Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{E20A19AD-27C1-4F59-AED2-A04636BD2575}
   Publisher: CargoCultDevelopment LLC.  
   Version: 12.3.4
   BuildVersion: 167772240
   UninstallString: MsiExec.exe /I{E20A19AD-27C1-4F59-AED2-A04636BD2575}
Name: CarelessData 12.3.4 r3
   PSChildName: {EA737DCD-71C9-4a06-97B5-BB2D4EE45564}_r3
   PSPath Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{EA737DCD-71C9-4a06-97B5-BB2D4EE45564}_r3
   Publisher: CargoCultDevelopment, LLC
   Version:
   BuildVersion:
   UninstallString: "C:\Program Files (x86)\CargoCultDevelopment\unins000.exe"
Done
 5 entries found

Back to WMIC and why you should not use it

Let's revisit using wmic again.

What is happening is that our syntax is using a WMI class, Win32_Product, that only displays products installed using Windows Installer. Now, we really should not be using Win32_Product because querying Win32_Product is quite dangerous because every time you run a query (wmic product is really a macro to wmic "select * from Win32_Product"), it will cause a consistency check of all the packages installed, leading to it trying to be helpful and trying to verify and repair the install. And that opens the door to some corruption. Microsoft suggests to use Win32reg_AddRemovePrograms, which may not show everything from wmic.

Bottom line, wmic is great but looking for package info is better done by Powershell.

Poor Man's Downconverting UTF-8 to ASCII7 in powershell

I will warn you this is really lousy and inelegant code. But I needed something quick. I will also warn that this was going to be focus primarily on doing the UTF-8 to ASCII7 conversion, but it went a bit sideways with me learning something new about Powershell. If you feel a bit disappointed, oh well.

I will do my best take the high road and not bitch and make snide comments about a certain software company whose product caused me to write this script. However, after rereading this article I think I failed miserably. You have been warned!

The Problem

So, I am dealing with yet a not so bright piece of software. Specifically, one that has a database whose fields for usernames, addresses, and even phone numbers and such things assume that those users are all living in the United States and have names like Bob or Sue or John or Travis (OK, maybe not that one since it has two syllables) who lived in towns like Cleveland or Tampa. This vendor developed the code in early 2000(!) hoping it would never have to deal with umlauts and accents and other characters. Probably they did that to save space since an ASCII7-only DB field is smaller than a UTF-aware one. Imagine how much money they would make their customers save on storage!

Then someone told their customers that there are lands past the dragons and the turtles, where strange people with mysterious habits and incomprehensible languages roam.

Like Sweden.

Imagine how shocked they were when now their software had to deal with names like Фёдор, Ljungström, Simón, 宮崎, and Häyhä and towns like Malmö. To address that, in 1987 a formal work group, which later became the Unicode Consortium was formed and, anfter much head-smacking, released its first version in 1991. Unfortunately, someone failed to mention that to this company. Fear not, however, for they were not the first and will not be the last to develop their code in ASCII and then realize they have to support other languages.

This is all nice and all, but we feed data to this program as text files. Which is in UTF-8 format since it contains people and places with names using non-ASCII characters. And as a result the database gets very unhappy.

The Solution

I called them to see if they had plan on supporting Unicode in the future. And they replied. Bottom line is they will not change their code to handle Unicode. So, if we are going to use this program we need to massage the input data.

As many times before in many different operating systems, if we are going to do this, it better be doable using some kind of scripting language. Being this Windows my language of choice will once again be Powershell. Before we write a single line of code, let's do something dangerously subversive and think about what we need to do.

1. We need to detect the non-ASCII characters. Specifically we are talking about the characters that are not within the original 7bit ASCII (characters 0 to 127 in decimal, or 0x00 and 0x7F respectively in hexadecimal) table; we talked a bit before about detecting different files formats. From there we know the UTF-8 characters we are looking for are represented by a pair: "ö" has a UTF8 Hex of 0xc3b6. Do note both characters > 0x7F.

2. Once we detect said character pair, we need to do something with them. We are going to convert them into a single or a combination of characters, and that is where it gets interesting. Take our friend ö: should it become o or oe? And the answer is it depends.

   Some countries came up with their own representation in ASCI7 of their non-ASCI7 characters. And they are not really universal. Also, if the import file uses some kind of fixed column formatting, we have to convert it to one single character. This is really not a coding problem; we can easily create a lookup table to replace the character. So I will just leave this here in the open for you to ponder deeply about. In this article we will cheat, and badly.
   

Let's talk about that cheating. Instead of bothering to do some proper converting, all we will do is replace the non-ASCII7 char pair with a single question mark (63 or 0x3F). This way, whoever sees the data will know we did something there and can (if they are the affected user) replace that "?" with whatever character they feel like. Talk about passing the hot potato around...

So, if we find a character whose decimal value is > 127, we can replace it with a "?" (decimal 63). Of course, being it UTF-8 the next character in the pair is also > 127. So, we need to replace the pair of characters with a single "?".

With that in mind, let's try coding that:

param($infile) 

# Convert UTF-8 document into ASCII7
function UTF2ASCII($encoded)
{
   $oldi = ''
   $converted = ''

   foreach ($line in $encoded)
   {
      # Convert each character in $line into a decimal number
      foreach($i in [int[]][char[]]$line)
      {
         # 0 <= ASCII7 <= 127
         if ($i -gt 127)
         {
            # KLUDGE: only change to "?" if previous char isn't.
            if ($oldi -ne 63)
            {
               $i = 63
            }
            else
            { continue }
         }
         $converted = $converted + [char]$i
         $oldi = $i
      }

   }

   return $converted
}

# Read in the contents of $infile and feed them to our function
UTF2ASCII(get-content $infile)

The if statements are the lousiest part of the code, specially the part that assumes if the previous char is "?" we must be dealing with the two-bit representation of the UTF-8 character; that is a rather tall assumption but I was in a hurry and could not think of an edge case. Now that I admitted my kludge, let's test it. The way you run this script is

utf2ascii.ps1 infile

which will output the modified text to the screen. If we want to save it into a file,

utf2ascii.ps1 infile > outfile

We also need a test file. I created it by gabbing a few paragraphs from the wikipedia page on the Automatgevär m/42 plus an extra line:

Olivenöl

Ag m/42, kitaip AG42, AG-42 ar Ljungman (šved. Automatgevär m/42) – švediškas savitaisis šautuvas, nuo 1942 m. iki 7-ojo dešimtmečio naudotas Švedijos armijos.

Šautuvą 1941 m. sukonstravo Erikas Eklundas iš kompanijos AB C. J. Ljungmans Verkstäder (Malmė, Švedija). Juos gaminti 1942 m. ėmė kompanija Carl Gustafs Stads Gevärsfaktori (Eskilstuna, Švedija). Švedijos armijai buvo pagaminta apie 30 000 Ag m/42. Tačiau šis šautuvas nebuvo pagrindinis Švedijos armijos šautuvas, juo buvo 6,5 mm pertaisomas šautuvas Mauser m/96.

Po kurio laiko buvo aptikti kai kurie šautuvo trūkumai (pvz., dujų vamzdelio rūdijimas), todėl 1953–1956 Švedijos armijos šautuvai Ag m/42 buvo modernizuoti ir pavadinti Ag m/42B. Modernizaciniai pakeitimai:

cat utftest.txt | wc -l (yes it is Linux, sue me) tells me that file is 7 lines long. So, let's run it and see what happens.

$ utf2ascii.ps1 infile
Oliven?l

Ag m/42, kitaip AG42, AG-42 ar Ljungman (?ved. Automatgev?r m/42) ? ?vedi?kas s
avitaisis ?autuvas, nuo 1942 m. iki 7-ojo de?imtme?io naudotas ?vedijos armijos
.

?autuv? 1941 m. sukonstravo Erikas Eklundas i? kompanijos AB C. J. Ljungmans Ve
rkst?der (Malm?, ?vedija). Juos gaminti 1942 m. ?m? kompanija Carl Gustafs Stad
s Gev?rsfaktori (Eskilstuna, ?vedija). ?vedijos armijai buvo pagaminta apie 30
000 Ag m/42. Ta?iau ?is ?autuvas nebuvo pagrindinis ?vedijos armijos ?autuvas,
juo buvo 6,5 mm pertaisomas ?autuvas Mauser m/96.

Po kurio laiko buvo aptikti kai kurie ?autuvo tr?kumai (pvz., duj? vamzdelio r?
dijimas), tod?l 1953?1956 ?vedijos armijos ?autuvai Ag m/42 buvo modernizuoti i
r pavadinti Ag m/42B. Modernizaciniai pakeitimai:
$

Looks great, right? Not quite. Problem with the above code is the following line:

UTF2ASCII(get-content $infile)

It just tries to read the entire file into memory in one sitting and then passes all that to UTF2ASCII(). That works fine with small files, but we will be filling memory/swap space and maybe even running out of them if the file is big enough. And that is a bit of shooting ourselves in the foot. Let's rewrite it a bit by creating a second function, ConvertFile(), which reads the file line by line. This way we need not to care how large the file is.

param($infile)
$count = 0

function ConvertFile($encoded)
{
   $oldi = ''
   $converted = ''

   foreach ($line in $encoded)
   {
      ConvertLine($line)
   }
}

function ConvertLine($line)
{
   foreach($i in [int[]][char[]]$line)
   {
      if ($i -gt 127)
      {
         if ($oldi -ne 63)
         {
            $i = 63
            $global:count++
         }
         else
         { continue }
      }
      $converted = $converted + [char]$i
      $oldi = $i
   }

   $converted = $converted + "`r`n"
   return $converted
}

ConvertFile(get-content $infile)
"$($count) characters were converted"

The $count global variable is just there to count how many times the script found and converted UTF-8 characters.

When we test the script with a large file, and found out we did solve the problem in the original code we mentioned above; you can try it out if you do not believe in me.

So, let's now do a few more tests.

$ utf2ascii.ps1 infile > outfile
$ cat outfile
Oliven?l

Ag m/42, kitaip AG42, AG-42 ar Ljungman (?ved. Automatgev?r m/42) ? ?vedi?kas s
avitaisis ?autuvas, nuo 1942 m. iki 7-ojo de?imtme?io naudotas ?vedijos armijos
.

?autuv? 1941 m. sukonstravo Erikas Eklundas i? kompanijos AB C. J. Ljungmans Ve
rkst?der (Malm?, ?vedija). Juos gaminti 1942 m. ?m? kompanija Carl Gustafs Stad
s Gev?rsfaktori (Eskilstuna, ?vedija). ?vedijos armijai buvo pagaminta apie 30
000 Ag m/42. Ta?iau ?is ?autuvas nebuvo pagrindinis ?vedijos armijos ?autuvas,
juo buvo 6,5 mm pertaisomas ?autuvas Mauser m/96.

Po kurio laiko buvo aptikti kai kurie ?autuvo tr?kumai (pvz., duj? vamzdelio r?
dijimas), tod?l 1953?1956 ?vedijos armijos ?autuvai Ag m/42 buvo modernizuoti i
r pavadinti Ag m/42B. Modernizaciniai pakeitimai:
35 characters were converted
$

Yeah, the fact the line stating how many characters were converted is in the outfile is a drag, but I will survive. I really want to count the number of lines though:

$ cat outfile | wc -l
16

$

Now that was unexpected to me, specially coming from a Linux background. How did my 7 lines became 16? You see, we run the script as

utf2ascii.ps1 infile > outfile

Thing is, in Powershell the > will split a long line to the width of the console window we are using. I read the explanation quite a few times and still am not satisfied. But, since we need this code to work, time to change it one more time!

param($ifile, $ofile)
$count = 0

function ConvertFile($originalFile, $convertedFile)
{
   $oldi = ''
   $converted = ''
   rm $convertedFile -ea SilentlyContinue

   foreach ($line in (get-content -path $originalFile))
   {
      ConvertLine($line) | `
      Out-File -Append -Encoding ASCII $ofile
   }
}

function ConvertLine($line)
{
   foreach($i in [int[]][char[]]$line)
   {
      if ($i -gt 127)
      {
         if ($oldi -ne 63)
         {
            $i = 63
            $global:count++
         }
         else
         { continue }
      }
      $converted = $converted + [char]$i
      $oldi = $i
   }

   return $converted
}

if ( [string]::IsNullOrEmpty($ifile) )
{
   Write-Host "Usage: powershell -ExecutionPolicy ByPass -File", `
               $MyInvocation.MyCommand.Name, "infile [outfile]"
   exit
}

# Create a $ofile if one was not given
if ( [string]::IsNullOrEmpty($ofile) )
{
   $ofile = ([io.path]::GetFileNameWithoutExtension("$ifile")) + `
            (get-date -format yyyymmdd) + `
            ([io.path]::GetExtension("$ifile"))
}

ConvertFile $ifile $ofile
"$($count) characters were converted"

Notes:

• Instead of sending output to console, which then would need to be redirected to a file, this script reads the input and output filenames. If an outputfilename is not provided, it will create one based on that of the inputfile.
  
• Corrected the issue of splitting the processed lines before saving them.
  
• Gives usage message if no parameters are given
  
• Outputfile is overwritten without questioning.
  
• Only output to the stdout is the number of conversions
  

I am bored, let's run it

$ utf2ascii.ps1 infile outfile
35 characters were converted

$

Note that it now wrote the converted character count message to the screen as promised. How about the number of lines?

$ cat outfile | wc -l
7

$

Much better! Now all we need to do is decide if we want to keep the "?" or make some conversion table, which I will leave the decision up to you.

Killing the VMWare vSphere Client from command line

Hope this will be a quick article; just putting it here so I will know how to do it next time. It took a bit longer to post it since I forgot to do the screen capture. Shame on me!

First of all, I will admit I use VMWare's vSphere client on occasion (at least enough to remember how they like their camelcase to be). I know vmware wants us to use the web interface (they even said all new features will be going to the web version), but there are times the C#-based vsphere client is more convenient even though it only runs on Windows. This convenience, however, is not flawless. There are times it might crash and crash horribly. Like now. I needed to access a vm client's console window because it was refusing to let me RDP into it.

Actually, we need to back this out a bit: I had first to remote into the windows desktop where I have the vsphere client so I could then have console access to the problematic vm. You see, we keep this special desktop in a separate vlan that can connect to the vm server. I am making a point to explain this because it might be important down the line.

When I launched the vsphere client, as soon as I passed the window where it asks for my authentication it barked about certificates (it does that all the time; I did not set that vm server up, but will talk about deploying certs in a future article) as shown on the picture to the right. And then it decided to freeze: I could not get out of the modal dialog box (the error message window), which would hover over every other window on the desktop. I could click on the window to my heart's content and it would never get focus, which in Plain English means I could not click on any of its buttons.

Usually this is a pain in you-know-exactly-what, but when you are remoting into a machine to run that problem, it is even more so. The standard way to deal with this is to run the task manager, which you can get by pressing the famous 3-finger salute (no, not that one), CTRL-ALT-DEL. Sometimes you can't send that key sequence out, like in my case who was connecting from a MacBookPro and was having a bit of an issue passing that to the Windows desktop. So, the next option is to do what I like to do anyway (translation: I could not be bothered to figure out how to send CTRL-ALT-DEL), which is the command line. Typing

taskmgr

on the DOS or PowerShell prompt will run the task manager GUI. Once it is up, find vmware client under the applications list,

And click on the End Task button. But, I think we can do better than that. We have the terminal window (DOS or Powershell) open; why not do it all there? Let's see first if we can list the applications (tasks) that are being run (I used more because I was using a powershell console; you could just run the program and then scroll until you find what you want):

PS C:\Users\dr-gori> tasklist|more

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
System Idle Process              0 Services                   0         24 K
System                           4 Services                   0         68 K
smss.exe                       324 Services                   0      1,000 K
csrss.exe                      480 Services                   0      6,704 K
csrss.exe                      572 Console                    1     25,272 K
wininit.exe                    580 Services                   0      1,744 K
winlogon.exe                   636 Console                    1      4,912 K
services.exe                   684 Services                   0     19,176 K
[...]
vmware-usbarbitrator64.ex    12560 Services                   0      8,016 K
audiodg.exe                   9008 Services                   0     18,608 K
taskmgr.exe                  15548 Console                    1     11,528 K
VpxClient.exe                 4800 Console                    1    169,556 K
vmware-vmrc.exe               2520 Console                    1     38,836 K
tasklist.exe                  9412 Console                    1      6,040 K

PS C:\Users\dr-gori> 

Yes, this shows more than just the applications. In fact, it feels more like when you list the processes being run in Linux by typing, say

ps -aux

The only thing we have to know is that the name of the vsphere client here is VpxClient.exe; don't look at me like that: I didn't come with that name. So, just like in Linux (and Unix in general), we can see VpxClient.exe has a process ID (PID) of 4800. Now, if Microsoft is calling it process ID, why they are calling the process a task? But I digress. In any case, let's kill the process, er task:

PS C:\Users\dr-gori> tskill 4800
PS C:\Users\dr-gori> 

And away the client went. Still need to figure out what is going on with it, but that will be for another time. Right now I can use the web-based client, which is not tied down to one single operating system, and do what I need to do.

Badlock Microsoft

It is 5 days since this happened but I am not over that. So, today I am doing some venting.

So on April 12, after weeks of hype-building, Microsoft finally announced details about the Badlock vulnerability. Before that, all we knew (just ask Naked Security) was its cute name and Heartbleed-derived logo. Yes I put it on the right since I do not want to focus on it. And then the stampede to patch servers -- both Windows and those running Samba -- began.

Before someone gets excited, beginning on OSX 10.7 Apple decided to use its own SMB stack, smbx, which should not be affected. Of course, "should" is an interesting word.

Back to the topic, Badlock in the Microsoft website is known as MS16-047 and consists of

• KB3148527 for more detailed (?) description of the issue and
  
• KB3149090 for the patch itself.
  

I too had to worry about patching Windows servers.

The machine in question is a Windows Server 2008 box. Yes, I know, it is ancient (I was going to mention I know places still running Windows 2003 server and XP desktops, but I better not scare anyone) but still supported by Windows. And, that is what I had to play with. More importantly, that is the one I remembered to do a screen capture, which is shown here.

As some of you know, I do not like clickety-click interfaces unless they let me run and query them using scripts. And I do not know how to do that in for that window, so here is how it looks like from the command line (in case you want to check whether a specific patch has been installed):

wmic qfe list brief | Select-String -pattern "3149090"
Security Update               KB3149090               DR-ZAIUS\Administrator  4/12/2016                                       

OK, it does not show all the info on the windows since I used the brief option, but it does verify said KB has been applied. So now we can go back to the dialog box. It shows a url, http://support.microsoft.com/kb/3149090/. When I applied said patch on April 12, the Badlock day, it went nowhere. As in page not found. Now it resolves to https://support.microsoft.com/en-us/kb/3149090

For some of you this might not sound a problem, but they have been announcing that April 12 was the Badlock Day for how long? And when it comes around you can't even go to the site and download the patch (using Windows upgrade worked tough)? This feels a bit like amateur hour to me. Microsoft, you had lots of time to get it ready, so no excuse.

Ok, venting is complete. Back to the normal programming.

Fun Fact

Now, if we jump into our DeLorean (or a more environmentally conscious time machine), we can go back to 2009 and find an interesting post from Microsoft called Credential Relaying Attacks on Integrated Windows Authentication. I could go and make the case that the issues it raises are suspiciously similar to those in badlock, but I would rather not. Instead, I would suggest you to take a look at the suggestions; they might be useful.

Listing software versions in SCCM

And here is another windows-related posting. So I needed to look for which Windows computers had a given program and which version they had installed. Since over here they use SCCM, I thought it might be something right up its alley. And it was.. after a fashion.

So, under \Monitoring\Overview\Queries it has a query called "All Systems with Specified Product Name and Version,"

which when you run it asks for the, well, product name and version and then returns the hosts that match that. So, when you click it, it pops a window

Where you enter the product name. Once you press "OK" then it asks for the product version. After that it should run and spit out the hosts matching the criteria and some other info, which we do not know what yet since we are good kids and did not look under the skirt.

Thing is, it returned nothing.

Hmmm, maybe I did not enter the proper data. So how about if we pick a program that we know to be installed in at least some of the machines? First we need to see what it looks like under SCCM:

What we are interested on are

• Product Name = Mozila Firefox 41.0.1 US(x86 en US)
  
• Product Version = 41.0.1

Don't ask me why Mozilla decided to put the Product Version in the Product Name; I've asked them before and never had an answer. Just between you and me, it really does annoy me because it caused some issues deploying packages using puppet. That said, I think we should save that conversation for another time.

I think we have been good children for long enough. Time for some skirt pulling! And here is how the query looks like (I did some pretty formatting so we can see what is going on):

select distinct 
   sys.Name, 
   sys.SMSAssignedSites, 
   sys.OperatingSystemNameandVersion, 
   sys.ResourceDomainORWorkgroup, 
   sys.LastLogonUserName, 
   sys.IPAddresses, 
   sys.IPSubnets, 
   sys.ResourceId, 
   sys.ResourceType 
from
   SMS_G_System_SoftwareProduct as prod 
   inner join SMS_R_System as sys 
on 
   sys.ResourceId = prod.ResourceID 
where 
   prod.ProductName like ##PRM:SMS_G_System_SoftwareProduct.ProductName## 
and 
   prod.ProductVersion like ##PRM:SMS_G_System_SoftwareProduct.ProductVersion##

The ##PRM:SMS_G_System_SoftwareProduct.ProductName## means that it will stop and ask for the SMS_G_System_SoftwareProduct.ProductVersion, which means we can have someone run the query and enter the search data without needing to mess with the code. So, we can see what they were trying to accomplish even though it does not work. So, let's do some thinking with our fingers try something else:

select 
distinct 
   SMS_R_System.Name 
from  
   SMS_R_System 
   inner join SMS_G_System_INSTALLED_SOFTWARE 
on 
   SMS_G_System_INSTALLED_SOFTWARE.ResourceId = SMS_R_System.ResourceId 
where 
   SMS_G_System_INSTALLED_SOFTWARE.ARPDisplayName like "Mozila Firefox%"

And that works! It prints out a list of hostnames which have that product. I will not be able to show the screen capture since it would show the name of production machines instead of my lab. So you will have to have some faith. While we are at it, how about if we add more useful info? I don't know about you, but I would like to know which version of the software is installed and when that installation (or upgrade) took place. I will not not comment on the extra attributes we are getting back because I think you can figure out

select 
distinct 
   SMS_R_System.Name,
   SMS_G_System_INSTALLED_SOFTWARE.ProductVersion,
   SMS_G_System_INSTALLED_SOFTWARE.InstallDate
from  
   SMS_R_System 
   inner join SMS_G_System_INSTALLED_SOFTWARE 
on 
   SMS_G_System_INSTALLED_SOFTWARE.ResourceId = SMS_R_System.ResourceId 
where 
   SMS_G_System_INSTALLED_SOFTWARE.ARPDisplayName like "Mozila Firefox%"

We could make this interactive by changing

   SMS_G_System_INSTALLED_SOFTWARE.ARPDisplayName like "Mozila Firefox%"

to

   ##SMS_G_System_INSTALLED_SOFTWARE.ARPDisplayName##

Incidentally, it seems that SMS_G_System_INSTALLED_SOFTWARE.ARPDisplayName gave the same result as SMS_G_System_INSTALLED_SOFTWARE.ProductName, but that does not mean that will always be the case.

md5 checksum in powershell

It seems I have been stuck in Windows recently. The main annoyance with that is it loves to fight me and hates to tell me what is going on. As a result, sometimes I need to write my own tools. And, yes, they do tend to emulate what I have used in Linux. Take for example md5sum, an example of a command that just happens to be related to the topic of this article.

Here is how md5sum works in Linux:

raub@desktop:~$ md5sum disallowedcert.sst
6e8bf95470e7f1ca0626012ea21e55f7 disallowedcert.sst
raub@desktop:~$

As the man page tells us, there are other options we can use. But, I do not use most of them. So, how about if we write a quick powershell script to emulate at least the basic usage of md5sum?

Warning

The md5 should not be used for, say, verify integrity of a file if you care about security; the MD5 algorithm can be hacked using collision attacks.

The code

Here is a first shot at the code

# emulate md5sum in powershell
foreach ($file in $args)
{
   $md5 = New-Object -typeName `
          System.Security.Cryptography.MD5CryptoServiceProvider
   $hash = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($file)))
   echo "$hash $file"
}

PS C:\Users\User\test> powershell -executionpolicy bypass -file .\md5.ps1 .\disallowedcert.sst
6E-8B-F9-54-70-E7-F1-CA-06-26-01-2E-A2-1E-55-F7
PS C:\Users\User\test> powershell -executionpolicy bypass -file .\md5.ps1 .\disallowedcert.sst .\e098
cf355c19953274d84772a1cec96dc3356ca.crt
6E-8B-F9-54-70-E7-F1-CA-06-26-01-2E-A2-1E-55-F7
1C-BC-A2-83-77-BA-25-04-3D-6D-42-6E-E2-11-34-55
PS C:\Users\User\test>

Let's improve on it a bit then

# emulate md5sum in powershell
foreach ($file in $args)
{
   $md5 = New-Object -typeName `
          System.Security.Cryptography.MD5CryptoServiceProvider
   $hash = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($file))) `
           -replace "-",""
   echo "$hash $file"
}

Let's try it out

PS C:\Users\User\test> powershell -executionpolicy bypass -file md5.ps1 md5sum.ps1 ntpTime.ps1 
F1930269F287A434993172A362269F8E md5sum.ps1
FC6ED0787947524543D34CFBA3BE641A ntpTime.ps1

PS C:\Users\User\test> 

That's much more like it.

Other versions

If you do not like my script there are a few versions out there you can download and run instead. Pick your poison!

Sending mail in powershell (poor man's ssmtp in Windows)

A lot of times in a server you want to have a service be able to send and email. Sometimes this service is nothing but a script you wrote. As mentioned in a previous post, sometimes you just want a lightweight solution. In Linux I like to use ssmtp, but it is not available in Windows. So let's see if we can come up with the next best thing.

If you know me or my posts long enough, this is usually the cue for Powershell. And you would be right: it does provide the function send-mailmessage that allows you to craft an email which can then be sent to a proper SMTP.

Here's a quick proof of concept that I called mailtest.ps1:

$touser = "That guy <80sguy@email.example.com>"
$hostname = hostname
$SMTP = "smtp.example.com"
$fromuser = "Me "
$body = "Nothing to see. Move along"

send-mailmessage -to $touser `
                 -from $fromuser `
                 -subject "Test mail from $($hostname)" `
                 -body $body `
                 -SmtpServer $SMTP

All it does is send a test mail to that guy from the machine we are currently at. It does grab the hostname and creates a return address based on that. I did that because in a production script I like to know where my email came from. How would I used it in a production system? Let's say I have a script that monitors the default gateway and let's me know if it changed (long story). In that script I might have something like

$touser = "Me "
$hostname = hostname
$SMTP = "smtp.example.com"
$fromuser = "Gateway monitoring on $hostname "

function Send-Mail ($message) {
    send-mailmessage -to $touser `
                     -from $fromuser `
                     -subject "Default Gateway on $($hostname) Changed!!!" `
                     -body $message `
                     -SmtpServer $SMTP 
}

Now, we have a function that sends an email to Me indicating the default gateway has changed. It could be called by something like this

function Verify-Gateway($originalGateway){
    [...]
        if( $currentGateway -ne $originalGateway ){
            $Card.SetGateways($originalGateway)
            $message = @"
$hostname lost its default gateway. 
It thinks it is $currentGateway, but it should 
be $originalGateway. So, we changed it back.
"@
            # Let's write to a log
            Log-Msg ($message)
            # Then email message
            Send-Mail($message)
        }
    }
}

Makes sense? You can add more features to it, but the above works fine to me including my multiline string. Noticed in my example the smtp server does not require me to authenticate since I am in a "blessed" network. If that was not the case, fear not since send-mailmessage has a way to use authentication. I will leave that as an exercise to you.

Setting NTP server and time in Windows using Powershell

And here we have yet another Windows-related post! Yes, I too make fun of Windows as much as required to be in the IT business... ok sometimes more. But, as I have said again and again, being able to solve problems using command line (powershell specifically) makes it feel more like Unix. I can handle that and so can you!

Most of the Windows boxes I met that use a time server to set their time use the Microsoft one, time.windows.com, no matter if they are the sole computer in a car shop or one of the thousands desktop and servers in an university. That is nice until you have to move away from local-only user accounts and deal with Kerberos and, by extension, Active Directory. You see, Kerberos likes to have its clients to be within 5 minutes of the authentication servers (KDCs). Syncing against the Microsoft time server assumes your machine is in a network that can access the Internet. Well, I have 8 of them which are in a vlan that can't (and really shouldn't). Updates to them are pushed through SCCM (when it feels like working, but I digress) and AD.

On the top of that, I have a perfectly good ntp server in my network this vlan can reach anyway. And its address is passed by dhcp. To add insult to injury, Microsoft does not support the dhcp option to care about ntp servers. Here is a list of the DHCP options supported right from their official docs.

So, as always, I need to do something to make it stop pissing me off. And, it will be in a script of some sort. This is Windows so bash is out and Powershell is in.

The plan is to be able to find which ntp server the Windows host is using and change it if we do not like it. And, while we are there, make sure the host's time is in sync with that of the ntp server. Windows uses W32Time and stores all of that in the registry, namely in HKLM:\SYSTEM\CurrentControlSet\services\W32Time, so if you want you can unleash regedit and go at it. Taking a cue from Unix and Linux, powershell treats the registry as a file tree. So, as far as it is concerned, the above is just a path which can be accessed and modified using Get-ItemProperty and Set-ItemProperty. Let's try it out by taking a look on what we have currently defined:

PS C:\> $timeRoot = "HKLM:\SYSTEM\CurrentControlSet\services\W32Time"
PS C:\> Get-ItemProperty  -path "$timeroot\parameters"


PSPath                 : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE
                         \SYSTEM\CurrentControlSet\services\W32Time\parameters
PSParentPath           : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE
                         \SYSTEM\CurrentControlSet\services\W32Time
PSChildName            : parameters
PSDrive                : HKLM
PSProvider             : Microsoft.PowerShell.Core\Registry
ServiceDll             : C:\Windows\system32\w32time.dll
ServiceMain            : SvchostEntry_W32Time
ServiceDllUnloadOnStop : 1
Type                   : NT5DS
NtpServer              : time.windows.com,0x9



PS C:\>

The 3 blank lines below NtpServer are not a typo; don't ask me why it spits those lines because they add absolutely no value to the output besides wasting screen real state. As you can see, it wants to use time.windows.com as the NtpServer. But, what is this 0x9 on the end of the name of the ntp server? Well, here is what I know about what 0x flags mean

• 0x01 SpecialInterval: interval in seconds between when W32Time pools for time. Requires HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient\SpecialPollInterval to be setup. By default W32Time checks the time at intervals based on the network speed, traffic, and phases of the moon. But, if you turn SpecialInterval on, it will check evet SpecialPoolInterval seconds. So, SpecialPoolInterval = 3600 means it will check time ever 3600s (or 1h).
  
• 0x02 UseAsFallbackOnly
  
• 0x04 SymmatricActive
  
• 0x08 Client
  
• 0x09 = 0x01 + 0x08. Yes, we can do math.
  

If we want to change it to, say, ntp.example.com, in powershell we would begin by

PS C:\> Set-ItemProperty  -path "$timeroot\parameters" -name NtpServer -Value "n
tp.example.com,0x9"
PS C:\>

And then checking again

PS C:\> Get-ItemProperty  -path "$timeroot\parameters"


PSPath                 : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE
                         \SYSTEM\CurrentControlSet\services\W32Time\parameters
PSParentPath           : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE
                         \SYSTEM\CurrentControlSet\services\W32Time
PSChildName            : parameters
PSDrive                : HKLM
PSProvider             : Microsoft.PowerShell.Core\Registry
ServiceDll             : C:\Windows\system32\w32time.dll
ServiceMain            : SvchostEntry_W32Time
ServiceDllUnloadOnStop : 1
Type                   : NT5DS
NtpServer              : ntp.example.com,0x9



PS C:\>

We changed the config, but we then need to restart the time server for that to take effect

Restart-Service -Name w32Time -Force

Let's see if we can put some of that together in a script, which I shall call ntpTime.ps1:

# SEE ALSO
# https://technet.microsoft.com/en-us/library/ee176960.aspx

$timeRoot = "HKLM:\SYSTEM\CurrentControlSet\services\W32Time"

# Name of ntp server(s) currently known by this host
function Get-NTPServer {
   $ntpserver = (Get-ItemProperty  -path "$timeroot\parameters" `
                 -name NtpServer).NtpServer -replace ",.*"
   return $ntpserver
}

# So we do not like the ntp servers this host knows and want to change them.
# Remember the 0x flags!
#
# 0x01 SpecialInterval    
# 0x02 UseAsFallbackOnly  
# 0x04 SymmatricActive
# 0x08 Client
#
function Set-NTPServer ($ntpServer) {
   Set-ItemProperty -path "$timeroot\parameters" -name NtpServer -Value $ntpServer
}

function Restart-Time {
   Restart-Service -Name w32Time -Force
}

# How far off are our time (in seconds) from the one in our ntp server?
function Get-NTPOffset ($ntpServer) {
   (w32tm /stripchart /computer:$ntpServer /samples:1)[-1].split("[")[0] `
   -replace ".*:" -replace "s.*"
}

# Adjust time by using the offset
function SetTime ($offsetSeconds) {
   set-date (Get-Date).AddSeconds($offsetSeconds)
}

## Using those silly functions ----------------------------------
$myNTP = "ntp.example.com"
$leserver = Get-NTPServer
if ( $leserver -eq $myNTP ){
   Set-NTPServer("$($myNTP),0x9")
}
SetTime(Get-NTPOffset($myNTP))
Restart-Time

I will put a more complete version in my github account, but the above is good enough to be productive. So, what it does is first see whether we are using the right ntp server ($myNTP since I needed a lame variable name). If not, it changes it. And then it adjust time as needed. Script can then be run (schtasks anyone?) at regular intervals or when the machine wakes up if it is a vm or laptop.

environment variables, date, and string concatenation in powershell

This is another of those quick posts. Sometimes in Linux/OSX I want (or even need) to rename or copy a file filename to path/filename_date. For instance, let's say the file is called cli64.log. I then can do something like

bash-3.2$ cp cli64.log cli64_`date +%Y%M%d-%H%M.log`
bash-3.2$ ls -lh cli64.log*
-rw-r--r--  1 dalek  staff   2.8K Feb 21  2013 cli64.log
-rw-r--r--  1 dalek  staff   2.8K Mar 23 16:29 cli64_20152923-1629.log
bash-3.2$

to append the date (as YearMonthDay which in this case turns out to be 20150323) and the time (HourMinute which when I did the above was 1629 or 4:29PM for those who cannot count past 12) to the name. So far so good.

As some of you have guessed -- maybe the title of this article was a dead giveaway -- I sometimes need to deal with Windows. And I do my best to make it behave as close to Linux (using Linux as placeholder for Linux/OSX/whatever since they behave the same in this case. In fact, the machine I ran the above command is a Mac Mini running OSX) as I can, which is why I use Powershell. So, how do I do the same copy command in Powershell?

Date

To get the date, the command we need is Get-Date. When you run it by itself, it gives something like

PS C:\Users\raub> get-date

Monday, March 23, 2015 4:41:13 PM


PS C:\Users\raub>

which is not useful for us; we want to make the date be part of the filename in the format we want. We will work this two part problem starting at the format and then worrying about the concatenation part.

According to the docs, and to http://ss64.com/ps/get-date.html (which has a convenient list of the time formats we can use), we can use the -format option. Let's try and see if we can replicate the output of date +%Y%M%d-%H%M:

PS C:\Users\raub> get-date -format 'yyyyMMdd-HHmm'
20150323-1643
PS C:\Users\raub>

That looks very similar to what we did in Linux. How abut adding that to the filename?

Concatenating

So, concatenating strings in Powershell is a bit interesting. Let's say we have $theTestFile=C:\Users\raub\monkey\testfile.txt and we want to append tmp to it. Now we can try a few things and see what we can come up with:

PS C:\Users\raub> echo "$theTestFile" + tmp
C:\Users\raub\monkey\testfile.txt
+
tmp
PS C:\Users\raub> echo "$theTestFile" += tmp
C:\Users\raub\monkey\testfile.txt
+=
tmp
PS C:\Users\raub> echo "$theTestFile += tmp"
C:\Users\raub\monkey\testfile.txt += tmp
PS C:\Users\raub> echo "$theTestFiletmp"

PS C:\Users\raub> echo "$theTestFile tmp"
C:\Users\raub\monkey\testfile.txt tmp
PS C:\Users\raub> Write-host "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub> echo "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub>

So, the parenthesis thingie seems to be what we want to do.

Putting it all together

Now we know how to get the date and concatenate, if we put the date thingie inside the parenthesis thingie, the equivalent of

cp cli64.log cli64_`date +%Y%M%d-%H%M.log`

in powershell is

copy cli64.log "cli64_$(Get-Date -format 'yyyyMMdd-HHmm').log"

Er, we are not done yet

But, you will point out, the title of this article mentions environment variables! Fair enough. So we will expand the original problem. Say, you also want to name the copy of the log file to not only included when it was copied but also the hostname. Reason here is that you or someone else who will get this file might need to know where this file came from. In Linux, that can be done with $HOSTNAME, as in

cp cli64.log cli64_$HOSTNAME-`date +%Y%M%d-%H%M.log`

but what about Windows and powershell? Enter the environment variables we talked about. We would hope the computer knows what it is called, amongst other things, right? Let's see what it knows

PS C:\Users\raub> ls env:

Name                           Value
----                           -----
ALLUSERSPROFILE                C:\ProgramData
APPDATA                        C:\Users\raub\AppData\Roaming
CommonProgramFiles             C:\Program Files\Common Files
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files
CommonProgramW6432             C:\Program Files\Common Files
COMPUTERNAME                   VBOX01
ComSpec                        C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK               NO
HOMEDRIVE                      C:
HOMEPATH                       \Users\raub
LOCALAPPDATA                   C:\Users\raub\AppData\Local
LOGONSERVER                    \\ZOOL
NUMBER_OF_PROCESSORS           1
OS                             Windows_NT
Path                           %SystemRoot%\system32\WindowsPowerShell\v1.0\;C:\ProgramData\Oracle\Java\javapath;C:\...
PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE         AMD64
PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
PROCESSOR_LEVEL                6
PROCESSOR_REVISION             3c03
ProgramData                    C:\ProgramData
ProgramFiles                   C:\Program Files
ProgramFiles(x86)              C:\Program Files (x86)
ProgramW6432                   C:\Program Files
PSModulePath                   C:\Users\raub\Documents\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowe...
PUBLIC                         C:\Users\Public
SESSIONNAME                    Console
SystemDrive                    C:
SystemRoot                     C:\Windows
TEMP                           C:\Users\raub\AppData\Local\Temp
TMP                            C:\Users\raub\AppData\Local\Temp
UATDATA                        C:\Windows\CCM\UATData\D9FFC898-CBB8-491d-D8CA-173A9FF1B077
USERDNSDOMAIN                  EXAMPLE.COM
USERDOMAIN                     EXAMPLE
USERNAME                       raub
USERPROFILE                    C:\Users\raub
windir                         C:\Windows


PS C:\Users\raub>

That looks very impressive. How do we get that info in a way we can use when renaming out file? Here is how you get the hostname:

PS C:\Users\raub>$env:computername
VBOX01
PS C:\Users\raub>

So, if we wanted to copy poor cli4.log to some directory in some drive, adding to the new filename the hostname and date the file was copied, we could do a lot worse than

copy cli64.log "J:\kitchen\fridge\cli64_$env:computername-$(Get-Date -format 'yyyyMMdd-HHmm').log"

So I think we might have made managing Windows be a bit saner than before.