Showing posts with label osx. Show all posts
Showing posts with label osx. Show all posts

Monday, September 24, 2018

Using ldapsearch and ldapmodify to talk to Active Directory

Why?

Great question! Here are a few lame excuses I was able to come with:

  • I like to use command line. This is a lame excuse because Windows have powershell. But,
  • I am more comfortable with Linux than Windows. Lame excuse since
    1. How many posts in this very blog I have made about using Windows?
    2. How many of said posts I have used the GUI when I could take care of business with Powershell?
    3. What is wrong with Powershell, at least of the applications I have used it for here so far?
I do have a couple of not so lame ones though:
  • I like to be able to access the network resources from any machine in the network running any OS. If I have a Linux box in an Active Directory-controlled network, chances are I will need to authenticate the Linux box against Active Directory (AD so I can save some keytaps). AD is Kerberos + ldap + sprinkles, so I better be able to use the usual kerberos/ldap Linux tools as one day I will need to figure out why things are boink.
  • It feels like I get more info using ldapsearch than the Windows tools, which is good when I do not know the name of an attribute, or how many instaces of said attribute are in use.

Using ldapsearch

Before we go mindlessly typing things, we need some data.
We need to know the name of the ldapserver.
Yes, if you have it configured in your ldap.conf file, you should not need it. But I prefer to assume nothing. If the domain was setup properly, we can ask it directly by typing nslookup -type=srv _ldap._tcp.DOMAIN where DOMAIN is the Active Directory domain name, not the DNS one; that caught me off guard. So, if our DNS dmain is example.com and the AD domain (we are very original) is ad.example.com, we have
raub@desktop:/tmp$ nslookup -type=srv _ldap._tcp.ad.example.com
Server:         192.168.0.10
Address:        192.168.0.10#53

Non-authoritative answer:
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC0.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC2.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC1.ad.example.com.

Authoritative answers can be found from:
ad.example.com      nameserver = addc1.ad.example.com.
ad.example.com      nameserver = ns.example.com.
ad.example.com      nameserver = ns2.example.com.
ad.example.com      nameserver = addc0.ad.example.com.
ad.example.com      nameserver = addc2.ad.example.com.
ADDC0.ad.example.com        internet address = 192.168.1.100
ADDC1.ad.example.com        internet address = 192.168.1.102
ADDC2.ad.example.com        internet address = 192.168.1.101
ns.example.com      internet address = 192.168.0.10
ns2.example.com     internet address = 192.168.0.10

raub@desktop:/tmp$
and we can use any of the ADDCN.ad.example.com (where N=0,1,2). Notice in my setup, just using ad.example.com also worked. I found out by using netcat to see if port 636 was open (I will leave the answer for "why port 636?" as an exercise to the reader)
raub@desktop:~$ nc -v ad.example.com 636
Connection to ad.example.com 636 port [tcp/ldaps] succeeded!
^C
raub@desktop:~$ 
We need to be able to authenticate against AD somehow.
For this discussion I will be using a username and password; we can also do it using a Kerberos TGT ticket.

Fun Fact: I have a user account, my normal one, which can look into some things in LDAP/AD but then I have another ("admin") account I can see more and edit stuff in AD. You will see later on me forgetting completely about that and how it affects me. But we are getting ahead of ourselves.

With that taken care of, we are going to begin by looking for some user: me

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" 
Enter LDAP Password:
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
objectClass: top
objectClass: posixAccount
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Wrong Droid
sn: Droid
title: Entropy Creators
description: Orthodontics
givenName: Wrong
initials: B 
distinguishedName: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
instanceType: 4
whenCreated: 20080109142820.0Z
whenChanged: 20180905201157.0Z
displayName: Droid, Wrong
uSNCreated: 243336
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
[...]
proxyAddresses: sip:raub@ad.example.com
proxyAddresses: smtp:raub@ad.example.com
proxyAddresses: X500:/o=UNC Exchange/ou=Exchange Administrative Group (FYDIBOH
 F23SPDLT)/cn=Recipients/cn=raub
proxyAddresses: SMTP:raub@email.example.com
displayNamePrintable: Wrong Droid
name: Wrong Droid
[...]
sExchPoliciesExcluded: {26491cfc-9e50-4857-861b-0cb8df22b5d7}
msExchUserAccountControl: 0
msExchELCMailboxFlags: 2
msRTCSIP-PrimaryHomeServer:
[...]
msExchOWAPolicy: CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=ad,DC=example,DC=com

# refldaps://ForestDnsZones.ad.example.com/DC=ForestDnsZones,DC=ad,DC=example,DC=com

# refldaps://DomainDnsZones.ad.example.com/DC=DomainDnsZones,DC=ad,DC=example,DC=com

# refldaps://ad.example.com/CN=Configuration,DC=ad,DC=example,DC=com

raub@desktop:~$

That is probably more info than you wanted to know about someone, but this has its applications. Since we now know every attribute associated with a given user (I should not be that special, at least as far as AD is concerned), we can build customized queries looking for only a specific bit of info. For instance, let's just get the groups I belong to or am a member of (hint hint):

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" memberOf 
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com

Fancy, huh? Now, if instead of doing (CN=raub) we did (CN=*raub*), it would return every entry that has a CN with raub in it. In my case that would mean two entries, raub and raub.admin (if you remember our Fun Fact you will know about it), but it could also have returned a device whose name matches that. And, they would have been printed one after the other (I do wonder if the order depends on the order they were added to LDAP/AD).

Using ldapmodify

Ok, we established we can probulate Active Directory using common household Linux/UNIX query tools. What if we want to change something? Let's say we have an AD group (specifically a distribution list) called moustache_operators (for those who own and operate moustaches) and want to add a member and delete another.

Why I do like ldapmodify to edit LDAP/AD

Main reason is because I can create a file (in the LDIF format) at my leisure (i.e. think about what I want to do) with pretty commands and comments describing what I want to do. If I like what I did, I can then document it and maybe even save the file in a wiki or somewhere that can be fed to Ansible/Puppet/Chef/Docker and reused.

Our little LDIF file, let's call it change.ldif, could look like this:

# Let's define the entity we will be fiddling with
dn: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com
# And then what we will be doing with it
changetype: modify
delete: member
member: CN=baldone,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
# Separator because we will be doing another change
-
add: member
member: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com

So let's try it:

raub@desktop:~$ ldapmodify -H "ldaps://addc0.ad.example.com:636" -D "raub@ad.example.com" -x -W -f change.ldif
Enter LDAP Password:
modifying entry "CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com"
ldap_modify: Insufficient access (50)
        additional info: 00002098: SecErr: DSID-03150F93, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

raub@desktop:~$ 

Why is it not working? Well, do you remember the Fun Fact I mentioned earlier in this article? This is how it shows it's ugly head. I should have used my raub.admin@ad.example.com account instead of raub@ad.example.com. If we do it right, it then works. I will not show the output of a successful connection here; what matters is verifying the deed is done, and we can do it using dear ol' ldapsearch:

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub) memberOf" dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com

What about Mac/OSX?

They too have ldapsearch; just use the terminal and off you go.

Wednesday, August 29, 2018

Customizing the prompt in OSX (or anything running bash)

Quick and dirty article whose only claim to fame is to show another example of treating a Mac running OSX a UNIX box: I have two Mac, an old Mini and a MacBook Air (I think I talked about it before). If you have been reading this blog, you expect me to use the shell a lot in them, and you would be right. One thing that annoys me is the default prompt used in both. The old, slow mini has the most useless one:

bash-3.2$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
Copyright (C) 2007 Free Software Foundation, Inc.
bash-3.2$

Why would I care about knowing at all times which bash version I am running, and even then it is incomplete info (compared to bash --version)? The MacBook Air is just slightly better, showing (in order) the name of the machine, the path, and then the username.

littleguy:~ raub$

It is similar in content to what I would see in my Linux boxes,

raub@desktop:~$
but I do not like the order; it does not flow right in my eyes. So I am going to change both to look more like Linux.

First let's see how those prompts are defined in both machines; that is found by looking at the content of the environment variablePS1:

bash-3.2$ echo $PS1
\s-\v\$
bash-3.2$
and
littleguy:~ raub$ echo $PS1
\h:\W \u\$
littleguy:~ raub$

From the official gnu page on controlling the Bash prompt, we know that

  • \h : hostname
  • \s : shell name
  • \u : username
  • \v : shell version
  • \w : current working directory
  • \$ : the generic symbol for users (as opposite to # for root)
Based on the above, what I really want is something like PS1="\u@\h:\w\$ ". So let's try it out without permanently committing to it

bash-3.2$ PS1="\u@\h:\w\$ "
raub@slowmac:~$ 

That seems to be exactly what I want. And, since it only exists in memory (i.e. did not commit it), if I did not like it, restarting the shell or opening a new tab in iTerm would brought it back to the original config. So, how do we make it permanent? In the slowmac, we have

raub@slowmac:~$ cat ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/opt/local/bin:/opt/local/sbin

export PATH

raub@slowmac:~$

So we might as well put it in the .bashrc file the lazy way.

cat >> ~/.bashrc << "EOF"
PS1="\u@\h:\w\$ "
EOF

Interestingly enough, the MacBook Air does not have a ~/.bash_profile. Maybe the slowmac did not have it either and I added it because I was developing programs in it a while ago. No problem, we can solve the MacBook Air problem by creating a ~/.bash_profile or just appending the proper lines to it, which is done the same was as we did to add the prompt lines to the slowmac's .bashrc:

cat >> ~/.bash_profile << "EOF"

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
EOF

Why do we put the prompt in the .bashrc instead of .bash_profile. You see, while ~/.bash_profile is executed when you login, ~/.bashrc is run whenever you start a new shell. By having our prompt defined in ~/.bashrc and then having ~/.bashrc called by ~/.bash_profile we cover our bases.

I am happy, but if you want to make your prompt fancier with colour and cows (really), I would suggest you to check a few more articles such as

In addition to the gnu page I mentioned above. Incidentally, everything I mentioned above should work with any Linux and UNIX distro using bash

Wednesday, August 22, 2018

Scheduling Time Machine backups from the command line

So I have an old Mac Mini which will just not start a backup to my time machine server. We could go over the reason but that is the subject for another article. The point is running it from the GUI does not work. However, I can go to the terminal window and do

raub@slowmac:~$ tmutil startbackup

all day and it works fine. In fact, I have been doing that manually to have a semblance of a backup:

dalek@strangepork:~$ tmutil latestbackup
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-07-16-234250
dalek@strangepork:~$ tmutil listbackups
[...]
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-05-01-130725
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-05-24-032218
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-06-21-001720
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-07-08-201058
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-07-16-234250
/Volumes/Time Machine Backups/Backups.backupdb/strangepork/2018-08-22-131023
dalek@strangepork:~$ 

But that is a bit of a drag. As you can see I am not manually doing it as often as I should. We need to automate this. Hmmm... recurring job... if this was Linux, Solaris, FreeBSD, or AIX, I would use a cronjob. But this is a Mac... running OSX. What should I do? Look for some App?

Wait a minute. OSX is UNIX with some sprinkles on the top. So, let's cron this out!

I am going to start the task off my normal account since I do not have to run startbackup from root. First let's see cronjobs I have right now:

dalek@strangepork:~$ crontab -l
#
dalek@strangepork:~$

Nothing at all, which is as good place to start as any. Now we need to add the entry. I like to specify the path of the program I am using in case there are old versions. So from

I will be using /usr/bin/tmutil. Now let's edit the crontab, which is done by typing crontab -e. That put me in a vim session (that can be configured); if you do not know it, it is a good time to learn. Here are a few quick pointers:

  • When in doubt, press the escape key a lot.
  • After you press esc a lot, if you want to save your changes and quit, type :wq, but if you do not want to save, type :q!
I do not know how often timemachine usually runs, so I will guess every two hours and will tell my cronjob to start at 15 minutes past the hour every 2 hours. In cron-lese, that looks like this

15  */2  * * *  /usr/bin/tmutil startbackup

After we save it, let's verify that our little work is committed:

dalek@strangepork:~$ crontab -l
#
15  */2  * * *  /usr/bin/tmutil startbackup
dalek@strangepork:~$

Then, give it a few hours and run tmutil listbackups to see if the backup cron jobs are being run every two hours.

Saturday, April 28, 2018

Using sed Capture Groups (Linux/Mac)

This will be a short one and belongs to the TIL bin: until 2 days ago I did not even know about capture groups and how to use it. So, I knew how to replace a matching string/pattern in sed

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v/swallow_speed/' 
As of today swallow_speed=23kph at STD
bash-3.2$ 
And how to replace from a given pattern all the way to the end of the line. Or starting from the beginning of the line to said pattern:

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*/swallow_speed=42/' 
As of today swallow_speed=42
bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/^.*swallow_v=/Can you believe that swallow_speed=/' 
Can you believe that swallow_speed=23kph at STD
bash-3.2$ 

If you are curious, the .* in the search pattern means "any character or list of characters here, be it zero or a lot of characters". The dot (.) does the any part and the asterisk (*) the how many. I think this is from regex, but don't quote me on that. Fine, what if I want to replace everything between two patterns, but leaving the second pattern alone? Tricky. You see, replacing everything between the two patterns, inclusive is not that hard

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*k/swallow_v=25/'
As of today swallow_v=25ph at STD
raub@desktop:~$ 

But to preserve the second pattern we need to use the Capture Groups mentioned in the title of this article. And that makes sense because if it is on the title I better use it. So, we are supposed to surround the capture group pattern with parenthesis and then we can refer to them. If it does not make sense, I too was confused, so let's keep on using our test string:

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*(k)/swallow_v=25$1/'
As of today swallow_v=23kph at STD
raub@desktop:~$ 

Er, it does not seem to have worked according to the plan. In fact, it was supposed to at least grab swallow_v=23k but as you can see it did not find the pattern. Is $1 the proper way to output the captured string? Going nowhere slowly.

After much soul searching, I found the -e requires the parenthesis to be escaped. And, the capture group pattern is output using \1 instead of $1.So we try again:

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e "s/swallow_v=.*\(k\)/swallow_v=25\1/"
As of today swallow_v=25kph at STD
raub@desktop:~$ 

much better!

What about the Mac? Same thing (the bash-3.2$ you have seen all day is it; the raub@desktop:~$ is the Linux box):

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*\(k\)/swallow_v=25\1/' 
As of today swallow_v=25kph at STD
bash-3.2$ 

I am not going to say it is perfect though:

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*\([[:blank:]]\)/swallow_v=25\1/' 
As of today swallow_v=25 STD
bash-3.2$ echo "As of today swallow_v=23kph at    STD" | sed -e 's/swallow_v=.*\([[:blank:]]\)/swallow_v=25\1/' 
As of today swallow_v=25 STD
bash-3.2$ 

References

  • A site I think has lots of interesting sed examples.

Tuesday, September 26, 2017

Forcing a fuse (sshfs) network fileshare to unmount in OSX

As some of you already know, I do have an old MacBook Air which I use as my main (as in the computer I sit in front of, not the computer I store data on. Laptops can be stolen, you know) machine until I find a new Linux laptop replacement. For this reason I need it to play nice with other machines, and that requires sometimes to mount a fileshare. If the other host is in the same VLAN, that is rather easy because there are ways to mount a Windows (SMB/CIFS) and even a Linux/UNIX (nfs) fileshare without breaking a sweat. But what if the machine is remote? If we can ssh into it, why not then use sshfs?

As we are aware of (since we read the link. There are a few more sshfs examples here), sshfs requires fuse. Since I am using OSX, which at the present time does not have it, I need to install. If you are curious, the one I use is FUSE for MacOS.

Mounting: business as usual

Let's say we are in the machine boris as user pickles trying to mount my home directory off desktop. We create the mountpoint (Let's use /tmp/D or ~/D so it looks more like what we would do in Linux:

boris:Documents pickles$ mkdir /tmp/D; sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ df -h
Filesystem                     Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk1                    112Gi   79Gi   33Gi    71% 20783599  8546848   71%   /
devfs                         364Ki  364Ki    0Bi   100%     1259        0  100%   /dev
map -hosts                      0Bi    0Bi    0Bi   100%        0        0  100%   /net
map auto_home                   0Bi    0Bi    0Bi   100%        0        0  100%   /home
raub@desktop.in.example.com:.  492Gi  389Gi  102Gi    80%   408428 32359572    1%   /private/tmp/D
boris:Documents pickles$

So far so good. To unmount it we can use diskutil, as in (Mac)

boris:Documents pickles$ diskutil umount /tmp/D
Unmount successful for /tmp/D
boris:Documents pickles$

or (Linux)

fusermount -u /tmp/D

Or go old school (both):

sudo mount /tmp/D

Since boris is a laptop, sometimes if we just let it go to sleep it will unmount it. Then, all we have to do is mount it again.

Mounting again: not so fast

Thing is, sometimes it does not work.

boris:Documents pickles$ mkdir /tmp/D; sshfs raub@desktop.in.example.com:. /tmp/D
mkdir: /tmp/D: File exists
fuse: bad mount point `/tmp/D': Input/output error
boris:Documents pickles$ 

Ok, maybe it did not automagically unmounted while laptop was off. So, let's tell it to do so:

boris:Documents pickles$ diskutil umount /tmp/D
Unmount failed for /tmp/D
boris:Documents pickles$ 

Just before you ask, sudo mount /tmp/D did not work either. What if the old sshfs processes did not cleanly closed and as a result are still lingering? To answer that we must elicit some help from one of grep's cousins, pgrep:

boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
7356 sshfs user@other.in.example.com:. /tmp/D
boris:Documents pickles$

Just as we guessed, there are not only but quite a few unhappy sshfs instances. Let's see if we can kill them:

boris:Documents pickles$ kill 384 1776 7356
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ kill 384
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ kill 1776
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
1776 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$
Hmmm, this is going nowhere slowly. Let's crank up a notch and force it to kill the mount.
boris:Documents pickles$ kill -9 1776
boris:Documents pickles$ pgrep -lf sshfs
384 sshfs raub@desktop.in.example.com:. /tmp/D
boris:Documents pickles$ kill -9 384
boris:Documents pickles$ pgrep -lf sshfs
\boris:Documents pickles$

Sounds like we got them all. Now, let's try and mount once more:

boris:Documents pickles$ mkdir /tmp/D; sshfs raub@desktop.in.example.com:. /tmp/D
mkdir: /tmp/D: File exists
raub@desktop.in.example.com's password:
boris:Documents pickles$

I think we have a winner!

Tuesday, August 15, 2017

Connecting to multiple VPNs using one single Cisco AnyConnect

Like many here, I remote into networks to work. I access organization X's network using Cisco's AnyConnect VPN client because that is what they use. When I first got involved, they told me to login to a given url in their webserver and get the client for my machine (a MacBook Air if you are curious; I do need to get a new Linux laptop but the Mac has been working great so far). Probably if my machine machine was a company-owned laptop they would have pushed the packaged using SCCM/Chocolatey (Windows) or Casper(now called jamf)/Munki (Mac). Or ansible, but that is another bag of cats. In any case, the point is I got their package, which was configured to work on their VPN. And, it works: double-click on the silly link, connect, enter my authentication info, and off I go.

Now also need to access organization B's machines. And they also chose to use AnyConnect. And just like X they also told me to install their package. Thing is if I do that it will wipe the X configuration, which would get annoying very quickly. I did try seeing if there was a way to add another profile from the client's menu but not luck. Maybe each company disabled the option so you can only use it to access their network; I do not know. Now what I could do since this is a Mac is rename Company X's VPN folder to, say, Cisco.Old (the default folder name is Cisco and then install Company B's VPN package.

This way, if I need to go to X, I would open Cisco.Old and then run that vpn client. If I then wanted to go to B, I would quit the client, go to Cisco, and then run that client. I do not know about you, but that looks a bit cumbersome to me. And, if my laptop was running Windows, I think it would not let me install 2 instances of the client that easily. There has to be a better way.

Probulating

First of all, let's assume there is a configuration file somewhere for the AnyConnect VPN client. Since I am using OSX, chances are it has some plist-sounding name. And I found something called com.cisco.Cisco-AnyConnect-Secure-Mobility-Client.plist in my preferences folder, /Users/raub/Library/Preferences, but it does not look particularly legible from the command line (yes, I know there is probably an app to do that but I like to do things from the command line):

bplist00Ñ^A^B]UILogLocation¥^C^D^E^F^G_^PA/Users/raub/.cisco/vpn/log/UIHistory_2017.08.28.23.35.34.010.txt_^PA/Users/dalek/.cisco/vpn/log/UIHistory_2017.08.28.23.53.04.734.txt_^PA/Users/raub/.cisco/vpn/log/UIHistory_2017.08.29.00.10.34.504.txt_^PA/Users/raub/.cisco/vpn/log/UIHistory_2017.08.29.00.28.05.785.txt_^PA/Users/raub/.cisco/vpn/log/UIHistory_2017.10.04.04.44.45.284.txt^@^H^@^K^@^Y^@^_^@c^@§^@ë^A/^@^@^@^@^@^@^B^A^@^@^@^@^@^@^@^H^@^@^@^@^@^@^@^@^@^@^@^@^@^@^As

So we make a copy of it and then run

plutil -convert xml1 com.cisco.Cisco-AnyConnect-Secure-Mobility-Client.plist
to convert it to something more legible, and then look inside it:

boris:~ raub$ cat com.cisco.Cisco-AnyConnect-Secure-Mobility-Client.plist




 UILogLocation
 
  /Users/raub/.cisco/vpn/log/UIHistory_2016.12.12.13.41.31.883.txt
  /Users/raub/.cisco/vpn/log/UIHistory_2016.12.12.13.58.40.264.txt
  /Users/raub/.cisco/vpn/log/UIHistory_2016.12.12.14.15.56.295.txt
  /Users/raub/.cisco/vpn/log/UIHistory_2017.02.14.06.03.40.692.txt
  /Users/raub/.cisco/vpn/log/UIHistory_2017.07.24.21.43.25.742.txt
 


boris:~ raub$

Hmmm, that does not look like what I want. Maybe the AnyConnect client has a global configuration file somewhere. And it does, and it is called glvpn-anyconnect-profile.xml and is located in /opt/cisco/anyconnect/profile/:

boris:~ raub$ ls /opt/cisco/anyconnect/profile/
AnyConnectProfile.xsd  glvpn-anyconnect-profile.xml
boris:~ raub$

If we look into it, this xml file starts as expected with some system-wide config settings

cat /opt/cisco/anyconnect/profile/glvpn-anyconnect-profile.xml
<?xml version="1.0" encoding="UTF-8"?>
<AnyConnectProfile xmlns="http://schemas.xmlsoap.org/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.xmlsoap.org/encoding/ AnyConnectProfile.xsd">
        <ClientInitialization>
                <UseStartBeforeLogon UserControllable="true">false</UseStartBeforeLogon>
                <AutomaticCertSelection UserControllable="true">true</AutomaticCertSelection>
                <ShowPreConnectMessage>false</ShowPreConnectMessage>
                <CertificateStore>All</CertificateStore>
                <CertificateStoreOverride>false</CertificateStoreOverride>
                <ProxySettings>Native</ProxySettings>
                <AllowLocalProxyConnections>true</AllowLocalProxyConnections>
                <AuthenticationTimeout>60</AuthenticationTimeout>
                <AutoConnectOnStart UserControllable="true">false</AutoConnectOnStart>
                <MinimizeOnConnect UserControllable="true">true</MinimizeOnConnect>
                <LocalLanAccess UserControllable="true">true</LocalLanAccess>
                <ClearSmartcardPin UserControllable="true">true</ClearSmartcardPin>
                <IPProtocolSupport>IPv4,IPv6</IPProtocolSupport>
                <AutoReconnect UserControllable="true">true

But then get to the part we have been anxiously waiting for: how to access company X's vpn:

<ServerList>
                <HostEntry>
                        <HostName>Company X VPN</HostName>
                        <HostAddress>vpn.companyx.com</HostAddress>
                </HostEntry>
        </ServerList>
</AnyConnectProfile>

It does not look very complicated to me: we probably could just add a new HostEntry for Company B, as in

<ServerList>
                <HostEntry>
                        <HostName>Company X VPN</HostName>
                        <HostAddress>vpn.companyx.com</HostAddress>
                </HostEntry>
                <HostEntry>
                        <HostName>Company B VPN</HostName>
                        <HostAddress>vpn.b-company.com</HostAddress>
                </HostEntry>
        </ServerList>
</AnyConnectProfile>

and be done. And that will work. But, I think we can do one better; can we avoid cluttering the profile file? Long story short is yes. Just put something like this

cat > B-profile.xml << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<AnyConnectProfile xmlns="http://schemas.xmlsoap.org/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.xmlsoap.org/encoding/AnyConnectProfile.xsd">
    <!--
        This section contains the list of hosts the user will be able to
        select from.
      -->
    <ServerList>
        <!--
            This is the data needed to attempt a connection to a specific
            host.
          -->
        <HostEntry>
            <!--
                Can be an alias used to refer to the host or an  FQDN or
                IP address.  If an FQDN or IP address is used, a
                HostAddress is not required.
              -->
            <HostName>Company B VPN</HostName>
            <HostAddress>vpn.b-company.com</HostAddress>
        </HostEntry>
    </ServerList>
</AnyConnectProfile>

in /opt/cisco/anyconnect/profile/:

boris:~ raub$ ls /opt/cisco/anyconnect/profile/
AnyConnectProfile.xsd  glvpn-anyconnect-profile.xml
B-profile.xml
boris:~ raub$

Now when we run the client, we can select either company's VPN:

What about Windows

I've never tried but there is a file called (starting at your homedir) .\AppData\Local\Cisco\Cisco AnyConnect Secure Mobility Client\preferences.xml which would be my starting point. The global profile folder is c:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Profile.

Final thoughts

I do not like that I have to configure the different profiles at the global level; I might share this laptop with other people and would like to have my profiles uncluttered away from theirs. But, at least now I can use multiple profiles to access different networks. Looking at the Windows configuration file, I wonder if I can do that int he Mac too. That will be the subject for another article.

Wednesday, January 11, 2017

Creating extended ASCII file in bash and maybe powershell

Does anyone remember extended ASCII (as opposite to UTF-8)? If you never heard of them, we are not talking about a proper character list that supports Russian or Japanese languages. All we are dealing with here is iso-8859-1, whose table can be found here.

I have a document in that format I need to convert to something else; if this reminds you of some 8bitmime issues we talked about, well, let's just say we could have used this to create the test file. With that said, the current situation is that I wrote a script to manipulate it which is not preserving that format; that we can talk about in a future post.

Bottom line is I need to create a small test file that I can later throw it and the script output on hexdump.

My test file will have only 3 lines,

Olivenöl
Bayerstraße 22
München

Nothing fancy; just enough to use one extended ASCII character per line. Now let's try to create the little file. Just to be different, instead of starting on Linux we will do most of the attempts in OSX. Once we have a working system, we can see if it also works on Linux.

Attempt #1

How about if we do the lazy thing and jus cut-n-paste the 3 lines above into a text file we opened usng vim, notepad++, or some pico clone? Done. Now let's see how it looks like

bash-3.2$ cat /tmp/chartest 
Olivenöl
Bayerstraße 22
München
bash-3.2$ 

That looks very promising. In fact, this might end up being a very short article. Before I publish it, should we see what hexdump thinks of it?

bash-3.2$ hexdump -Cv /tmp/chartest 
00000000  4f 6c 69 76 65 6e c3 b6  6c 0d 0a 42 61 79 65 72  |Oliven..l..Bayer|
00000010  73 74 72 61 c3 9f 65 20  32 32 0d 0a 4d c3 bc 6e  |stra..e 22..M..n|
00000020  63 68 65 6e 0d 0a                                 |chen..|
00000026
bash-3.2$ 

Correct me if I am wrong but it seems each extended ASCII character is taking two characters to be represented instead of just one single character. For instance ö is being represented by two characters, 0xC3B6. That sounds more like UTF-8/Unicode/whatever (if you want to know what to look for, they all start with a 0xC3) but not extended ASCII. Also, it is using carriage return (CR, 0x0D in hexadecimal) and line feed (LF, 0x0A) characters to separate the lines. But this is very Windowsy, not OSX/Linux style, where lines are separated by the line feed (0x0A) character only.

Attempt #2

What if we paste the lines onto the terminal and use echo to write that to the test file? Well, let's make a single line test file and see what happens

bash-3.2$ echo "Olivenöl" > /tmp/chartest 
bash-3.2$ hexdump -Cv /tmp/chartest 
00000000  4f 6c 69 76 65 6e c3 b6  6c 0a                    |Oliven..l.|
0000000a
bash-3.2$ 

Still using two characters to represent ö; at least it did not add a CR. Now this is getting annoying; is there anything else we can try?

Attempt #3

It turns out there is and we do not need any extra stuff. You see, echo has this -e option that allows you to pass a character by its hexadecimal code. From the extended ASCII table we know that ö= 0xF6 and ß= 0xDF (just to pick two examples). We also know that CRFL = 0x0D0A. I know I whined about that before, but the reason is I want to be able to decide when I want to use those characters and when I do not want, as opposite to having some program or script making a choice for me.

Let's try again, this time passing the extended ASCII characters explicitly:

bash-3.2$ echo -e "Oliven\xf6l\x0d\x0aBayerstra\xdfe 22\x0d\x0aM\xfcnchen\x0d" > /tmp/chartest
bash-3.2$ 
bash-3.2$ cat /tmp/chartest 
Oliven�l
Bayerstra�e 22
M�nchen
bash-3.2$ hexdump -Cv /tmp/chartest 
00000000  4f 6c 69 76 65 6e f6 6c  0d 0a 42 61 79 65 72 73  |Oliven.l..Bayers|
00000010  74 72 61 df 65 20 32 32  0d 0a 4d fc 6e 63 68 65  |tra.e 22..M.nche|
00000020  6e 0d 0a                                          |n..|
00000023
bash-3.2$ 

That's more like it: only one character is used to represent each character in the file. Isn't it interesting when we cat the file it is replacing the extended characters with ? But, if hexdump says they are there that is good enough for me.

What about powershell?

Even though the name of this blog implied Unix, we use enough powershell we might as well see if we can do the same. But, we have to accept we start with a bit of a handicap: powershell really really wants to write UTF-8 or unicode instead of extended ascii/iso-8859-1. Let me show you what I mean by trying to create a small file with just one single word on it, Olivenöl. As we seen before, ö = 0xF6 = 246. And that should still be true in powershell; let's find out:

PS > 'Oliven' + [char]246 + 'l'
Olivenöl
PS >

Looks like we are getting somewhere, right? For our next trick, we will save that to a file (| out-file .\chartest.txt is equivalent to doing > .\chartest.txt.

PS > 'Oliven' + [char]246 + 'l' | out-file .\chartest.txt
PS > cat .\chartest.txt
Olivenöl
PS >

Hey chief! It seems to be working fine? Why are you make this huge drama about this? That is a very good question. I will let dear old hexdump do the talking:

$ hexdump -Cv chartest.txt
00000000  ff fe 4f 00 6c 00 69 00  76 00 65 00 6e 00 f6 00  |..O.l.i.v.e.n...|
00000010  6c 00 0d 00 0a 00                                 |l.....|
00000016

$

Each character is now represented by 2 characters. Smells like Unicode, right? ok, smart guy. Now just force it to save as ASCII then. Will do:

PS > 'Oliven' + [char]246 + 'l' | out-file -encoding ASCII .\chartest.txt
PS > cat .\chartest.txt
Oliven?l
PS >

And hexdump

$ hexdump -Cv chartest.txt
00000000  4f 6c 69 76 65 6e 3f 6c  0d 0a                    |Oliven?l..|
0000000a

$

It converted the characters into ?. Helpful, isn't it? The Microsoft Scripting Guys forum pretty much tells you should save file as unicode or UTF-8 and then convert it somehow. Far from me to disagree with them, at least in this article since it makes for a great cliffhanger. In a future article we will talk about how to get extended ASCII properly in powershell just like we did in bash. It will be a bit longer but doable.

Monday, March 21, 2016

Resizing a VirtualBox .vdi file from command line

Another quick one: Let's say I am running VirtualBox (or vbox for short) and have a Microsoft Windows virtual client. Let's date ourselves and make it a Windows 7 vm. And I just found out its disk, win7.vdi is a bit small (40GB)and I would like to up it a bit, say, to 60GB. And me being the klutz I am, or have this unexplained desire to do everything on command line so I can script it later, I do not know how to do that using the GUI. So, like many times before, cue in for some command line action!

The command we will be using today is vboxmanage modifymedium or, as it used to be called, vboxmanage modifyhd. Let's see which options we have avaiable:

Oracle VM VirtualBox Command Line Management Interface Version 5.0.16
(C) 2005-2016 Oracle Corporation
All rights reserved.

Usage:

VBoxManage modifymedium     [disk|dvd|floppy] 
                            [--type normal|writethrough|immutable|shareable|
                                    readonly|multiattach]
                            [--autoreset on|off]
                            [--property ]
                            [--compact]
                            [--resize |--resizebyte ]

Why we would modify a dvd? Well this is really not a real dvd, but one of those .vdi files, which you can build to pretend to be a DVD. Other formats are also supported but .vdi is the native one. Now, I usually try to run my examples in a Linux host unless they specifically only run on a different OS (try running Powershell in Linux). Since VirtualBox can run in OSX, Windows, and Linux, I chose Windows this time. So, let's try to add 20GB to it:

PS C:\Program Files\Oracle\VirtualBox> .\vboxmanage modifyhd \
'C:\users\raub\VirtualBox VMs\win7\win7.vdi' --resize +20480
0%...
Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage.exe: error: Resize medium operation for this format is not implemented yet!
PS C:\Program Files\Oracle\VirtualBox>

You probably guessed I do not have the virtualbox binary path added to my default path, so I cheated and went to that directory. Of course you could have provided the full path to the binary, which would not be as lazy. In any case, it did not like my pretending this is a lv and I was expanding it of 20GB (i.e. the "+" did not work out). So, let's just tell it to do 60GB (I am cheating with the disk size) total and be done.

PS C:\Program Files\Oracle\VirtualBox> .\vboxmanage modifyhd \
'C:\users\raub\VirtualBox VMs\win7\win7.vdi' --resize 60000
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
PS C:\Program Files\Oracle\VirtualBox>

Of course now we have to resize the drive within the vm client itself, and that depends on the OS that is being run by that client.

notes:

  • I merged all snapshots before doing this. I did not test whether this would work with "snapshoted" disks.
  • I have read before about people having issues with vmdk files. I do not know if that still happens or not, but I thought it was worth mentioning. If you have errors, just do as the first reply and then convert it to vdi do your resizing, and then convert it back to vdmk.
  • I like to give the full path to the .vdi fle.

Sunday, April 19, 2015

using curl and awk to get the latest android sdk

Here is another of those short posts for me to remember how I did something. So I want to retrieve the latest version of the Android SDK. The reason is not relevant to this article, but since I need something to fill it up and make it sound more important than it is, I will tell you: I want to build a docker container to build Android packages and want to make sure the Dockerfile will grab the latest SDK.

As some of you know, the SDK is available at https://developer.android.com/sdk/index.htm. Since I want to automate this, I would love to be able to have the url for the latest release without having to know which one is that. Asterisk, for instance, allows you to grab the -latest one which would as the name says get the latest asterisk file without any thinking required. If you look at the Android SDK page, you will know what is the latest version number and can get it by clicking on it, but doing it command line seems to be not as fun as getting the -latest one.

One of my objectives is to get the file using curl instead of wget. don't get me wrong, I like and use wget a lot. And, try to forget about the usual wget vs curl wars; the reason is much more pedestrian: OSX comes with curl. Since I wanted to run this not only on Linux but also on OSX without having to use brew or macports, curl it is.

Let's see if we can make a quick script to do the deed. I was going to see if I could come up with a clever script, but someone cut me to it. I really like how they used gsub, so I just had to see if I could apply that idea. Here is my first attempt

#!/bin/sh
# getdroid.sh

site="https://developer.android.com/sdk/index.html"
os="linux"

filename=`curl -s --insecure ${site} | \
awk '/id=\"'${os}'-tools\"/{
    gsub(/.*href=\"/,"")
    gsub(/\">.*/,"");
    if ($0) print
}'`

curl -O ${filename}

The short version of what I am doing:

  1. for now we only care about a linux SDK, hence the os="linux". That said, you can see i am leaving it open to look for the SDK for another OS; you just need to know which pattern to look for.
  2. Get a copy of the page source.
  3. Look for the line that contains the link to the linux package.
  4. Once that is found, grab the link and fetch the file.

The awk line looks rather busy because the search (the id= thingie) part of the statement escapes out to grab ${os}. You could make it easier to understand by using the normal garden-veriety search -- grep -- command to get the line that matches the id="linux-tools pattern, which then is piped into awk:

#!/bin/sh
# getdroid.sh

site="https://developer.android.com/sdk/index.html"
os="linux"

filename=`curl -s --insecure ${site} | \
grep "id=\"${os}-tools" | \
awk '{
    gsub(/.*href=\"/,"")
    gsub(/\">.*/,"");
    if ($0) print
}'`

curl -O ${filename}

Of course, as I mentioned I want to also use the same script on OSX; as it it, it will run on a Mac, but I will leave fetching the OSX version of the SDK as an exercise to the reader.

How about the NDK? It is found at https://developer.android.com/tools/sdk/ndk/index.html. You can do the same thing as I did above, but have to account for having a 32 and a 64bit version; just provide that to the script or make it aware of the OS you are running. Personally I would not do the later because I might want to, say, download the Windows 64bit files from a Mac. I will post my final script on github as soon as I have time; for now, what I put here should be enough to get you started.

Monday, March 23, 2015

environment variables, date, and string concatenation in powershell

This is another of those quick posts. Sometimes in Linux/OSX I want (or even need) to rename or copy a file filename to path/filename_date. For instance, let's say the file is called cli64.log. I then can do something like

bash-3.2$ cp cli64.log cli64_`date +%Y%M%d-%H%M.log`
bash-3.2$ ls -lh cli64.log*
-rw-r--r--  1 dalek  staff   2.8K Feb 21  2013 cli64.log
-rw-r--r--  1 dalek  staff   2.8K Mar 23 16:29 cli64_20152923-1629.log
bash-3.2$
to append the date (as YearMonthDay which in this case turns out to be 20150323) and the time (HourMinute which when I did the above was 1629 or 4:29PM for those who cannot count past 12) to the name. So far so good.

As some of you have guessed -- maybe the title of this article was a dead giveaway -- I sometimes need to deal with Windows. And I do my best to make it behave as close to Linux (using Linux as placeholder for Linux/OSX/whatever since they behave the same in this case. In fact, the machine I ran the above command is a Mac Mini running OSX) as I can, which is why I use Powershell. So, how do I do the same copy command in Powershell?

Date

To get the date, the command we need is Get-Date. When you run it by itself, it gives something like

PS C:\Users\raub> get-date

Monday, March 23, 2015 4:41:13 PM


PS C:\Users\raub>
which is not useful for us; we want to make the date be part of the filename in the format we want. We will work this two part problem starting at the format and then worrying about the concatenation part.

According to the docs, and to http://ss64.com/ps/get-date.html (which has a convenient list of the time formats we can use), we can use the -format option. Let's try and see if we can replicate the output of date +%Y%M%d-%H%M:

PS C:\Users\raub> get-date -format 'yyyyMMdd-HHmm'
20150323-1643
PS C:\Users\raub>
That looks very similar to what we did in Linux. How abut adding that to the filename?

Concatenating

So, concatenating strings in Powershell is a bit interesting. Let's say we have $theTestFile=C:\Users\raub\monkey\testfile.txt and we want to append tmp to it. Now we can try a few things and see what we can come up with:

PS C:\Users\raub> echo "$theTestFile" + tmp
C:\Users\raub\monkey\testfile.txt
+
tmp
PS C:\Users\raub> echo "$theTestFile" += tmp
C:\Users\raub\monkey\testfile.txt
+=
tmp
PS C:\Users\raub> echo "$theTestFile += tmp"
C:\Users\raub\monkey\testfile.txt += tmp
PS C:\Users\raub> echo "$theTestFiletmp"

PS C:\Users\raub> echo "$theTestFile tmp"
C:\Users\raub\monkey\testfile.txt tmp
PS C:\Users\raub> Write-host "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub> echo "$($theTestFile)tmp"
C:\Users\raub\monkey\testfile.txttmp
PS C:\Users\raub>
So, the parenthesis thingie seems to be what we want to do.

Putting it all together

Now we know how to get the date and concatenate, if we put the date thingie inside the parenthesis thingie, the equivalent of

cp cli64.log cli64_`date +%Y%M%d-%H%M.log`
in powershell is
copy cli64.log "cli64_$(Get-Date -format 'yyyyMMdd-HHmm').log"

Er, we are not done yet

But, you will point out, the title of this article mentions environment variables! Fair enough. So we will expand the original problem. Say, you also want to name the copy of the log file to not only included when it was copied but also the hostname. Reason here is that you or someone else who will get this file might need to know where this file came from. In Linux, that can be done with $HOSTNAME, as in
cp cli64.log cli64_$HOSTNAME-`date +%Y%M%d-%H%M.log`
but what about Windows and powershell? Enter the environment variables we talked about. We would hope the computer knows what it is called, amongst other things, right? Let's see what it knows
PS C:\Users\raub> ls env:

Name                           Value
----                           -----
ALLUSERSPROFILE                C:\ProgramData
APPDATA                        C:\Users\raub\AppData\Roaming
CommonProgramFiles             C:\Program Files\Common Files
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files
CommonProgramW6432             C:\Program Files\Common Files
COMPUTERNAME                   VBOX01
ComSpec                        C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK               NO
HOMEDRIVE                      C:
HOMEPATH                       \Users\raub
LOCALAPPDATA                   C:\Users\raub\AppData\Local
LOGONSERVER                    \\ZOOL
NUMBER_OF_PROCESSORS           1
OS                             Windows_NT
Path                           %SystemRoot%\system32\WindowsPowerShell\v1.0\;C:\ProgramData\Oracle\Java\javapath;C:\...
PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE         AMD64
PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
PROCESSOR_LEVEL                6
PROCESSOR_REVISION             3c03
ProgramData                    C:\ProgramData
ProgramFiles                   C:\Program Files
ProgramFiles(x86)              C:\Program Files (x86)
ProgramW6432                   C:\Program Files
PSModulePath                   C:\Users\raub\Documents\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowe...
PUBLIC                         C:\Users\Public
SESSIONNAME                    Console
SystemDrive                    C:
SystemRoot                     C:\Windows
TEMP                           C:\Users\raub\AppData\Local\Temp
TMP                            C:\Users\raub\AppData\Local\Temp
UATDATA                        C:\Windows\CCM\UATData\D9FFC898-CBB8-491d-D8CA-173A9FF1B077
USERDNSDOMAIN                  EXAMPLE.COM
USERDOMAIN                     EXAMPLE
USERNAME                       raub
USERPROFILE                    C:\Users\raub
windir                         C:\Windows


PS C:\Users\raub>
That looks very impressive. How do we get that info in a way we can use when renaming out file? Here is how you get the hostname:
PS C:\Users\raub>$env:computername
VBOX01
PS C:\Users\raub>

So, if we wanted to copy poor cli4.log to some directory in some drive, adding to the new filename the hostname and date the file was copied, we could do a lot worse than

copy cli64.log "J:\kitchen\fridge\cli64_$env:computername-$(Get-Date -format 'yyyyMMdd-HHmm').log"

So I think we might have made managing Windows be a bit saner than before.

Friday, March 20, 2015

Quick notes on using git (gitolite) + NetBeans + ssh keys

Most of my posts (hopefully) talk about something that might be useful/helpful to others. Others are to help me not repeat a mistake. This is one of the latter ones; don't expect it to be very impressive. Really. Just look the other way, or at least have the decency of waiting until to laugh at my expense. Deal?

If you remember, I setup a git server using gitolite and docker not long ago. And then I found out one of the future users wanted to access it using NetBeans in Windows. As I have never used that IDE before, I looked in its website and found some info on how to make it talk to github. Well that sounded promising. First place I got stuck was creating the ssh key pair. You see, I know how to use ssh-keygen in Linux/Unix/OSX, but did not know how to do that in Windows. I could install cygwin and do it command line, and that would be great for me but not as nice for a typical Windows user. And, I would be installing a lot of crap this user did not need.

Searching around the net, I found a quick article about how to generate a ssh key on windows using putty (that is exactly how it is called). Now we are making progress; we just need to go and get puttygen, whcih can be obtained by itself, and create the keys. Following the last link, I created a key pair -- 4092 SSH2 RSA without passphrase since this is a test -- as shown in the picture on the left (you can click on it to make it bigger or something). Note the field I highlighted; I will refer to it later. For now, let's ignore that.

Here is the second place I screwed up. At first I thought the buttons to export the public and private keys were what I needed. So, I clicked on them and saved the files. The fact it wanted to give the private key the extension .ppk should have woken me up, but I dozed through that. Completely.

But, as I completely ignored that warning sign, I went back to the NetBeans Instructions and put the private key where it should be. And then I put the public key in my gitolite server. And then tried.

And it did not work.

So I decided to take a look at the keys. Here is the private one (trimmed out a bit to show the format while keeping this article small):

PuTTY-User-Key-File-2: ssh-rsa
Encryption: none
Comment: rsa-key-20150320
Public-Lines: 12
AAAAB3NzaC1yc2EAAAABJQAAAgEA1pA7YEyqbDyVAjjK+VCJpCGxfOwf8WGJ6J4Q
PaE1KMs0TBt5rEPlOpIdsJwOPMBvIUGG8hIHTyxrmOhxj221GYLPNSku7BdLCrt8
[....]
Q/G9k8opoE/0UPtFC0ykkGJg3Vhjq2XRGn3nYJaps7hP8ZUnvkiBuOmBR5SETnFi
3PrYFZ3lgYf1Nz2FgPMjg1uufIxWWv3MUb2Nya44x3JYQUkNfXCjUVmKiK/gE5sM
enufKa0=
Private-Lines: 28
AAACAFb8M79BrN/FiIRcNpxswi3IeGMTnj9DN+iuyFBWHHhSYU9JaK2e/BDTc9H6
E5wWqZfcS3bkttrofqXGBIZBO5S4fYRBINxAy2U3QG8mkBRWoBX/FuyztWBXxKug
pPIDWliV0oiTP0Q+PSXQ4LKTNcZm4UYlKg0Qk+ejR7FTpELqbv0nJQvMulvnEqVk
[...]
a6KahmtK1FFcFUtnLUFW+t7d59TYB/aB6HByzf5JDZPE9dtVpdBF33NSHe0Z4i2e
9XybtlNfYqKWEfaFzaeJqkyhRH5woG7le8GPU67BKVg/mKHYnG3Tk93NvhcOXuyx
7I2AKDeR1M0qMl0aH0ympymKoNxJHD2CBweLyJSoG3QdXY5DxoRvdn/gMeGRdLp0
mR80D0IN/tUduNT/69lB+I1oTU4N+WB/NcHCicOZyzHQRIKnmHC/raBM5ln2Q/ih
qtM=
Private-MAC: 56aa3a9bcb05a65a89110a5de990d5021cfb9273

It sure does not look like the ones I created using ssh-keygen. The public key also looked a bit different. Then it hit me: because gitolite uses openssh, it expects the key to be puttygen is exporting the keys in a different format. So, how do we make this work? Well, do you remember the field I highlighted in the first picture? That is the public key already in the ssh format; that is what gitolite needs. So, cut-n-paste that to, say, smurf.pub inside the keydir directory in the gitolite config file.

Next is to export the private key in the right format. That is done by clickign on Conversions->Exporting OpenSSH key, naming it as something helpful; I named mine smurf_rsa to remind me I happened to have created a RSA key.

Time to go back to Netbeans. The picture on the left shows the setup I used. I was in a hurry so I used the testing repo, which is a bit of a village bicycle in my server: everyone who can connect to localgit can access, read, and write to and in general monkey with it. The key was fed and then I told it to finish. This time it worked or seemed to: proceeded to want me to create a project (with all the little files and directories the IDE creates). I let it do the deed and later on was able to check out what was created.

The moral of this tale is make sure you use the right key format or things will get very interesting. Either that or only drink warm beer if the fridge was built by Lucas, the Prince of Darkness. Or something like that. On the bright side, you managed to reach the end of this tale! You now may put your seat and tray in the upright position and start to laugh.

I did.

Sunday, February 15, 2015

Backing up sqlite to another machine

Much have been written about backing up mysql/mariadb (mysqldump anyone?), but what about lowly sqlite? It might not have as many features and does not run as its own process, but sqlite is rather useful in its own right (embedded anyone?). If you have an android phone you are running sqlite. And that does not mean its data is not worth saving, so let's do some saving.

I will call the backup server backupbox and the machine running sqlite webserver. Note that usually a sqlite db is not run in a separate server like other databases, which is why in this example we claim it is a backend to some website.

Requirements and Assumptions

  1. sqlite3. After all, it is a bit challenging backing up a database if you do not have the database.
  2. Path to the database file you want to backup. Remember, sqlite is primarily a file; if you do not know where the file is, backing it up might pose a few challenges. In this example, it is found at /var/local/www/db/stuff.db.
  3. Both backupbox (IP 192.168.42.90) and webserver are running Linux or OSX. If there is an interest we can talk about when one (or both) of them are running Windows.
  4. We will break the backup process in two steps: backing up the database (to local drive) and then copying backup into the backup server. Reason is that we can then time each step to run at convenient times to them. Also, if one of the steps fail, it is much easier to take care of that. I know how monolithic do-everything-plus-clean-kitchen programs are the hip and fashionable solution nowadays, but this is my blog and I prefer simple and (hopefully) easy to maintain solutions whenever possible. Deal with it.

Procedure

  1. On server to be backed up, webserver
    1. Create backup user
      useradd -m backupsqlite -G www-data
      sudo -u backupsqlite mkdir -p /home/backupsqlite/.ssh
      touch /home/backupsqlite/.ssh/authorized_keys
      chown backupsqlite:backupsqlite /home/backupsqlite/.ssh/authorized_keys
      
    2. Create script to dump the database in ~backupsqlite/sqlite-backup.bz2.
      cat > /usr/local/bin/backupsqlite << 'EOF'
      #!/bin/bash
      BACKUP_USER=sqlitebackup
      BACKUP_GROUP=services
      DATABASE=/var/local/www/db/stuff.db
      
      sqlite3 ${DATABASE} ".dump" |
      sudo -u  ${BACKUP_USER} bzip2 -c > /home/${BACKUP_USER}/sqlite-backup.bz2
      chmod 0600 /home/${BACKUP_USER}/sqlite-backup.bz2
      EOF
      chmod 0700 /usr/local/bin/backupsqlite
    3. Run the above script manually as user sqlitebackup. Do not continue until this step is successful.
    4. Now you know your hard work paid off, how about running this script once a day? Maybe at 3:00am the natives will be quiet enough so you can safely run the backup script:
      cat > /etc/cron.d/backupsqlite << 'EOF'
      MAILTO=admin@example.com
      0 3 * * *  backupsqlite    /usr/local/bin/backupsqlite
      EOF
  2. On backup server, backupbox:
    1. Create ssh key pair to authenticate the connection. Note we are assuming we are will be running this script from backupbox's root account; that is not required and probably not the smartest thing to do, but will work fine for our little test. You could have used Kerberos or LDAP or something else, but would need to make changes as needed.
      ssh-keygen -f /root/.ssh/sqlitebackup-id-rsa
      You will need to copy sqlitebackup-id-rsa.pub to webserver and place it in ~backupsqlite/.ssh/authorized_keys by any means you want. If you are a better typist than me, you could even enter it manually.
    2. Test: can you retrieve the backup file?
      rsync -az -e "ssh -i /root/.ssh/sqlitebackup-id-rsa " \
      backupsqlite@webserver.example.com:sqlite-backup.bz2 .
      We can restrict this connection later. Right now let's just make sure this step works. If not, find out why before continuing.
    3. Now let's create the script that will get the database and put it, say, in /export/backup/databases
      cat > /usr/local/bin/backupsqlite << 'EOF'
      #!/bin/bash
      BACKUP_USER=sqlitebackup
      BACKUP_PATH='/export/backup/databases'
      DATABASE_SERVER=webserver.example.com
      $DATABASE=sqlite-backup.bz2
      KEY='/root/.ssh/sqlitebackup-id-rsa'
      
      cd $DATABASE_PATH
      rsync -az -e "ssh -i $KEY " $BACKUP_USER@$DATABASE_SERVER:$DATABASE .
      EOF
      chmod 0700 /usr/local/bin/backupsqlite

      Test it before continuing. Note there are other ways to do this step, like add the above rsync statement to a larger backup script; I tried to hint at that and the fact this could be the start of a function that would loop over all the servers you need to grab backup files from. So final implementation is up to you.

    4. If this backup is to be run independently, create a cron job to run it at a convenient time. How does 11:45 in the evening sounds?

      cat > /etc/cron.d/backupsqlite << 'EOF'
      MAILTO=admin@example.com
      45 23 * * *  root    /usr/local/bin/backupsqlite
      EOF

      Otherwise, tell your main backup script about this new entry.

  3. And back to webseerver
    1. Now time to make access to the database more restrict. We will be rather lazy here: we will start with what we have done in a previous article and make a few changes here and there.
      sudo -u backupsqlite cat > /home/backupsqlite/cron/validate-rsync << 'EOF'
      #!/bin/sh
      case "$SSH_ORIGINAL_COMMAND" in
      rsync\ --server\ --sender\ -vlogDtprze.iLsf\ --ignore-errors\ .\ sqlite-backup.bz2)
      $SSH_ORIGINAL_COMMAND
      ;;
      *)
      echo "Rejected"
      ;;
      esac
      EOF
      chmod +x /home/backupsqlite/cron/validate-rsync
      You probably noticed that I named the scripts running in both machines the same. Why not? If you do not like that, change it!
    2. Then we need to modify /home/backupsqlite/.ssh/authorized_keys to tell it to accept only connections from our backup server using our key pair and then only allow the backup script to be run. In other words, add something like
      from="192.168.42.90",command="/home/backupsqlite/cron/validate-rsync"
      to the beginning of the line in /home/backupsqlite/.ssh/authorized_keys containing the public key.

References

The Sqlite dump command

Wednesday, December 31, 2014

Restrictive rsync + ssh

Some of you have probably used rsync to backup files and directories from one machine to another. If one of those machines is in in an open network, you probably are doing it inside a ssh tunnel. If not, you should. And, it is really not that hard to do.

Let's say you wanted to copy directory called pickles inside the user bob's home directory at flyingmonkey.example.com, which is a Linux/Unix box out in the blue yonder. If you have rsync
installed (most Linux distros do come with it or offer it as a package), you could do something like:

rsync -az -e ssh bob@flyingmonkey.example.com:pickles /path/to/backup/dir/

The -e ssh is what tells rsync to do all of its monkeying about inside a ssh tunnel. And, when you run the above statement, it will then ask for bob's password and then proceed to copy the directory
~bob/pickles inside the directory /path/to/backup/dir. Which is great but I think we can do better.

Look Ma! No passwords!

First thing I want to get rid of is needing to enter a password. Yeah, it was great while we are testing it, but if we have a flyingmonkey loose in the internet, I would like to make it a bit harder for someone to break into it; I think I owe that to the Wicked Witch of the West.

The other reason is that then we can do the rsync dance automagically, using a script that is run whenever it feels like. In other words, backup. For this discussion we will just cover backup as in copying new stuff over old stuff; incremental backup is doable using rsync but will be the subject for another episode.

So, how are we going to do that? you may ask. Well, ssh allows you to authenticate using public/private key pairs. Before we continue, let's make sure sshd in flyingmonkey is configured to accept them:

bob@flyingmonkey:~$ grep -E 'PubkeyAuthentication|RSAAuthentication' /etc/ssh/sshd_config 
#RSAAuthentication yes
#PubkeyAuthentication yes
#RhostsRSAAuthentication no
bob@flyingmonkey:~$
Since PubkeyAuthentication and RSAAuthentication are set to yes, we are good to go. Now if flyingmonkey runs OSX, you would want to use /etc/ssh/sshd_config instead.

A quick note on ssh keys: they are very nice way to authenticate because they make life of whoever is trying to break into your machine rather hard. Now, just guessing the password does not do you much good; you need to have the key. And, to add insult to injury, you can have a passphrase in the key itself.

Enough digressing. The next step is to create the key pair. The tool I would use in Linux/Solaris/OSX is ssh-keygen because I like to do command line thingies. So, we go back to the host that will be rsnc'ing to flyingmonkey and create it by doing

ssh-keygen -b 4096 -t rsa -C backup-key -f ~/.ssh/flyingmonkey
which will create a 4096 bit (a lot of places still use 1024 and some now are announcing they have new state-of-the-art ultra secure settings of 2048 bits. So unless your server can't handle it, use 4096 or better) RSA key pair called flyingmonkey and flyingmonkey.pub in your .ssh directory:
raub@backup:~$ ls -lh .ssh/flyingmonkey*
-rw------- 1 raub raub 3.2K Dec 31 11:30 .ssh/flyingmonkey
-rw-r--r-- 1 raub raub  732 Dec 31 11:30 .ssh/flyingmonkey.pub
raub@backup:~$
During the creation process, it will ask for a passphrase. Since we are going to have a script using this keypair, it might not make sense to have a passphrase associated to it. Or it might, and there are ways to provide said passphrase to script in some secure way. But this post is getting long so I will stick to the easy basic stuff. If you remember, we said this is a public/private key authentication; that means it uses to keys: public and private. The public is taken to the machine you want to ssh into while the private stays, well, private. Let's look at the public key (it is a single line):
raub@backup:~$ cat .ssh/flyingmonkey.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACACsgpy/ihq31kv+Zji6Eknr46nbyx38uPE54X3STbaNC8oCheulVk
/+bTmrFy8Ne8RcTeWYd93wwabgBVDJYzjnsuwUgBO/JPXE4GiQrcnIz5fPsqJqslYxR5WnuUfkYPsAYgJL33XWZWi
dPu+A38OSxXf7UAfpKe5WXa93knXIERUA7NOzCKO96YzpW96i7LxAs20bsmNAw6bZbrZG8Dn3EssIK8CtEUvw4nWb
uFKZQS+b5AM8q20+IrGGVG193H6Rm3/iw0jip0VQOFozUB6yjToyZ5MTzShjb+f56o3+VUG2Bel7OMDfYXYYEKIoj
+cmTMLP5yu4v1t5dTkN3osneK/+2KHwXFTQY48TqxyqH+ZqEFy2X+kXoKff/89aD8lwj+uYKl6HNKhveKSZMNq/yc
7jCc05hLQCQkyC9/1lY9LI2UMHq2kqsgbdmR3uu3Oua2y1HhyeR9hqP9Om+kLu2K7cIXu5NBO9ro0vWBJII7T+z98
awbGH4jSryOxvAlpTT7d+POev13oOWonIwyTmkT72Q+/qJhPU/Vdtd7n5gSUomRT8dJQH+2hyA8c3+YPSW2VckBY/
Ax5aGX+AFoy1Y6WKpUWMIbwHJqizpdEd3WQWzivR1psfsjFzqrdG5SOSZFH2SHvzdNQOTz0FbYvgBV2Egq7WXv98C
se9ZDx backup-key
raub@backup:~$ 
You probably notices the backup-key string on the end of the file; we put it there using the -C (comment) option. Usually it writes the username@host, which is OK most of the times but I wanted something to remind me of what this key is supposed to do. You can change it later if you want.

So we go back to flyingmonkey and place the contents of the public key, flyingmonkey.pub in ~bob/.ssh/authorized_keys by whatever means you want. cut-n-paste works fine.

cat flyingmonkey.pub >> authorized_keys
also does a great job. Or you can even use ssh-copy-id if you feel frisky. Just remember the contents of flyingmonkey.pub is a single line. Of course, if flyingmonkey is a windows machine, you will do something else probably involving clicking on a few windows, but the principle is the same: get the bloody key into the account in the target host you want to connect to.

Once that is done, connect using ssh by providing the key

ssh -i .ssh/flyingmonkey bob@flyingmonkey.example.com

Can you login fine? Great; now try rsync

rsync -az -e 'ssh -i .ssh/flyingmonkey' bob@flyingmonkey.example.com:pickles /path/to/backup/dir/
Do not continue until the above works. Note in a real script the private key will probably be somewhere only the user which runs the backup script can access.

Limiting

So far so good. We eliminated the need to use a password so we can write a script to use the above. But, we can still ssh using that key to do other things besides just rsync. Time to finally get to the topic of this post.

If the IP/hostname of the host you are backing up flyingmonkey from does not change, you can begin by adding that to the front of the ~bob/.ssh/authorized_keys entry for the flyingmonkey public key. Now, if the backup server is in a private/NATed lan, you want to use the IP for its gateway. In this example, let's say we all all inside a private lan and the IP for backup server is 192.168.42.24:

from="192.168.42.24" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACACsgpy/ihq31kv+Zji6Eknr46nbyx38uPE54X3STbaNC8oCheulVk
/+bTmrFy8Ne8RcTeWYd93wwabgBVDJYzjnsuwUgBO/JPXE4GiQrcnIz5fPsqJqslYxR5WnuUfkYPsAYgJL33XWZWi
dPu+A38OSxXf7UAfpKe5WXa93knXIERUA7NOzCKO96YzpW96i7LxAs20bsmNAw6bZbrZG8Dn3EssIK8CtEUvw4nWb
uFKZQS+b5AM8q20+IrGGVG193H6Rm3/iw0jip0VQOFozUB6yjToyZ5MTzShjb+f56o3+VUG2Bel7OMDfYXYYEKIoj
+cmTMLP5yu4v1t5dTkN3osneK/+2KHwXFTQY48TqxyqH+ZqEFy2X+kXoKff/89aD8lwj+uYKl6HNKhveKSZMNq/yc
7jCc05hLQCQkyC9/1lY9LI2UMHq2kqsgbdmR3uu3Oua2y1HhyeR9hqP9Om+kLu2K7cIXu5NBO9ro0vWBJII7T+z98
awbGH4jSryOxvAlpTT7d+POev13oOWonIwyTmkT72Q+/qJhPU/Vdtd7n5gSUomRT8dJQH+2hyA8c3+YPSW2VckBY/
Ax5aGX+AFoy1Y6WKpUWMIbwHJqizpdEd3WQWzivR1psfsjFzqrdG5SOSZFH2SHvzdNQOTz0FbYvgBV2Egq7WXv98C
se9ZDx backup-key
This is a small improvement: only host that can connect is the one with this IP, be it legit or faking that. Test it.

Next step is specify which commands that can be run when connected using this key. And that one again will require playing with ~bob/.ssh/authorized_keys. This time we will specify the command:

from="192.168.42.24",command="/home/bob/.ssh/validate-rsync" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACACsgpy/ihq31kv+Zji6Eknr46nbyx38uPE54X3STbaNC8oCheulVk
[...]
se9ZDx backup-key
And define validate-rsync as
cat > .ssh/validate-rsync << 'EOF'
#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
rsync\ --server\ --sender\ -vlogDtprze.iLsf\ .\ pickles)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac
EOF
chmod +x .ssh/validate-rsync
And this is where it get really exciting. All that validate-rsync is doing is seeing if the command being sent is not only an rsync command but a specific one. Once we figure out how to get the proper SSH_ORIGINAL_COMMAND, we can change the line
rsync\ --server\ --sender\ -vlogDtprze.iLsf\ .\ pickles)
to what it needs to be to match our backup script and test. Note that if you change the rsync statement, you will need to change the case.

Friday, December 26, 2014

Getting the SSH_ORIGINAL_COMMAND

Let's say you want to have an account you can ssh into but only run very specific commands in it. A good way to achieve that is to write a wrapper script that is called from your authorized_keys file. So you could have a wrapper that looks like this:

#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
    "/usr/bin/rsync "*)
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Permission denied."
        exit 1
        ;;
esac
But, what if you really want to be really precise on the command? Using the above example, not only running rsync but also specifying the path and the arguments? You could cheat and find what the command you are sending is supposed to look like by replacing (temporarily) your wrapper script with this
/bin/sh

DEBUG="logger" # Linux
#DEBUG="syslog -s -l note" # OSX

if [ -n "$SSH_ORIGINAL_COMMAND" ]; then
        $DEBUG "Passed SSH command $SSH_ORIGINAL_COMMAND"
elif [ -n "$SSH2_ORIGINAL_COMMAND" ]; then
        $DEBUG "Passed SSH2 command $SSH2_ORIGINAL_COMMAND"
else
        $DEBUG Not passed a command.
fi
Then you run the ssh command and see what it looks like in the log file. Copy that to your original wrapper script, and you are good to go. So
ssh -t -i /home/raub/.ssh/le_key raub@virtualpork echo "Hey"
Results in
Dec 26 13:34:05 virtualpork syslog[64541]: Passed SSH command echo Hey
While
rsync -avz -e 'ssh -i /home/raub/.ssh/le_key' raub@virtualpork:Public /tmp/backup/
results in
Dec 26 13:28:17 virtualpork syslog[64541]: Passed SSH command rsync --server 
--sender -vlogDtprze.iLs . Public
The latter meaning our little wrapper script would then look like
#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
    "rsync --server --sender -vlogDtprze.iLs . Public")
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Permission denied."
        exit 1
        ;;
esac

Saturday, December 13, 2014

Adding a disk to a libvirt/kvm vm client

So you have a kvm virtualization infrastructure which you, being lazy like me, manage using libvirt. You then created a vm client with the virtual disk partitioned just right (using LVM or not; pick your poison). But later you realized you needed another disk. Maybe it is because you need to have data encrypted to meet HIPAA or PCI requirements. Maybe you just want to keep your data in a different drive. The point is you need another drive and don't want to/should not resize the current virtual disk associated with the VM.
So, first thing you do is create the new disk. If you are using KVM, chances are you have been using qcow2 disks. Or maybe vmdk, iSCSI, or, like me, lvm. No matter what, you probably know how to create a new disk, so I will assumed you took time to figure out how large it needs to be to fit your needs and created the little bastard. Because, as I mentioned before, I am lazy, I will say we are running libvirt in linux, with a vm client called vmclient and creating LVs to use as virtual disks. So, we need a 10GB virtual disk that we'll call data because we are friends with Captain Obvious.
lvcreate -L 10G -n data vmhost_vg0
creates as we know a 10GB lv as /dev/vmhost_vg0/data. To make it easier on us, we will shut down the vm client. Once that is done (do check it using virsh list --all, will you?), we then run
virsh edit vmclient
Note we could have created a properly configured xml file and fed into the config, but I do forget how to make it properly configured so I prefer to cheat.
When the config file is open, look for the disk entries; they should look like this:
    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/vmclient_boot"/>
      <target bus="virtio" dev="vda"/>
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci"/>
    </disk>

which is how our original virtual disk is configured to be used in this vm client. Of course, if you were using qcow, vmdk, or something else, the entry might look a bit different; make a note of how it differs from my example and move on. This is why I said I like to cheat: I can see how the old disk was defined and copy that instead of trying to figure out how to do it.

Now you need to add the new drive. As you guessed from the above, we will do it by copying the above entry and changing it a bit. Now, we do not need to copy everything; we just need enough so virsh knows what we want. It will fill the blanks. So, after we copy the relevant bits below the already defined disk and change the drive name, we would have something like this

    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/data"/>
      <target bus="virtio" dev="vda"/>
    </disk>

Save it; it should close without issues. If not, got back and see if you missed something. If it did not bark, use virsh dump xml vmclient to see your handiwork. Mine looks like this:

    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/vmclient_boot"/>
      <target bus="virtio" dev="vda"/>
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci"/>
    </disk>
    <disk device="disk" type="file">
      <driver cache="none" io="native" name="qemu" type="raw"/>
      <source file="/dev/vmhost_vg0/data"/>
      <target bus="virtio" dev="vda"/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/%gt
    </disk>

As you can see, it added the pci bus address entry on its own. Now all we need to do is reboot the vmclient, format the new disk on whatever way we want, and start using it.

Friday, November 08, 2013

Getting IP and MACs for hosts in a network without using nmap

Well, sometimes you need to find out which IPs in a given network are being used and what is the MAC addresses associated with them. Perhaps you want to make sure the machines are the ones that should be there (MAC spoofing notwithstanding). Or, as it happened to me before, you need to see which IPs your dhcp server have given out are actually being used. You can come up with other reasons too, even if they arenot honorable! Impress your friends! Be the life of the party!

Anyway, you can do some of this using nmap, but what if it is not available or you just want to use common Linux household commands? I have done something like this before using nslookup, but since dig is supposed to replace it, how about if we rewrite my old cold to use it? So, let's say you want to know what is in 10.0.0/24? You could do something like:

subnet=10.0.0
for i in $(seq 1 254)
  do ans=`ping -qnc 1 $subnet.$i | grep -c '100% packet loss'`
  [ "$ans" == 1 ] || echo "(+) $subnet.$i (`dig -x $subnet.$i +short` `arp -an $subnet.$i|awk '{ print $4 }'`) "
done

Or something a bit fancier, which would ask you to enter the first 3 octets of the network:

nonono() {
  printf "Enter subnet (only the first 3 octets): "
  read subnet
  for i in $(seq 1 254)
    do ans=`ping -qnc 1 $subnet.$i | grep -c '100% packet loss'`
    [ "$ans" == 1 ] || echo "(+) $subnet.$i (`dig -x $subnet.$i +short` `arp -an $subnet.$i|awk '{ print $4 }'`) "
  done
}
nonono

Here is the code in action:

raub@desktop:~$ nonono
Enter subnet (only the first 3 octets): 10.0.0
(+) 10.0.0.1 (router. 00:24:54:9s:2a:12) 
(+) 10.0.0.3 (brownie.my.domain.com. 64:6b:b3:b0:76:e1) 
(+) 10.0.0.16 (cookie.my.domain.com. 02:50:4d:c4:17:1a) 
(+) 10.0.0.18 (tomato.my.domain.com. 02:50:4d:c4:17:1a) 
(+) 10.0.0.19 (vmhost.my.domain.com. bc:5f:f4:ad:d7:8d) 
(+) 10.0.0.21 (scan.my.domain.com. c0:ff:ee:4f:96:a9) 
(+) 10.0.0.110 (pickles.my.domain.com. 00:9f:f3:46:23:90) 
(+) 10.0.0.238 (pizza.my.domain.com. c0:ff:ee:67:1c:3c) 
(+) 10.0.0.249 (desktop.my.domain.com. entries) 
raub@desktop:~$ 

Some stuff worth mentioning:

  1. Some machines have MAC beginning with c0:ff:ee. Those are VMs of mine running of vmhost; I use that so I can quickly identify them as VMs; you might want to follow the same idea if you have to deal with large amounts of VMs in server rooms.
  2. cookie and tomato have the same MAC. The reason is they are the same machine. I just configured its interface to do interface aliasing (eth0:0 and eth0:1 for instance) so one IP could be for a fileserver and another for a web server (really bad idea, which is why I thought you would like it). You can read about it in, say, here.
  3. Do note what we are calling subnet really isn't; it is just the first 3 octets in a class C network. In other words, it assumes your network is of the type a.b.c.0/24. You could change the code to handle any network provided the network IP and subnet mask; I will leave that as an exercise to you.
  4. If you want to run it in OSX, use arp -n instead of arp -an.

Tuesday, March 19, 2013

On directory/folder and group ownership

Easy problem here: we have this directory, say /export/projects/web which is supposedly owned by the group developers,
raub@banana:~$ ls -lhd /export/projects/web
drwxrwxr-x 19 bob developers 4.0K 2013-03-19 13:36 /export/projects/web
raub@banana:~$
The idea is that it is a shared folder, a place the developers can put share files amongst themselves without others being able to change/delete them. In other words, we want any file or directory created inside projects to inherit its group ID. At east that is the idea. In reality when any member of that group creates a file there, it is owned by that user's default group, not by developers.
raub@banana:~$ touch /export/projects/web/here
raub@banana:~$ ls -lh /export/projects/web
-rw-rw-r--. 1 raub raub       0 Mar 19 13:15 here
raub@banana:~$ 
We can do something about it. First we set the setgid bit to make sure /export/projects/web is owned by developers:

raub@banana:~$ chmod g+s /export/projects/web
raub@banana:~$ ls -lhd /export/projects/web
drwxrwxr-x 19 bob developers 4.0K 2013-03-19 13:36 /export/projects/web
raub@banana:~$ ls -lh /export/projects/web
-rw-rw-r--. 1 raub developers 0 Mar 19 13:15 here
raub@banana:~$

Then, we should find all files in that directory with different groups and set them to be owned by developers:

raub@banana:~$ for i in `find /export/projects/web -type f ! -group developers`; do chown :developers $i; done
raub@banana:~$

Did it work? Let's find out!

raub@banana:~$ touch /export/projects/web/here
raub@banana:~$ ls -lh /export/projects/web
-rw-rw-r--. 1 raub developers 0 Mar 19 13:15 here
-rw-rw-r--. 1 raub developers 0 Mar 19 13:15 there
raub@banana:~$ 

References:

http://superuser.com/questions/102253/how-to-make-files-created-in-a-directory-owned-by-directory-group
http://www.cyberciti.biz/faq/how-do-i-find-all-the-files-owned-by-a-particular-user-or-group/

Monday, January 28, 2013

Restoring time on sleeping (linux) vms

One of the most annoying issues in a VM is keeping accurate time.When a vm client is saved/paused and then restored, its clock will be still set to whatever time was when it was paused. That can lead to many issues including not being able to login using Kerberos. Rebooting the machine will force the clock to be reset, but what if we do not want to (or cannot) reboot? What we could use a script that will monitor the drift and if it is too far -- say one minute off -- it will automagically adjust the clock in the vm client.
There are many virtualization packages out there, some of which (in no order) are VMWare ESX/ESXi, Virtual Box, kvm, and Microsoft Virtual Server. The first three I personally have used; if you want to see a more complete list I would suggest to try the Wikipedia entry on Hypervisors. Suffice to say some of those programs allow host machine (the physical machine running the program) expose its clock to the guest machine (or vm), others do not. So let's see how we can check and set the clock based on whether the guest can see the host clock or not. For this discussion I will stick to Linux because I feel lazy today.

Using the vm host clock

Let's say the vm program of your choice can pass the vm host's clock to the client (I do know that KVM can do that and believe VMWare ESXi can too), you could write something like this:
cat > /usr/local/bin/driftcheck << 'EOF'
#!/bin/sh
### Detect drift in the vm client vh most. If it is massive; adjust
### it.

HWCLOCK=/sbin/hwclock

# Max drift in seconds. Kerberos does not like time offsets > 5min, 
# so we set it to 1m = 1*60s
MAX_DRIFT=` echo "1*60" | bc -l`

VMHOST_TIME=`date +'%s' -d "$(${HWCLOCK} -r -u | cut -d' ' -f-7)"`
MY_TIME=`date +%s`
DRIFT=$( echo "${VMHOST_TIME} - ${MY_TIME}" | bc | tr -d -)

if [ "${DRIFT}" -gt "${MAX_DRIFT}" ]
then   
        ${HWCLOCK} -s -u
fi
EOF
chmod +x /usr/local/bin/driftcheck
What it does is get the hardware clock of this vm client, which actually is the vm host's clock. One thing this requires is the vm client is configured to take the vm host's clock in utc time. YMMV here, but if you are using kvm and libvit, you would have a line like this
<clock offset="utc"></clock>
somewhere in the xml file defining the client.
The commented lines inside the if statement are there just so when you test it out you can see what is going on. For production you probably want leave them commented out
Now, we would probably want to to have it being called often but not really crazy. So, for now let's say we create a cron job to run driftcheck every 5 minutes?
cat > /etc/cron.d/driftcheck << 'EOF'
*/5 * * * * root /usr/local/bin/driftcheck > /dev/null 2>&1
EOF
Of course, you should adjust it to fit your needs. Do note I put driftcheck in /usr/local/bin/; it just felt like a nice place this season.

Using NTP

As we mentioned above, sometimes we cannot (or will not) use the clock off the vm host. If we are using a ntp server, we are good. Ok, you might argue but, if the drift/skew is too long, ntpd will refuse to adjust the clock. And, even if we force it, it will take hours or even days!. Not if we nudge it a bit by using an old friend, ntpdate:
cat > /usr/local/bin/driftcheck << 'EOF'
#!/bin/sh
### Detect drift against a reliable time source. If it is massive;
### instead of relying on ntp (which will not work), do something
### a bit more drastic

NTPQ=/usr/bin/ntpq
NTPDATE=/usr/sbin/ntpdate

# Max drift in ms. Kerberos does not like time offsets > 5min, so
# we set it to 1min = 1*60*1000ms
MAX_DRIFT=` echo "60*1000" | bc -l`

# Find the ntp server we are using in this host
NTP_SERVER=`sed -ne '/^server/p' /etc/ntp.conf | awk '{ print $2 }'| head -1`
# Get current drift
DRIFT=`${NTPQ} -p ${NTP_SERVER}|grep '*'|awk '{print $9 }'`
[ -z $DRIFT ] && DRIFT=`${NTPQ} -p |tail -n +3| awk '{print $9 }'`
DRIFT=$( echo "${DRIFT}/1" | bc | tr -d -)

if [ "${DRIFT}" -gt "${MAX_DRIFT}" ]
then
        echo ${NTPDATE} -u ${NTP_SERVER} >> /tmp/ntp
        ${NTPDATE} -u ${NTP_SERVER} >> /tmp/ntp
fi
EOF
chmod +x /usr/local/bin/driftcheck
The reason we want to look for * is that according to the
NTP troubleshooting page, it is the source you are currently synchronized to.
The cron file will be the same as what we had in the hardware clock session, so I decided not to copy it.
And that is pretty much it. Remember they are just ideas. Those scripts should work as is but you can/should customize/adapt them. If you have any questions, concerns, or just want to confuse me, do leave a message.

References

Wednesday, February 11, 2009

Of Macs and serial ports

My trusty iBook, as all Macintosh computers manufactured in the last few years, have no serial port. That has never stopped me from doing work as I had a Linux laptop, a Dell Latitude D600, which I would bring whenever I needed to talk to a Cisco switch or use as console for a Unix workstation (say, Sun Solaris or IBM AIX box).

But, then, the Dell laptop died. And I needed to configure a cisco switch from scratch... at least configure it enough so I could then telnet to it. To do that I needed to connect the famous Cisco blue console cable to the Mac. I needed a usb-to-serial cable.

Not knowing where to find one of those usb-to-serial cables, I decided to try one of my favorite places: geeks.com. I not only found it but here is a picture of the cable:

Clicking on the image *should* lead you to the link for the cable. After I received it, I connected it to the Mac. The laptop was aware of the device, even recognizing its chipset. But, it would not be available for use. Here is what I mean:

Mireille:~ dalek$ ls /dev/tty.*
/dev/tty.Bluetooth-Modem                /dev/tty.Nokia6103-NokiaPCSuite-1
/dev/tty.Bluetooth-PDA-Sync             /dev/tty.modem
/dev/tty.Nokia6103-Dial-upnetwor-2
Mireille:~ dalek$ 

Clearly, I need a driver for it. Examining the information shown by the machine about the driver, we see the chipset is made by prolific. After a bit of searching online, I found the manufacturer's site and downloaded the drivers from its site. Do note in that page that they also have drivers for Windows and even Linux. I do not know if Linux would ever need such a driver; finding that out is for a different episode. Anyway, after installing it, we had to reboot the laptop. After that, it was time to connect the usb-to-serial cable and find out if it was seen as a device we could use. Can you spot the new entry?

Mireille:~ dalek$ ls /dev/tty.*
/dev/tty.Bluetooth-Modem                /dev/tty.Nokia6103-NokiaPCSuite-1
/dev/tty.Bluetooth-PDA-Sync             /dev/tty.modem
/dev/tty.Nokia6103-Dial-upnetwor-2      /dev/tty.usbserial
Mireille:~ dalek$ 

Now we have a device, tty.usbserial, we can try it out. We could install minicom using fink, but we can be a bit lazy and use, of all things, screen. Believe it or not, screen can also be used to connect to a terminal device. So, if you type something like

Mireille:~ dalek$ screen /dev/tty.usbserial 9600

you would be telling screen to connect to our usb-to-serial cable, identified as tty.usbserial, at 9600baud which happens to be the default port speed for a Cisco switch. Neat, huh?