Using ldapsearch and ldapmodify to talk to Active Directory

Why?

Great question! Here are a few lame excuses I was able to come with:

• I like to use command line. This is a lame excuse because Windows have powershell. But,
  
• I am more comfortable with Linux than Windows. Lame excuse since
  1. How many posts in this very blog I have made about using Windows?
     
  2. How many of said posts I have used the GUI when I could take care of business with Powershell?
     
  3. What is wrong with Powershell, at least of the applications I have used it for here so far?

I do have a couple of not so lame ones though:

• I like to be able to access the network resources from any machine in the network running any OS. If I have a Linux box in an Active Directory-controlled network, chances are I will need to authenticate the Linux box against Active Directory (AD so I can save some keytaps). AD is Kerberos + ldap + sprinkles, so I better be able to use the usual kerberos/ldap Linux tools as one day I will need to figure out why things are boink.
  
• It feels like I get more info using ldapsearch than the Windows tools, which is good when I do not know the name of an attribute, or how many instaces of said attribute are in use.

Using ldapsearch

Before we go mindlessly typing things, we need some data.

We need to know the name of the ldapserver.

Yes, if you have it configured in your ldap.conf file, you should not need it. But I prefer to assume nothing. If the domain was setup properly, we can ask it directly by typing nslookup -type=srv _ldap._tcp.DOMAIN where DOMAIN is the Active Directory domain name, not the DNS one; that caught me off guard. So, if our DNS dmain is example.com and the AD domain (we are very original) is ad.example.com, we have

raub@desktop:/tmp$ nslookup -type=srv _ldap._tcp.ad.example.com
Server:         192.168.0.10
Address:        192.168.0.10#53

Non-authoritative answer:
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC0.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC2.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC1.ad.example.com.

Authoritative answers can be found from:
ad.example.com      nameserver = addc1.ad.example.com.
ad.example.com      nameserver = ns.example.com.
ad.example.com      nameserver = ns2.example.com.
ad.example.com      nameserver = addc0.ad.example.com.
ad.example.com      nameserver = addc2.ad.example.com.
ADDC0.ad.example.com        internet address = 192.168.1.100
ADDC1.ad.example.com        internet address = 192.168.1.102
ADDC2.ad.example.com        internet address = 192.168.1.101
ns.example.com      internet address = 192.168.0.10
ns2.example.com     internet address = 192.168.0.10

raub@desktop:/tmp$

and we can use any of the ADDCN.ad.example.com (where N=0,1,2). Notice in my setup, just using ad.example.com also worked. I found out by using netcat to see if port 636 was open (I will leave the answer for "why port 636?" as an exercise to the reader)

raub@desktop:~$ nc -v ad.example.com 636
Connection to ad.example.com 636 port [tcp/ldaps] succeeded!
^C
raub@desktop:~$ 

We need to be able to authenticate against AD somehow.

For this discussion I will be using a username and password; we can also do it using a Kerberos TGT ticket.

Fun Fact: I have a user account, my normal one, which can look into some things in LDAP/AD but then I have another ("admin") account I can see more and edit stuff in AD. You will see later on me forgetting completely about that and how it affects me. But we are getting ahead of ourselves.

With that taken care of, we are going to begin by looking for some user: me

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" 
Enter LDAP Password:
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
objectClass: top
objectClass: posixAccount
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Wrong Droid
sn: Droid
title: Entropy Creators
description: Orthodontics
givenName: Wrong
initials: B 
distinguishedName: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
instanceType: 4
whenCreated: 20080109142820.0Z
whenChanged: 20180905201157.0Z
displayName: Droid, Wrong
uSNCreated: 243336
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
[...]
proxyAddresses: sip:raub@ad.example.com
proxyAddresses: smtp:raub@ad.example.com
proxyAddresses: X500:/o=UNC Exchange/ou=Exchange Administrative Group (FYDIBOH
 F23SPDLT)/cn=Recipients/cn=raub
proxyAddresses: SMTP:raub@email.example.com
displayNamePrintable: Wrong Droid
name: Wrong Droid
[...]
sExchPoliciesExcluded: {26491cfc-9e50-4857-861b-0cb8df22b5d7}
msExchUserAccountControl: 0
msExchELCMailboxFlags: 2
msRTCSIP-PrimaryHomeServer:
[...]
msExchOWAPolicy: CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=ad,DC=example,DC=com

# refldaps://ForestDnsZones.ad.example.com/DC=ForestDnsZones,DC=ad,DC=example,DC=com

# refldaps://DomainDnsZones.ad.example.com/DC=DomainDnsZones,DC=ad,DC=example,DC=com

# refldaps://ad.example.com/CN=Configuration,DC=ad,DC=example,DC=com

raub@desktop:~$

That is probably more info than you wanted to know about someone, but this has its applications. Since we now know every attribute associated with a given user (I should not be that special, at least as far as AD is concerned), we can build customized queries looking for only a specific bit of info. For instance, let's just get the groups I belong to or am a member of (hint hint):

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" memberOf 
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com

Fancy, huh? Now, if instead of doing (CN=raub) we did (CN=*raub*), it would return every entry that has a CN with raub in it. In my case that would mean two entries, raub and raub.admin (if you remember our Fun Fact you will know about it), but it could also have returned a device whose name matches that. And, they would have been printed one after the other (I do wonder if the order depends on the order they were added to LDAP/AD).

Using ldapmodify

Ok, we established we can probulate Active Directory using common household Linux/UNIX query tools. What if we want to change something? Let's say we have an AD group (specifically a distribution list) called moustache_operators (for those who own and operate moustaches) and want to add a member and delete another.

Why I do like ldapmodify to edit LDAP/AD

Main reason is because I can create a file (in the LDIF format) at my leisure (i.e. think about what I want to do) with pretty commands and comments describing what I want to do. If I like what I did, I can then document it and maybe even save the file in a wiki or somewhere that can be fed to Ansible/Puppet/Chef/Docker and reused.

Our little LDIF file, let's call it change.ldif, could look like this:

# Let's define the entity we will be fiddling with
dn: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com
# And then what we will be doing with it
changetype: modify
delete: member
member: CN=baldone,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
# Separator because we will be doing another change
-
add: member
member: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com

So let's try it:

raub@desktop:~$ ldapmodify -H "ldaps://addc0.ad.example.com:636" -D "raub@ad.example.com" -x -W -f change.ldif
Enter LDAP Password:
modifying entry "CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com"
ldap_modify: Insufficient access (50)
        additional info: 00002098: SecErr: DSID-03150F93, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

raub@desktop:~$ 

Why is it not working? Well, do you remember the Fun Fact I mentioned earlier in this article? This is how it shows it's ugly head. I should have used my raub.admin@ad.example.com account instead of raub@ad.example.com. If we do it right, it then works. I will not show the output of a successful connection here; what matters is verifying the deed is done, and we can do it using dear ol' ldapsearch:

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub) memberOf" dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com

What about Mac/OSX?

They too have ldapsearch; just use the terminal and off you go.

generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()

This will be a quick post about something that was biting my ass these last few days and what was the real cause. After you read it, you are welcome to laugh at my expense. Go ahead! I deserve it!

I was working in a kerberos/ldap (linux) server and needed to debug the connection to a given client. The ldap connection uses TLS, GnuTLS specifically since the two machines were ubuntu servers, which means we also had to worry about certs. And since kerberos is in the picture, we need to configure for that. To help in solving other issues, which I should comment about later (at least those were clever problems not like this one), I was running slapd in debug mode,

/usr/sbin/slapd -d 256 -h "ldap:/// ldapi:/// ldaps:///" -g openldap -u openldap -F /etc/ldap/slapd.d

and that did help solve the other issue I had. Some of you will notice I am also running ldaps (port 636), which I really do not need since TLS should take care of the encryption thingie. But, I digress for this post, so let's go back on topic. What I then noticed was some very problems with ldap. For instance, if I created a kerberos ticket and then tried to run ldapsearch, I would then get the following error:

root@services:~# export KRB5CCNAME=/tmp/host.tkt
root@services:~# ldapsearch -vvv
ldap_initialize(  )
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Other (e.g., implementation specific) error (80)
        additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
root@services:~#

Here is what the server sees:

53261bde conn=1043 fd=19 ACCEPT from IP=192.168.1.181:44610 (IP=0.0.0.0:389)
53261bde conn=1043 op=0 EXT oid=1.3.6.1.4.1.1466.20037
53261bde conn=1043 op=0 STARTTLS
53261bde conn=1043 op=0 RESULT oid= err=0 text=
53261bde conn=1043 fd=19 TLS established tls_ssf=128 ssf=128
53261bde conn=1043 op=1 BIND dn="" method=163
53261bde SASL [conn=1043] Failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
53261bde conn=1043 op=1 RESULT tag=97 err=80 text=SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
53261bde conn=1043 op=2 UNBIND
53261bde conn=1043 fd=19 closed

Since I do not have many clever things to talk about and fill the space until the solution, how about if we talk about what some of those lines mean?

• IP=192.168.1.181:44610 (IP=0.0.0.0:389): Client 192.168.1.181 is connecting from its port 44610 to my port 389.
  
• oid=1.3.6.1.4.1.1466.20037: Start TLS extended request (per rfc2830).
  
• BIND dn="": anonymous if we are doing a SIMPLE bind. If we are however doing SASL bind, it is not used.
  
• tag=97: result from a client bind operation.

As you noticed, at least from reading the title of this post, the error line is this generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () thingie. Here is where it annoyed me to no end: what minor code? It is supposed to put some kind of message between the parenthesis, like "No principal in keytab matches desired name" or "Ticket expired". Then I would be able to search online for something. Instead, zilch. I could not find a single entry where the minor code parenthesis thingie was empty. Not very helpful today are we?

Solution

So, what was wrong? Me. User error. Do you remember how I was running slapd? Do you also remember the part about kerberos? Well, in the /etc/default/slapd (that'll be /etc/sysconfig/ldap for you RedHat/CentOS/Fedora folks) I have defined

export KRB5_KTNAME=/etc/ldap/ldap.keytab

which means ldap knows then where the keytab containing the ldap service principal hides. Can you see where this is going? No? Let's look again at how I am running slapd, shall we?

/usr/sbin/slapd -d 256 -h "ldap:/// ldapi:/// ldaps:///" -g openldap -u openldap -F /etc/ldap/slapd.d

As you can see, I did not pass a KRB5_KTNAME to slapd. As soon as I fed that to slapd, all was once again well in the Land of Ooo.

Restoring time on sleeping (linux) vms

One of the most annoying issues in a VM is keeping accurate time.When a vm client is saved/paused and then restored, its clock will be still set to whatever time was when it was paused. That can lead to many issues including not being able to login using Kerberos. Rebooting the machine will force the clock to be reset, but what if we do not want to (or cannot) reboot? What we could use a script that will monitor the drift and if it is too far -- say one minute off -- it will automagically adjust the clock in the vm client.
There are many virtualization packages out there, some of which (in no order) are VMWare ESX/ESXi, Virtual Box, kvm, and Microsoft Virtual Server. The first three I personally have used; if you want to see a more complete list I would suggest to try the Wikipedia entry on Hypervisors. Suffice to say some of those programs allow host machine (the physical machine running the program) expose its clock to the guest machine (or vm), others do not. So let's see how we can check and set the clock based on whether the guest can see the host clock or not. For this discussion I will stick to Linux because I feel lazy today.

Using the vm host clock

Let's say the vm program of your choice can pass the vm host's clock to the client (I do know that KVM can do that and believe VMWare ESXi can too), you could write something like this:

cat > /usr/local/bin/driftcheck << 'EOF'
#!/bin/sh
### Detect drift in the vm client vh most. If it is massive; adjust
### it.

HWCLOCK=/sbin/hwclock

# Max drift in seconds. Kerberos does not like time offsets > 5min, 
# so we set it to 1m = 1*60s
MAX_DRIFT=` echo "1*60" | bc -l`

VMHOST_TIME=`date +'%s' -d "$(${HWCLOCK} -r -u | cut -d' ' -f-7)"`
MY_TIME=`date +%s`
DRIFT=$( echo "${VMHOST_TIME} - ${MY_TIME}" | bc | tr -d -)

if [ "${DRIFT}" -gt "${MAX_DRIFT}" ]
then   
        ${HWCLOCK} -s -u
fi
EOF
chmod +x /usr/local/bin/driftcheck

What it does is get the hardware clock of this vm client, which actually is the vm host's clock. One thing this requires is the vm client is configured to take the vm host's clock in utc time. YMMV here, but if you are using kvm and libvit, you would have a line like this

<clock offset="utc"></clock>

somewhere in the xml file defining the client.
The commented lines inside the if statement are there just so when you test it out you can see what is going on. For production you probably want leave them commented out
Now, we would probably want to to have it being called often but not really crazy. So, for now let's say we create a cron job to run driftcheck every 5 minutes?

cat > /etc/cron.d/driftcheck << 'EOF'
*/5 * * * * root /usr/local/bin/driftcheck > /dev/null 2>&1
EOF

Of course, you should adjust it to fit your needs. Do note I put driftcheck in /usr/local/bin/; it just felt like a nice place this season.

Using NTP

As we mentioned above, sometimes we cannot (or will not) use the clock off the vm host. If we are using a ntp server, we are good. Ok, you might argue but, if the drift/skew is too long, ntpd will refuse to adjust the clock. And, even if we force it, it will take hours or even days!. Not if we nudge it a bit by using an old friend, ntpdate:

cat > /usr/local/bin/driftcheck << 'EOF'
#!/bin/sh
### Detect drift against a reliable time source. If it is massive;
### instead of relying on ntp (which will not work), do something
### a bit more drastic

NTPQ=/usr/bin/ntpq
NTPDATE=/usr/sbin/ntpdate

# Max drift in ms. Kerberos does not like time offsets > 5min, so
# we set it to 1min = 1*60*1000ms
MAX_DRIFT=` echo "60*1000" | bc -l`

# Find the ntp server we are using in this host
NTP_SERVER=`sed -ne '/^server/p' /etc/ntp.conf | awk '{ print $2 }'| head -1`
# Get current drift
DRIFT=`${NTPQ} -p ${NTP_SERVER}|grep '*'|awk '{print $9 }'`
[ -z $DRIFT ] && DRIFT=`${NTPQ} -p |tail -n +3| awk '{print $9 }'`
DRIFT=$( echo "${DRIFT}/1" | bc | tr -d -)

if [ "${DRIFT}" -gt "${MAX_DRIFT}" ]
then
        echo ${NTPDATE} -u ${NTP_SERVER} >> /tmp/ntp
        ${NTPDATE} -u ${NTP_SERVER} >> /tmp/ntp
fi
EOF
chmod +x /usr/local/bin/driftcheck

The reason we want to look for * is that according to the
NTP troubleshooting page, it is the source you are currently synchronized to.
The cron file will be the same as what we had in the hardware clock session, so I decided not to copy it.
And that is pretty much it. Remember they are just ideas. Those scripts should work as is but you can/should customize/adapt them. If you have any questions, concerns, or just want to confuse me, do leave a message.

References

• http://time.to.pullthepl.ug/blog/2012/8/29/easy-ntp-stats-without-ntpstat/
• https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.6_Technical_Notes/Known_Issues-kvm.html
• http://linux.derkeiler.com/Mailing-Lists/Fedora/2012-04/msg00204.html

Poor Man's LDAP Replication checker (Kerberos Involved!)

Warning: this article is a bit short, perhaps even curt. As such it might brush over too many concepts and assume you know how to setup replication in ldap. You have been warned!

If you are using ldap, you probably want to have more than one ldap server. You know, one dies and the other takes over kinda thing. Now, that involves some means to keep the ldap databases in the different ldap servers in sync. Currently the preferred method is called syncrepl, and you can find info on setting it up online (I might add some thoughts on that later). Problem is you need to make sure they are all in sync.

One way to do so is by monitoring the contextCSN. So if you have two ldap servers, master.domain.com and slave.monetra.com (let's say, as the name implies, we have a master and a slave ldap servers), after you start replication you could go to master and do:

ldapsearch -z1 -LLLQY EXTERNAL -H ldapi:/// -s base contextCSN
dn: dc=domain,dc=com
contextCSN: 20120113185836.364944Z#000000#000#000000

Then you would go to the slave and run the same command. And then compare the output, namely the funny number after contextCSN:, between the two. If they match, all should be well. If not, time to go check the log files in the two machines; depending on how you set it up, that would mean starting with auth.log and syslog.

Now we know what we need, what if we could make a little script to compare these values between all the ldap servers you have (that are replicating)? Well, we should be able to connect to those ldap servers from any machine that can do so, and query for contextCSN. And then, it is a matter of comparing them. In the following script, let's assume we have 3 ldap servers: one master and two slaves.

#!/bin/bash

# KRB5CCNAME=/tmp/host.tkt
LDAPs=( master.domain.com slave1.domain.com slave2.domain.com )
LDAP_NUMBER=${#LDAPs[@] }
ldap_reply[0]="";

function getLDAPinfo()
{
   local i
   for ((i=0; i < ${LDAP_NUMBER}; i++))
      do
         ldap_reply[$i]=`ldapsearch -z1 -LLLQ -H ldap://${LDAPs[$i]} -s base contextCSN | grep contextCSN | awk '{ print $2 }'`
         echo ${ldap_reply[$i]}
   done
}

function checkSyncStatus()
{
   local i
   local j
   for ((i=0; i < ${LDAP_NUMBER} -1; i++))
      do
         echo -n "$i x "
         for ((j=$i + 1; j < ${LDAP_NUMBER} ; j++))
            do
               echo -n " $j"
               if [[ "${ldap_reply[$i]}" != "${ldap_reply[$j]}" ]]; 
                  echo -n "(Bad)"
               else
                  echo -n "(Ok) "
               fi
         done
         echo
   done
}

echo "Number of LDAP servers: ${LDAP_NUMBER}"
getLDAPinfo
checkSyncStatus

Note the arguments for ldapsearch might change a bit (you might need a -x -Z or whatever; you know what you need to do to run ldapsearch in your environment). If the three machines are in sync, when you run the above code, the output should look something like:

Number of LDAP servers: 3
20120113185836.364944Z#000000#000#000000
20120113185836.364944Z#000000#000#000000
20120113185836.364944Z#000000#000#000000
0 x  1(Ok) 2(Ok)
1 x  2(Ok)

Now, the code is not complete, and that is for a reason. I really wanted to show what it is doing. The echo "Number of LDAPs: ${LDAP_NUMBER}" is there just to verify the number of ldap servers: we have 3 and in the LDAPs array they would be [0], [1], and [2]. It should be commented out/not be there in the production version of the script. The echo statement in getLDAPinfo() is there just to show the output of (and how to get said output) the ldapsearch command so you can see they are all matching; you can also take it off. And the same goes for the echo statements in getLDAPinfo. What they allow is to show which contextCSN values we are comparing, and whether they match ((Ok)) or not. Note we are checking not only the master with each slave but each slave against the others. It is a bit overkill but why not?

As I said, this code is incomplete; what you would need to do, after removing/commenting out the echo statements, is to decide how to use this information. What I have done is when ${ldap_reply[$i]} != ${ldap_reply[$j]}, it then writes down a message saying the contextCSN values for LDAPs[$i] and LDAPs[$j] do not match in an email that is then sent to me. Maybe you want to do something else, but you get the idea.

The only missing step now is to create a cron job to call this script every so often.

Ok, smart guy, you might say, what about the kerberos part you mentioned on the title? Well, if you go back to the script, you will notice a line containing KRB5CCNAME=/tmp/host.tkt commented out. We authenticate ldap access against kerberos. Also, since each machine has its own kerberos principal and keytab, we use it to create a kerberos cache named /tmp/host.tkt:

FQDN=$(hostname -f)
sudo kinit -k -t /etc/krb5.keytab -c /tmp/host.tkt "host/$FQDN@DOMAIN.COM"

which is owned by the root users and used by different services in each ldap/kerberos client. Well, if it is there, we might as well use it, right?