Using ldapsearch and ldapmodify to talk to Active Directory

Why?

Great question! Here are a few lame excuses I was able to come with:

• I like to use command line. This is a lame excuse because Windows have powershell. But,
  
• I am more comfortable with Linux than Windows. Lame excuse since
  1. How many posts in this very blog I have made about using Windows?
     
  2. How many of said posts I have used the GUI when I could take care of business with Powershell?
     
  3. What is wrong with Powershell, at least of the applications I have used it for here so far?

I do have a couple of not so lame ones though:

• I like to be able to access the network resources from any machine in the network running any OS. If I have a Linux box in an Active Directory-controlled network, chances are I will need to authenticate the Linux box against Active Directory (AD so I can save some keytaps). AD is Kerberos + ldap + sprinkles, so I better be able to use the usual kerberos/ldap Linux tools as one day I will need to figure out why things are boink.
  
• It feels like I get more info using ldapsearch than the Windows tools, which is good when I do not know the name of an attribute, or how many instaces of said attribute are in use.

Using ldapsearch

Before we go mindlessly typing things, we need some data.

We need to know the name of the ldapserver.

Yes, if you have it configured in your ldap.conf file, you should not need it. But I prefer to assume nothing. If the domain was setup properly, we can ask it directly by typing nslookup -type=srv _ldap._tcp.DOMAIN where DOMAIN is the Active Directory domain name, not the DNS one; that caught me off guard. So, if our DNS dmain is example.com and the AD domain (we are very original) is ad.example.com, we have

raub@desktop:/tmp$ nslookup -type=srv _ldap._tcp.ad.example.com
Server:         192.168.0.10
Address:        192.168.0.10#53

Non-authoritative answer:
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC0.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC2.ad.example.com.
_ldap._tcp.ad.example.com   service = 0 100 389 ADDC1.ad.example.com.

Authoritative answers can be found from:
ad.example.com      nameserver = addc1.ad.example.com.
ad.example.com      nameserver = ns.example.com.
ad.example.com      nameserver = ns2.example.com.
ad.example.com      nameserver = addc0.ad.example.com.
ad.example.com      nameserver = addc2.ad.example.com.
ADDC0.ad.example.com        internet address = 192.168.1.100
ADDC1.ad.example.com        internet address = 192.168.1.102
ADDC2.ad.example.com        internet address = 192.168.1.101
ns.example.com      internet address = 192.168.0.10
ns2.example.com     internet address = 192.168.0.10

raub@desktop:/tmp$

and we can use any of the ADDCN.ad.example.com (where N=0,1,2). Notice in my setup, just using ad.example.com also worked. I found out by using netcat to see if port 636 was open (I will leave the answer for "why port 636?" as an exercise to the reader)

raub@desktop:~$ nc -v ad.example.com 636
Connection to ad.example.com 636 port [tcp/ldaps] succeeded!
^C
raub@desktop:~$ 

We need to be able to authenticate against AD somehow.

For this discussion I will be using a username and password; we can also do it using a Kerberos TGT ticket.

Fun Fact: I have a user account, my normal one, which can look into some things in LDAP/AD but then I have another ("admin") account I can see more and edit stuff in AD. You will see later on me forgetting completely about that and how it affects me. But we are getting ahead of ourselves.

With that taken care of, we are going to begin by looking for some user: me

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" 
Enter LDAP Password:
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
objectClass: top
objectClass: posixAccount
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Wrong Droid
sn: Droid
title: Entropy Creators
description: Orthodontics
givenName: Wrong
initials: B 
distinguishedName: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
instanceType: 4
whenCreated: 20080109142820.0Z
whenChanged: 20180905201157.0Z
displayName: Droid, Wrong
uSNCreated: 243336
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
[...]
proxyAddresses: sip:raub@ad.example.com
proxyAddresses: smtp:raub@ad.example.com
proxyAddresses: X500:/o=UNC Exchange/ou=Exchange Administrative Group (FYDIBOH
 F23SPDLT)/cn=Recipients/cn=raub
proxyAddresses: SMTP:raub@email.example.com
displayNamePrintable: Wrong Droid
name: Wrong Droid
[...]
sExchPoliciesExcluded: {26491cfc-9e50-4857-861b-0cb8df22b5d7}
msExchUserAccountControl: 0
msExchELCMailboxFlags: 2
msRTCSIP-PrimaryHomeServer:
[...]
msExchOWAPolicy: CN=Default,CN=OWA Mailbox Policies,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=ad,DC=example,DC=com

# refldaps://ForestDnsZones.ad.example.com/DC=ForestDnsZones,DC=ad,DC=example,DC=com

# refldaps://DomainDnsZones.ad.example.com/DC=DomainDnsZones,DC=ad,DC=example,DC=com

# refldaps://ad.example.com/CN=Configuration,DC=ad,DC=example,DC=com

raub@desktop:~$

That is probably more info than you wanted to know about someone, but this has its applications. Since we now know every attribute associated with a given user (I should not be that special, at least as far as AD is concerned), we can build customized queries looking for only a specific bit of info. For instance, let's just get the groups I belong to or am a member of (hint hint):

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub)" memberOf 
dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com

Fancy, huh? Now, if instead of doing (CN=raub) we did (CN=*raub*), it would return every entry that has a CN with raub in it. In my case that would mean two entries, raub and raub.admin (if you remember our Fun Fact you will know about it), but it could also have returned a device whose name matches that. And, they would have been printed one after the other (I do wonder if the order depends on the order they were added to LDAP/AD).

Using ldapmodify

Ok, we established we can probulate Active Directory using common household Linux/UNIX query tools. What if we want to change something? Let's say we have an AD group (specifically a distribution list) called moustache_operators (for those who own and operate moustaches) and want to add a member and delete another.

Why I do like ldapmodify to edit LDAP/AD

Main reason is because I can create a file (in the LDIF format) at my leisure (i.e. think about what I want to do) with pretty commands and comments describing what I want to do. If I like what I did, I can then document it and maybe even save the file in a wiki or somewhere that can be fed to Ansible/Puppet/Chef/Docker and reused.

Our little LDIF file, let's call it change.ldif, could look like this:

# Let's define the entity we will be fiddling with
dn: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com
# And then what we will be doing with it
changetype: modify
delete: member
member: CN=baldone,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
# Separator because we will be doing another change
-
add: member
member: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com

So let's try it:

raub@desktop:~$ ldapmodify -H "ldaps://addc0.ad.example.com:636" -D "raub@ad.example.com" -x -W -f change.ldif
Enter LDAP Password:
modifying entry "CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com"
ldap_modify: Insufficient access (50)
        additional info: 00002098: SecErr: DSID-03150F93, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

raub@desktop:~$ 

Why is it not working? Well, do you remember the Fun Fact I mentioned earlier in this article? This is how it shows it's ugly head. I should have used my raub.admin@ad.example.com account instead of raub@ad.example.com. If we do it right, it then works. I will not show the output of a successful connection here; what matters is verifying the deed is done, and we can do it using dear ol' ldapsearch:

raub@desktop:~$ ldapsearch -H "ldaps://addc0.ad.example.com:636" 
-D "raub@ad.example.com" -W -b "dc=ad,dc=example,dc=com" -LLL -s sub "(CN=raub) memberOf" dn: CN=raub,OU=Users,OU=Identity,DC=ad,DC=example,DC=com
memberOf: CN=cookie_recipes,OU=Distribution Groups,OU=Special Users,DC=ad,DC=example,DC=com
memberOf: CN=servers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=third_floor_printers,OU=Groups,OU=APE,OU=EXAMPLE,DC=ad,DC=example,DC=com
memberOf: CN=moustache_operators,OU=Distribution Lists,OU=Special Users,DC=ad,DC=example,DC=com

What about Mac/OSX?

They too have ldapsearch; just use the terminal and off you go.

generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()

This will be a quick post about something that was biting my ass these last few days and what was the real cause. After you read it, you are welcome to laugh at my expense. Go ahead! I deserve it!

I was working in a kerberos/ldap (linux) server and needed to debug the connection to a given client. The ldap connection uses TLS, GnuTLS specifically since the two machines were ubuntu servers, which means we also had to worry about certs. And since kerberos is in the picture, we need to configure for that. To help in solving other issues, which I should comment about later (at least those were clever problems not like this one), I was running slapd in debug mode,

/usr/sbin/slapd -d 256 -h "ldap:/// ldapi:/// ldaps:///" -g openldap -u openldap -F /etc/ldap/slapd.d

and that did help solve the other issue I had. Some of you will notice I am also running ldaps (port 636), which I really do not need since TLS should take care of the encryption thingie. But, I digress for this post, so let's go back on topic. What I then noticed was some very problems with ldap. For instance, if I created a kerberos ticket and then tried to run ldapsearch, I would then get the following error:

root@services:~# export KRB5CCNAME=/tmp/host.tkt
root@services:~# ldapsearch -vvv
ldap_initialize(  )
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Other (e.g., implementation specific) error (80)
        additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
root@services:~#

Here is what the server sees:

53261bde conn=1043 fd=19 ACCEPT from IP=192.168.1.181:44610 (IP=0.0.0.0:389)
53261bde conn=1043 op=0 EXT oid=1.3.6.1.4.1.1466.20037
53261bde conn=1043 op=0 STARTTLS
53261bde conn=1043 op=0 RESULT oid= err=0 text=
53261bde conn=1043 fd=19 TLS established tls_ssf=128 ssf=128
53261bde conn=1043 op=1 BIND dn="" method=163
53261bde SASL [conn=1043] Failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
53261bde conn=1043 op=1 RESULT tag=97 err=80 text=SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
53261bde conn=1043 op=2 UNBIND
53261bde conn=1043 fd=19 closed

Since I do not have many clever things to talk about and fill the space until the solution, how about if we talk about what some of those lines mean?

• IP=192.168.1.181:44610 (IP=0.0.0.0:389): Client 192.168.1.181 is connecting from its port 44610 to my port 389.
  
• oid=1.3.6.1.4.1.1466.20037: Start TLS extended request (per rfc2830).
  
• BIND dn="": anonymous if we are doing a SIMPLE bind. If we are however doing SASL bind, it is not used.
  
• tag=97: result from a client bind operation.

As you noticed, at least from reading the title of this post, the error line is this generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () thingie. Here is where it annoyed me to no end: what minor code? It is supposed to put some kind of message between the parenthesis, like "No principal in keytab matches desired name" or "Ticket expired". Then I would be able to search online for something. Instead, zilch. I could not find a single entry where the minor code parenthesis thingie was empty. Not very helpful today are we?

Solution

So, what was wrong? Me. User error. Do you remember how I was running slapd? Do you also remember the part about kerberos? Well, in the /etc/default/slapd (that'll be /etc/sysconfig/ldap for you RedHat/CentOS/Fedora folks) I have defined

export KRB5_KTNAME=/etc/ldap/ldap.keytab

which means ldap knows then where the keytab containing the ldap service principal hides. Can you see where this is going? No? Let's look again at how I am running slapd, shall we?

/usr/sbin/slapd -d 256 -h "ldap:/// ldapi:/// ldaps:///" -g openldap -u openldap -F /etc/ldap/slapd.d

As you can see, I did not pass a KRB5_KTNAME to slapd. As soon as I fed that to slapd, all was once again well in the Land of Ooo.

Poor Man's LDAP Replication checker (Kerberos Involved!)

Warning: this article is a bit short, perhaps even curt. As such it might brush over too many concepts and assume you know how to setup replication in ldap. You have been warned!

If you are using ldap, you probably want to have more than one ldap server. You know, one dies and the other takes over kinda thing. Now, that involves some means to keep the ldap databases in the different ldap servers in sync. Currently the preferred method is called syncrepl, and you can find info on setting it up online (I might add some thoughts on that later). Problem is you need to make sure they are all in sync.

One way to do so is by monitoring the contextCSN. So if you have two ldap servers, master.domain.com and slave.monetra.com (let's say, as the name implies, we have a master and a slave ldap servers), after you start replication you could go to master and do:

ldapsearch -z1 -LLLQY EXTERNAL -H ldapi:/// -s base contextCSN
dn: dc=domain,dc=com
contextCSN: 20120113185836.364944Z#000000#000#000000

Then you would go to the slave and run the same command. And then compare the output, namely the funny number after contextCSN:, between the two. If they match, all should be well. If not, time to go check the log files in the two machines; depending on how you set it up, that would mean starting with auth.log and syslog.

Now we know what we need, what if we could make a little script to compare these values between all the ldap servers you have (that are replicating)? Well, we should be able to connect to those ldap servers from any machine that can do so, and query for contextCSN. And then, it is a matter of comparing them. In the following script, let's assume we have 3 ldap servers: one master and two slaves.

#!/bin/bash

# KRB5CCNAME=/tmp/host.tkt
LDAPs=( master.domain.com slave1.domain.com slave2.domain.com )
LDAP_NUMBER=${#LDAPs[@] }
ldap_reply[0]="";

function getLDAPinfo()
{
   local i
   for ((i=0; i < ${LDAP_NUMBER}; i++))
      do
         ldap_reply[$i]=`ldapsearch -z1 -LLLQ -H ldap://${LDAPs[$i]} -s base contextCSN | grep contextCSN | awk '{ print $2 }'`
         echo ${ldap_reply[$i]}
   done
}

function checkSyncStatus()
{
   local i
   local j
   for ((i=0; i < ${LDAP_NUMBER} -1; i++))
      do
         echo -n "$i x "
         for ((j=$i + 1; j < ${LDAP_NUMBER} ; j++))
            do
               echo -n " $j"
               if [[ "${ldap_reply[$i]}" != "${ldap_reply[$j]}" ]]; 
                  echo -n "(Bad)"
               else
                  echo -n "(Ok) "
               fi
         done
         echo
   done
}

echo "Number of LDAP servers: ${LDAP_NUMBER}"
getLDAPinfo
checkSyncStatus

Note the arguments for ldapsearch might change a bit (you might need a -x -Z or whatever; you know what you need to do to run ldapsearch in your environment). If the three machines are in sync, when you run the above code, the output should look something like:

Number of LDAP servers: 3
20120113185836.364944Z#000000#000#000000
20120113185836.364944Z#000000#000#000000
20120113185836.364944Z#000000#000#000000
0 x  1(Ok) 2(Ok)
1 x  2(Ok)

Now, the code is not complete, and that is for a reason. I really wanted to show what it is doing. The echo "Number of LDAPs: ${LDAP_NUMBER}" is there just to verify the number of ldap servers: we have 3 and in the LDAPs array they would be [0], [1], and [2]. It should be commented out/not be there in the production version of the script. The echo statement in getLDAPinfo() is there just to show the output of (and how to get said output) the ldapsearch command so you can see they are all matching; you can also take it off. And the same goes for the echo statements in getLDAPinfo. What they allow is to show which contextCSN values we are comparing, and whether they match ((Ok)) or not. Note we are checking not only the master with each slave but each slave against the others. It is a bit overkill but why not?

As I said, this code is incomplete; what you would need to do, after removing/commenting out the echo statements, is to decide how to use this information. What I have done is when ${ldap_reply[$i]} != ${ldap_reply[$j]}, it then writes down a message saying the contextCSN values for LDAPs[$i] and LDAPs[$j] do not match in an email that is then sent to me. Maybe you want to do something else, but you get the idea.

The only missing step now is to create a cron job to call this script every so often.

Ok, smart guy, you might say, what about the kerberos part you mentioned on the title? Well, if you go back to the script, you will notice a line containing KRB5CCNAME=/tmp/host.tkt commented out. We authenticate ldap access against kerberos. Also, since each machine has its own kerberos principal and keytab, we use it to create a kerberos cache named /tmp/host.tkt:

FQDN=$(hostname -f)
sudo kinit -k -t /etc/krb5.keytab -c /tmp/host.tkt "host/$FQDN@DOMAIN.COM"

which is owned by the root users and used by different services in each ldap/kerberos client. Well, if it is there, we might as well use it, right?

getting getent passwd of members of other ldap groups

Usually when you deploy ldap you also want to make sure when you do getent passwd it will not only show local users but also the ldap users. This is usually done in nslcd.conf if you are using nss-pam-lapd and might look like something like this (shortened a bit for the sake of brevity):

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

uri ldap://ldap-thingie.domain.com
base dc=domain,dc=com
# [...]
# Customize certain database lookups
base   passwd   ou=people,dc=domain,dc=com
base   group    ou=groups,dc=domain,dc=com
# [...]
scope  passwd   one
scope  group    one
scope  netgroup one
scope  networks one

Now, let's say for whatever reason you want to create another group, say, vegetables, whose members are not people (we will discuss the metaphysical implications of that some other time):

dn: cn=vegetables,ou=groups,dc=domain,dc=com
objectClass: posixGroup
cn: vegetables
gidNumber: 2424

Since vegetables belong to ou=groups, you will see it if otu do, say, getent group vegetables. So, let's add a member to that group, say swampthing:

dn: uid=swampthing,ou=vegetables,dc=domain,dc=com
uid: swampthing
cn: Swamp Thing
givenName: Swamp 
sn: Thing
objectClass: inetOrgPerson
objectClass: posixAccount
loginShell: /usr/bin/treebark
uidNumber: 1995
gidNumber: 2424
homeDirectory: /home/swamp
gecos: Swamp Thing
mail: swampthing@domain.com

when you try to look for it using getent passwd swampthing, it will not show up. But, doing a quick ldapsearch -x "(objectClass=posixAccount)" will find our green fellow. What is going on here? Well, look back at our nslcd.conf on the top of this article. base passwd ou=people,dc=domain,dc=com really translates to "hey man! If you are looking for a user (someone under passwd) in ldap, check ou=people,dc=domain,dc=com!" Problem is Mr. Thing is not on ou=people! Then, we should tell nslcd that looking for a user in ou=vegetables is ok too:

base   passwd   ou=people,dc=domain,dc=com
base   passwd   ou=vegetables,dc=domain,dc=com
base   group    ou=groups,dc=domain,dc=com

And now, when we try getent passwd swampthing again or even id swampthing, we will get info on Mr. Swamp.

Of course, I picked a rather silly name and group for this example, but there is nothing stopping the group to be, say, ou=services and the user mysql-backup. Does that give you evil ideas?

.