# OpenLDAP causing cron to core dump



## Ted (Feb 5, 2016)

I recently installed an OpenLDAP server and was setting up the client nodes when we realized that it was causing cron to core dump and stop running.


```
Jan 29 08:27:00 <server-name> kernel: pid 61506 (cron), uid 0: exited on signal 11 (core dumped)
```

I followed this walkthrough https://www.freebsd.org/doc/en/articles/ldap-auth/index.html to setup the server and client nodes, the server and clients are running the latest FreeBSD 10.2.

Has anyone ever heard of this before or what might be causing it? Or even point me in a direction as to where to look for the problem, any guidance is appreciated.


----------



## Juha Nurmela (Feb 5, 2016)

Knows nothing about LDAP, just some generic troubleshooting:

Are there multiple pam libraries ? `find / -name "libpam.so*"`
Does

```
# gdb /us/sbin/cron cron.core
where
```
reveal anything ?
Anything alarming in

```
# killall cron
# cron -x bit,ext,load,misc,pars,proc,sch
```


```
# ktrace -d cron # descendants
...wait...
# kdump
```

Juha


----------



## Ted (Feb 12, 2016)

Juha,

When we ran the find command we found two libpam files:


```
/usr/obj/usr/src/tmp/usr/lib/libpam.so.5
/usr/obj/usr/src/tmp/usr/lib/libpam.so
/usr/obj/usr/src/lib/libpam/libpam/libpam.so
/usr/obj/usr/src/lib/libpam/libpam/libpam.so.5
/usr/obj/usr/src/lib32/usr/lib32/libpam.so.5
/usr/obj/usr/src/lib32/usr/lib32/libpam.so
/usr/obj/lib32/usr/src/lib/libpam/libpam/libpam.so
/usr/obj/lib32/usr/src/lib/libpam/libpam/libpam.so.5
/usr/lib/libpam.so
/usr/lib/libpam.so.5
/usr/lib32/libpam.so.5
/usr/lib32/libpam.so
```

When we ran 
	
	



```
cron -x bit,ext,load,misc,pars,proc,sch
```
 some alarming output was:


```
load_entry()...about to parse username
load_entry()...got root
Segmentation fault (core dumped)
```

The kdump file is attached as a text file.

Is there anything that stands out to you that might cause cron to core dump?

Thanks for any help that you might be able to provide.


----------



## Juha Nurmela (Feb 12, 2016)

Ugh, sorry. I goofed. It's `ktrace -i cron`, not -d.

Juha


```
$ ldd /usr/sbin/cron
/usr/sbin/cron:
  libpam.so.5 => /usr/lib/libpam.so.5 (0x28070000)
  libutil.so.9 => /lib/libutil.so.9 (0x2807b000)
  libc.so.7 => /lib/libc.so.7 (0x2808d000)
$ ldd /usr/lib/pam_unix.so
/usr/lib/pam_unix.so:
  libpam.so.5 => /usr/lib/libpam.so.5 (0x281f4000)
  libutil.so.9 => /lib/libutil.so.9 (0x281ff000)
  libcrypt.so.5 => /lib/libcrypt.so.5 (0x28211000)
  libypclnt.so.4 => /usr/lib/libypclnt.so.4 (0x28235000)
  libc.so.7 => /lib/libc.so.7 (0x2806c000)
```


----------



## Ted (Feb 12, 2016)

Thanks for the responses, we are looking over the kdump right now, but I wanted to let you know when I ran the ldd commands, we got the same results.

Will let you know what we find.

Thanks.


----------



## Juha Nurmela (Feb 12, 2016)

Try ldd with the pam_*ldap*.so files too.
cron was close to printing root's user id. It tanked in login_getclass() or getpwnam().

Juha


----------



## Ted (Feb 12, 2016)

So here is the kdump that we have from `ktrace -i cron`, do you see anything that stands out?

Thanks again.


----------



## Juha Nurmela (Feb 12, 2016)

So it fails in getpwnam(), it seems. It can't find nss_files.so for "passwd: files ldap" name service, but the builtin might be enough.

*scratch* Doesn't anything else coredump?
Juha


```
...
"/usr/lib/libpam.so.5"
"/lib/libutil.so.9"
"/lib/libc.so.7"

fork setsid
write pidfile
...
read /etc/crontab
read login.conf.db did login_getclass() ?
read nsswitch.conf  does getpwnam() ?

"/lib/nss_files.so.1"
"/usr/lib/nss_files.so.1"
"/usr/lib/compat/nss_files.so.1"
"/usr/local/lib/nss_files.so.1"
"/lib/nss_files.so.1"
"/usr/lib/nss_files.so.1" all missing

"/usr/local/lib/nss_ldap.so.1" these were found and mapped
"/usr/local/lib/libldap-2.4.so.2"
"/usr/local/lib/liblber-2.4.so.2"
"/usr/lib/libkrb5.so.11"
"/usr/lib/libcom_err.so.5"
"/usr/lib/libgssapi_krb5.so.10"
"/usr/local/lib/libssl.so.8"
"/usr/local/lib/libcrypto.so.8"
"/usr/lib/libasn1.so.11"
"/lib/libcrypt.so.5"
"/lib/libcrypto.so.7"
"/usr/lib/libhx509.so.11"
"/usr/lib/libroken.so.11"
"/usr/lib/libwind.so.11"
"/usr/lib/libheimbase.so.11"
"/usr/lib/private/libheimipcc.so.11"
"/usr/lib/libgssapi.so.10"
"/lib/libthr.so.3"

coredump
```


----------



## Ted (Feb 12, 2016)

So far the only service to coredump was cron(8).

The /etc/nsswitch.conf file was:


```
group: compat
passwd: compat
```

In order to have LDAP work we changed it to:


```
group: files ldap
passwd: files ldap
```

As soon as I remove ldap, cron(8) works fine, but then in the process breaks LDAP.


----------



## Juha Nurmela (Feb 12, 2016)

Between the library loads and the SIGSEGV there is a short interlude, it queries kern.usrstack and RLIMIT_STACK, then mprotects (successfully) 0.5GB of stack read, write, execute. Maybe this is as it should be, but being just before the crash it feels bad.

hands raised,
Juha


----------



## Juha Nurmela (Feb 12, 2016)

Ah... The curious interlude is just libthr initialization. __pthread_map_stacks_exec().

Juha


----------



## Ted (Feb 15, 2016)

Juha,

So I am attaching two dump files, one from our server that is running LDAP and cron is crashing (ldap) and the other (noldap) that is not running ldap and cron is not crashing. I have done a diff on these two files but nothing is glaring at me as to point to the issue other than nsswitch.conf file, turns LDAP on and off so to speak.

Any ideas?

Thanks.


----------



## Juha Nurmela (Feb 15, 2016)

There not much in the noldap trace, it does not reach the problem point. "Already running".

Debugging (with or without the cron.core) is not for humans, there are no symbols. If you have base system sources, try

```
# cd /usr/src/usr.sbin/cron/
# make
# gdb ./cron/cron
(gdb) run -x bit,ext,load,misc,pars,proc,sch
...
(gdb) where
```

Juha


----------



## Oleg Smirnov (Aug 3, 2016)

Hi!

Your trouble in  conflict of libcrypto libraries:

"/usr/local/lib/libcrypto.so.8"
"/lib/libcrypto.so.7"

Your cron used libcrypto.7, and your nss_ldap and pam_ldap (and openldap-client) use libcrypto.8 - it's trouble.

You need to rebuild openldap, nss and pam with openssl version from base system:

SSL_DEFAULT=base

Do not forget about sources of your system (/usr/src) before build  It's need for build with "base" openssl!


----------



## Ted (Aug 3, 2016)

Hi Oleg,

What it turned out to be was I needed to install the package nss_pam_ldapd and enable it on start up and that corrected the issue.

Thanks for your input though, when I have a spare client I will try your method.


----------

