# crontab not working after upgrading to FreeBSD 12.0-RELEASE



## eydaimon (Dec 12, 2018)

Just upgraded to FreeBSD 12.0-RELEASE yesterday and my crontab is no longer running.

Trying with 
	
	



```
*/1 * * * *  echo foo > ~/moop.txt
```
 this morning but I'm not getting anywhere.  No email, no output from the echo.

Any insights? The only thing I found in the log is this, and I don't know if it's the cause or how to address



```
root@luffy /u/h/d/.c/restic# cd /var/log
root@luffy /v/log# tail cron
Dec 12 09:16:26 luffy 1 2018-12-12T09:16:26.878263-08:00 luffy crontab 30750 - - (daniel) BEGIN EDIT (daniel)
Dec 12 09:16:42 luffy 1 2018-12-12T09:16:42.745588-08:00 luffy crontab 30750 - - (daniel) REPLACE (daniel)
Dec 12 09:16:43 luffy 1 2018-12-12T09:16:43.751293-08:00 luffy crontab 30750 - - (daniel) END EDIT (daniel)
Dec 12 09:17:00 luffy /usr/sbin/cron[1174]: (daniel) RELOAD (tabs/daniel)
Dec 12 09:17:00 luffy /usr/sbin/cron[30831]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat"
Dec 12 09:17:00 luffy /usr/sbin/cron[30831]: in openpam_load_module(): no pam_nologin.so found
Dec 12 09:17:00 luffy /usr/sbin/cron[30831]: (CRON) error (can't start PAM)
Dec 12 09:18:00 luffy /usr/sbin/cron[30864]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat"
Dec 12 09:18:00 luffy /usr/sbin/cron[30864]: in openpam_load_module(): no pam_nologin.so found
Dec 12 09:18:00 luffy /usr/sbin/cron[30864]: (CRON) error (can't start PAM)
```


----------



## SirDice (Dec 12, 2018)

eydaimon said:


> Just upgraded to FreeBSD 12.0-RELEASE yesterday


How did you upgrade? The errors look like it was only partially done.


----------



## eydaimon (Dec 12, 2018)

I followed the handbook:  https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/updating-upgrading-freebsdupdate.html

1. freebsd-update -r 12.0-RELEASE upgrade
2. freebsd-update install
3. shutdown -r now
4. freebsd-update install


----------



## eydaimon (Dec 12, 2018)

I've tried running freebsd-update install again, but then I get an error: [FONT=Courier New]Cannot identify running kernel[/FONT]


----------



## DutchDaemon (Dec 12, 2018)

Yeah, interesting. have the exact same thing on an otherwise stock system.


```
Dec 12 20:35:00 poudriere /usr/sbin/cron[72769]: (root) CMD (/usr/libexec/atrun)
Dec 12 20:35:00 poudriere /usr/sbin/cron[72768]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:35:00 poudriere /usr/sbin/cron[72768]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:35:00 poudriere /usr/sbin/cron[72768]: (CRON) error (can't start PAM)
```


```
# ls -al /usr/lib/pam_nologin.so*
lrwxr-xr-x  1 root  wheel     16 Dec 12 18:21 /usr/lib/pam_nologin.so -> pam_nologin.so.6
-r--r--r--  1 root  wheel  15016 Dec 12 18:20 /usr/lib/pam_nologin.so.6
```

Emphasis mine:


```
# strings /usr/lib/pam_nologin.so.6
no_warn
Got user: %s
nologin
Unable to get login class for user %s
Opened %s file
pam_sm_acct_mgmt
ignorenologin
Administrator refusing you: %s
Q`^B
libutil.so.9
libpam.so.6
libc.so.7
pam_nologin.so.6
_fini
_init
_Jv_RegisterClasses
__cxa_finalize
FBSD_1.0
__stack_chk_fail
__stack_chk_guard
_openpam_log
close
free
>>>>> fstat
>>>>> FBSD_1.5
getpwnam
login_close
login_getcapbool
login_getcapstr
login_getpwclass
malloc
open
openpam_get_option
pam_error
pam_get_user
pam_sm_acct_mgmt
read
uJUH
AWAVAUATSH
[A\A]A^A_]
/var/run/nologin
```


```
# freebsd-version -ku
12.0-RELEASE
12.0-RELEASE
# uname -a
FreeBSD poudriere 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64
```

Nothing out of the ordinary, an update-upgrade like any other.


----------



## DutchDaemon (Dec 12, 2018)

This is my first upgrade to 12 (logically: it's the package builder for everything else) .. I think I'll hold off on any more installs.


----------



## DutchDaemon (Dec 12, 2018)

Whilst running poudriere, messages is even less congratulatory ..


```
Dec 12 20:10:00 poudriere /usr/sbin/cron[16040]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:10:00 poudriere /usr/sbin/cron[16040]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:15:00 poudriere /usr/sbin/cron[16846]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:15:00 poudriere /usr/sbin/cron[16846]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:20:00 poudriere /usr/sbin/cron[19021]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:20:00 poudriere /usr/sbin/cron[19021]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:25:00 poudriere /usr/sbin/cron[21931]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:25:00 poudriere /usr/sbin/cron[21931]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:30:00 poudriere /usr/sbin/cron[37278]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:30:00 poudriere /usr/sbin/cron[37278]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:35:00 poudriere /usr/sbin/cron[72768]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:35:00 poudriere /usr/sbin/cron[72768]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:35:01 poudriere kernel: pid 73980 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:35:01 poudriere kernel: pid 74814 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:39:57 poudriere kernel: pid 13561 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:40:00 poudriere /usr/sbin/cron[16460]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:40:00 poudriere /usr/sbin/cron[16460]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:42:35 poudriere kernel: pid 58377 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:42:46 poudriere kernel: pid 68564 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:45:00 poudriere /usr/sbin/cron[24871]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:45:00 poudriere /usr/sbin/cron[24871]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:45:49 poudriere kernel: pid 82878 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:48:33 poudriere kernel: pid 29279 (conftest), uid 65534: exited on signal 11 (core dumped)
Dec 12 20:50:00 poudriere /usr/sbin/cron[96351]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Dec 12 20:50:00 poudriere /usr/sbin/cron[96351]: in openpam_load_module(): no pam_nologin.so found
Dec 12 20:51:40 poudriere kernel: pid 18725 (conftest), uid 65534: exited on signal 11 (core dumped)
```


----------



## yuripv (Dec 12, 2018)

DutchDaemon said:


> >>>>> fstat >>>>> FBSD_1.5



This looks weird, someone forgot to properly resolve the merge conflict?

EDIT: or am I just being stupid and you added that to show it's present?


----------



## DutchDaemon (Dec 12, 2018)

yuripv said:


> This looks weird, someone forgot to properly resolve the merge conflict?
> 
> EDIT: or am I just being stupid and you added that to show it's present?



"Emphasis mine" -- that was it.


----------



## DutchDaemon (Dec 12, 2018)

BTW, the install order here was:
`freebsd-update upgrade -r 12.0-RELEASE`
`freebsd-update install`
`shutdown -r now`
`.. poudriere build of new packages in a fresh 12.0 jail ...`
`pkg upgrade -fy`
`service -R`
`freebsd-update install`


----------



## abishai (Dec 12, 2018)

Is it release 'feature' ? I run RC2 on my laptop normally.


----------



## drhowarddrfine (Dec 12, 2018)

When upgrading my little server, I noticed one of the manual changes was for cron. I didn't save it, and I didn't care for that throwaway server, but it changed one of the numeric values from something like 100 to 1000. I'm on my phone so I guess that's most unhelpful but something someone can look into if it applies  to anything.


----------



## eydaimon (Dec 12, 2018)

Seems that restarting the service fixed the problem for me.

I did  a [FONT=Courier New]service cron stop[/FONT], then I ran it in the foreground with [FONT=Courier New]cron -n -x load[/FONT] to debug, and it worked with that so I started the service up again and now it works fine.


----------



## eydaimon (Dec 12, 2018)

no more errors in the logfile either. so my guess is that the cron started before the install was finished, and this caused some issues


----------



## DutchDaemon (Dec 13, 2018)

Had to wait for a huge Poudriere run to finish, but a reboot appears to have fixed this. I'm sure it'll be addressed in a patch or an ERRATA.


----------



## meteor8488 (Feb 5, 2019)

I updated two of my server to 12.0, both have the cron issue:


```
Feb  5 16:45:00 r430 /usr/sbin/cron[80795]: (root) CMD (/usr/libexec/atrun)
Feb  5 16:46:00 r430 /usr/sbin/cron[80796]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Feb  5 16:46:00 r430 /usr/sbin/cron[80796]: in openpam_load_module(): no pam_nologin.so found
Feb  5 16:46:00 r430 /usr/sbin/cron[80796]: (CRON) error (can't start PAM)
Feb  5 16:50:00 r430 /usr/sbin/cron[80798]: (root) CMD (/usr/libexec/atrun)
Feb  5 16:55:00 r430 /usr/sbin/cron[80801]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 16:55:00 r430 /usr/sbin/cron[80802]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:00:00 r430 /usr/sbin/cron[80816]: (root) CMD (newsyslog)
Feb  5 17:00:00 r430 /usr/sbin/cron[80817]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 17:00:00 r430 /usr/sbin/cron[80818]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:05:00 r430 /usr/sbin/cron[80830]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:10:00 r430 /usr/sbin/cron[80832]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:11:00 r430 /usr/sbin/cron[80834]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 17:15:00 r430 /usr/sbin/cron[80846]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:20:00 r430 /usr/sbin/cron[80848]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:22:00 r430 /usr/sbin/cron[80850]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 17:25:00 r430 /usr/sbin/cron[80862]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:30:00 r430 /usr/sbin/cron[80864]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:33:00 r430 /usr/sbin/cron[80866]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 17:35:00 r430 /usr/sbin/cron[80878]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:40:00 r430 /usr/sbin/cron[80880]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:44:00 r430 /usr/sbin/cron[80882]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 17:45:00 r430 /usr/sbin/cron[80894]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:50:00 r430 /usr/sbin/cron[80896]: (root) CMD (/usr/libexec/atrun)
Feb  5 17:55:00 r430 /usr/sbin/cron[80899]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 17:55:00 r430 /usr/sbin/cron[80900]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:00:00 r430 /usr/sbin/cron[80914]: (root) CMD (newsyslog)
Feb  5 18:00:00 r430 /usr/sbin/cron[80915]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 18:00:00 r430 /usr/sbin/cron[80916]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:05:00 r430 /usr/sbin/cron[80928]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:10:00 r430 /usr/sbin/cron[80930]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:11:00 r430 /usr/sbin/cron[80932]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 18:15:00 r430 /usr/sbin/cron[80944]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:20:00 r430 /usr/sbin/cron[80946]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:22:00 r430 /usr/sbin/cron[80948]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 18:25:00 r430 /usr/sbin/cron[80960]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:30:00 r430 /usr/sbin/cron[80962]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:33:00 r430 /usr/sbin/cron[80965]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 18:35:00 r430 /usr/sbin/cron[80977]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:40:00 r430 /usr/sbin/cron[80979]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:44:00 r430 /usr/sbin/cron[80981]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 18:45:00 r430 /usr/sbin/cron[80993]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:46:00 r430 /usr/sbin/cron[80994]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
Feb  5 18:46:00 r430 /usr/sbin/cron[80994]: in openpam_load_module(): no pam_nologin.so found
Feb  5 18:46:00 r430 /usr/sbin/cron[80994]: (CRON) error (can't start PAM)
Feb  5 18:50:00 r430 /usr/sbin/cron[81004]: (root) CMD (/usr/libexec/atrun)
Feb  5 18:51:15 r430 1 2019-02-05T18:51:15.952948+08:00 r430.chiphell.com crontab 81010 - - (root) LIST (root)
Feb  5 18:55:00 r430 /usr/sbin/cron[81017]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 18:55:00 r430 /usr/sbin/cron[81018]: (root) CMD (/usr/libexec/atrun)
Feb  5 19:00:00 r430 /usr/sbin/cron[81032]: (root) CMD (newsyslog)
Feb  5 19:00:00 r430 /usr/sbin/cron[81033]: (operator) CMD (/usr/libexec/save-entropy)
Feb  5 19:00:00 r430 /usr/sbin/cron[81034]: (root) CMD (/usr/libexec/atrun)
root@r430:~ # crontab -l
```

Does anyone know a fix?


----------



## DutchDaemon (Feb 5, 2019)

Widely reported issue. After finishing the upgrade a `service cron restart` should be enough.


----------

