# FreeBSD 10 reboots/crashes every day at 4:00 am



## schmidp (Feb 27, 2014)

Hello,

I have a problem with my FreeBSD 10 installation.
It reboots or crashes everyday at around 4:00 am.

The last log entry in /var/logs/all.log is:


```
Feb 25 04:00:00 rna /usr/sbin/cron[14974]: (root) CMD (/usr/libexec/atrun)
Feb 25 04:01:00 rna /usr/sbin/cron[14996]: (root) CMD (adjkerntz -a)
Feb 25 08:50:12 rna syslogd: restart
Feb 25 08:50:12 rna syslogd: kernel boot file is /boot/kernel/kernel
```

I already tried manually running `adjkerntz -a` to see if that is the problem, but that does not seem to be the case as the command runs fine.

The setup uses ZFS on an encrypted geli() device and swap is on a graid() mirror on top of a encrypted geli() device, so the kernel cannot write a kernel crash dump.

Is a reboot the normal behaviour if the kernel crashes?
Can I reconfigure it somehow to stop on a crash and print some debug information to the console?

I’m relatively new to FreeBSD and would be happy for any pointers in the right direction on how to find the cause of the reboot.

Thank you,

	Philipp


----------



## xtaz (Feb 27, 2014)

I've heard of this happening because of the scripts that get run in /etc/periodic/*/. Though the daily one is usually run at 3am rather than 4 but it's worth checking /etc/crontab. Several of the scripts do a `find` which run through the entire filesystem looking for different things. With zfs this can cause memory usage to massively increase until it runs out. I have no experience of zfs myself but I've heard people saying "arc size" needs increasing if that makes any sense?

I've just found this thread: https://forums.freebsd.org/viewtopic.php?&t=42396. Might be similar?


----------



## SirDice (Feb 28, 2014)

schmidp said:
			
		

> Is a reboot the normal behaviour if the kernel crashes?


Yes, because there's no other way to recover from an error like that.



> Can I reconfigure it somehow to stop on a crash and print some debug information to the console?


There should be a switch somewhere but I forgot where. It will keep the message on the screen until you press a key on the console. Mind you, your server will be down until that happens. Is there any way you can set the swap to a non-encrypted device? At least temporarily, a crash dump may provide some extra information as to why it crashed.


----------



## ondra_knezour (Feb 28, 2014)

SirDice said:
			
		

> There should be a switch somewhere but I forgot where.


May be `sysctl debug.debugger_on_panic=1`?

Some reference for kernel debugging (some may be dated):
http://www.freebsd.org/doc/en/books/dev ... debug.html
http://www.bsdcan.org/2008/schedule/att ... rticle.pdf
http://software.intel.com/sites/default ... 321772.pdf


----------



## trh411 (Feb 28, 2014)

ondra_knezour said:
			
		

> SirDice said:
> 
> 
> 
> ...


It's my understanding that sysctl only works if you have debugging (KDB, DDB) enabled in the kernel.

EDIT: See Glossary of Kernel Options for Debugging


----------



## schmidp (Mar 7, 2014)

Hi,

thank you for your replies!

I checked and my kernel has debug support:


```
options         KDB                     # Enable kernel debugger support.
options         KDB_TRACE               # Print a stack trace for a panic.
```

And the sysctrl flag is enabled:


```
root@server:~ # sysctl debug.debugger_on_panic
debug.debugger_on_panic: 1
```

But it just reboots instantly like if someone pressed the reset button.

I also tried disabling periodic by editing /etc/crontab, but again: reboot at 4:03am...

Any other ideas?

Tonight I'll try running a GENERIC kernel and see if that helps...

Thank you, Philipp


----------



## schmidp (Mar 7, 2014)

Morning,

The server does survive the night with a GENERIC kernel 

I'll now try to find the option that causes the crash/reboot...

- Philipp


----------



## schmidp (Mar 27, 2014)

I have now found the root cause:

If a jail is running and the jail executes


```
/etc/periodic/daily/450.status-security
```

the server crashes. This happens every night at 4:03 CET.

If I run


```
/etc/periodic/daily/450.status-security
```

manually in the jail, the server also crashes immediately.

Now I wanted to try to run each script in 


```
/etc/periodic/security
```

manually, but I get:


```
ASSERTION FAILED: Unexpected value for  $PERIODIC: ‘'
```

I then ran:


```
setenv PERIODIC "security daily”
```

which allowed me to run each security script separately. 

If I run:


```
root@jail:/etc/periodic/security # ./520.pfdenied
```

the machine immediately reboots.

Looking at 520.pfdenied I tried running the command:


```
root@jail:~ # pfctl -sr -v
```

directly, which also crashes the host immediately.


----------



## memq (Apr 4, 2014)

I don't know if this is still true for 10.0... As of 9.1, having


```
options         VIMAGE
```

in the kernel configuration rules out all firewals except ipfw.
IIRC, just _thinking_ abot pf would crash my machine.


----------

