# FreeBSD 9.2 crashed, please help to find out why



## Sartrik (Oct 20, 2013)

Hi,

I have one server acting as a host for VirtualBox-OSE. Not a long time ago I updated from 9.1 to 9.2 on 10.10.2013. The system worked fine for 10 days and on 19.10.2013 it crashed. I do not know the reason, there is nothing interesting in the log files.  After a system update I rebuilt VB-OSE-kmod. I tried to use `kgdb` to debug the dump but I was not able to find the reason.

Any ideas? Should I send the crash dump to the VirtualBox developers?

Dumps:

`uname -a`

```
FreeBSD xbsdhost 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013     root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```
 
`kldstat`

```
Id Refs Address            Size     Name
 1   29 0xffffffff80200000 15b93c0  kernel
 2    1 0xffffffff817ba000 24708    geom_mirror.ko
 3    2 0xffffffff817df000 50e30    pf.ko
 4    1 0xffffffff81830000 12e58    if_bridge.ko
 5    2 0xffffffff81843000 82c8     bridgestp.ko
 7    3 0xffffffff81859000 17d08    netgraph.ko
 8    3 0xffffffff81871000 5b298    vboxdrv.ko
 9    1 0xffffffff81a12000 1583db   zfs.ko
10    1 0xffffffff81b6b000 56f7     opensolaris.ko
11    1 0xffffffff81b71000 2bce     pflog.ko
12    2 0xffffffff81b74000 2dad     vboxnetflt.ko
13    1 0xffffffff81b77000 4446     ng_ether.ko
14    1 0xffffffff81b7c000 4046     vboxnetadp.ko
```
 
`gdb`

```
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:234
234     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) list *0xffffffff80923b6f
0xffffffff80923b6f is in callout_tickstofirst (/usr/src/sys/kern/kern_timeout.c:382).
warning: Source file is more recent than executable.
377             curticks = cc->cc_ticks;
378             while( skip < ncallout && skip < limit ) {
379                     sc = &cc->cc_callwheel[ (curticks+skip) & callwheelmask ];
380                     /* search scanning ticks */
381                     TAILQ_FOREACH( c, sc, c_links.tqe ){
382                             if (c->c_time - curticks <= ncallout)
383                                     goto out;
384                     }
385                     skip++;
386             }
(kgdb) backtrace
#0  doadump (textdump=<value optimized out>) at pcpu.h:234
#1  0xffffffff8090d486 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:449
#2  0xffffffff8090d987 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:637
#3  0xffffffff80cf20d0 in trap_fatal (frame=0xc, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:879
#4  0xffffffff80cf2431 in trap_pfault (frame=0xffffff80002ce870, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:795
#5  0xffffffff80cf29e4 in trap (frame=0xffffff80002ce870) at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80cdbd13 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff80923b6f in callout_tickstofirst (limit=250) at /usr/src/sys/kern/kern_timeout.c:381
#8  0xffffffff80d68721 in getnextcpuevent (event=0xffffff80002ce9b0, idle=1) at /usr/src/sys/kern/kern_clocksource.c:282
#9  0xffffffff80d6a731 in cpu_idleclock () at /usr/src/sys/kern/kern_clocksource.c:791
#10 0xffffffff80ce2b61 in cpu_idle (busy=0) at /usr/src/sys/amd64/amd64/machdep.c:804
#11 0xffffffff809382ea in sched_idletd (dummy=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:2625
#12 0xffffffff808db67f in fork_exit (callout=0xffffffff809381d0 <sched_idletd>, arg=0x0, frame=0xffffff80002ceb00) at /usr/src/sys/kern/kern_fork.c:992
#13 0xffffffff80cdc23e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606
#14 0x0000000000000000 in ?? ()
(kgdb) up
#1  0xffffffff8090d486 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:449
warning: Source file is more recent than executable.
```
 
`last`

```
# last
admin1991  pts/0    x.dyn.plu Sun Oct 20 00:22   still logged in
boot time                                  Sun Oct 20 00:19
shutdown time                              Sun Oct 20 00:17
admin1991  pts/0    x.dyn.plu Sat Oct 19 23:57 - shutdown  (00:20)
boot time                                  Sat Oct 19 18:46           [highlight]<<<<----CRASH[/highlight]
admin1991  pts/0   x.dyn.plu Fri Oct 18 21:39 - 22:33  (00:54)
admin1991  pts/0    x.dyn.plu Thu Oct 17 20:31 - 20:39  (00:07)
admin1991  pts/0    x.dyn.plu Wed Oct 16 20:43 - 00:51  (04:08)
admin1991  pts/0    x.dyn.plu Tue Oct 15 21:49 - 21:51  (00:02)
admin1991  pts/0    x.dyn.plu Mon Oct 14 22:11 - 22:15  (00:04)
admin1991  pts/0    x.dyn.plu Sun Oct 13 20:54 - 21:00  (00:05)
admin1991  pts/0    x.dyn.plu Sat Oct 12 16:43 - 16:45  (00:01)
admin1991  pts/1    x.dyn.plu Sat Oct 12 16:26 - 16:43  (00:16)
admin1991  pts/0    x.dyn.plu Sat Oct 12 16:15 - 16:27  (00:11)
admin1991  pts/1    x.dyn.plu Thu Oct 10 20:45 - 21:34  (00:49)
admin1991  pts/0    x.dyn.plu Thu Oct 10 20:38 - 20:45  (00:07)
admin1991  pts/0    x.dyn.plu Thu Oct 10 16:32 - 16:33  (00:00)
admin1991  pts/1    x.dyn.plu Thu Oct 10 16:13 - 16:32  (00:19)
admin1991  pts/2    x.dyn.plu Thu Oct 10 16:03 - 16:15  (00:12)
admin1991  pts/0    x.dyn.plu Thu Oct 10 14:54 - 16:05  (01:11)
boot time                                  Thu Oct 10 14:52
[highlight]shutdown time                              Thu Oct 10 14:51   <<<<9.1 --> 9.2[/highlight]
admin1991  pts/0    x.dyn.plu Thu Oct 10 14:09 - shutdown  (00:41)
boot time                                  Thu Oct 10 14:06
shutdown time                              Thu Oct 10 13:57
boot time                                  Thu Oct 10 13:54
shutdown time                              Thu Oct 10 13:27
boot time                                  Thu Oct 10 13:20
shutdown time                              Thu Oct 10 13:03
boot time                                  Thu Oct 10 13:02
shutdown time                              Thu Oct 10 13:01
boot time                                  Wed Oct  9 22:09
shutdown time                              Wed Oct  9 22:08
admin1991  pts/0    x.dyn.plu Wed Oct  9 21:39 - shutdown  (00:28)
boot time                                  Wed Oct  9 21:34
shutdown time                              Wed Oct  9 21:33
admin1991  pts/0    x.dyn.plu Wed Oct  9 19:59 - shutdown  (01:34)
admin1991  pts/0    x           Tue Oct  8 15:13 - 15:28  (00:14)
admin1991  pts/0    x.dyn.plu Mon Oct  7 23:15 - 23:23  (00:08)
admin1991  pts/0    x.dyn.plu Sun Oct  6 22:59 - 23:04  (00:04)
admin1991  pts/0    x.dyn.plu Fri Oct  4 21:07 - 21:48  (00:40)
admin1991  pts/0    x.dyn.plu Thu Oct  3 22:49 - 22:53  (00:03)
admin1991  pts/1    x         Wed Oct  2 23:54 - 00:03  (00:08)
admin1991  pts/0   x         Wed Oct  2 23:44 - 01:54  (02:10)
admin1991  pts/0   x         Tue Oct  1 23:12 - 23:22  (00:09)
```
 
core.txt.1

```
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address<->= 0x4010
fault code<----><------>= supervisor read data, page not present
instruction pointer<--->= 0x20:0xffffffff80923b6f
stack pointer<->        = 0x28:0xffffff80002ce920
frame pointer<->        = 0x28:0xffffff80002ce960
code segment<--><------>= base 0x0, limit 0xfffff, type 0x1b
<------><------><------>= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags<------>= resume, IOPL = 0
current process><------>= 11 (idle: cpu3)
trap number<---><------>= 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff80947986 at kdb_backtrace+0x66
#1 0xffffffff8090d9ae at panic+0x1ce
#2 0xffffffff80cf20d0 at trap_fatal+0x290
#3 0xffffffff80cf2431 at trap_pfault+0x211
#4 0xffffffff80cf29e4 at trap+0x344
#5 0xffffffff80cdbd13 at calltrap+0x8
#6 0xffffffff80d68721 at getnextcpuevent+0x141
#7 0xffffffff80d6a731 at cpu_idleclock+0x81
#8 0xffffffff80ce2b61 at cpu_idle+0x91
#9 0xffffffff809382ea at sched_idletd+0x11a
#10 0xffffffff808db67f at fork_exit+0x11f
#11 0xffffffff80cdc23e at fork_trampoline+0xe
Uptime: 9d3h50m43s
Dumping 1955 out of 32606 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/modules/vboxdrv.ko...Reading symbols from /boot/modules/vboxdrv.ko.symbols...done.
done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/modules/vboxnetflt.ko...Reading symbols from /boot/modules/vboxnetflt.ko.symbols...done.
done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/modules/vboxnetadp.ko...Reading symbols from /boot/modules/vboxnetadp.ko.symbols...done.
done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:234
234<--->pcpu.h: No such file or directory.
<------>in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:234
#1  0xffffffff8090d486 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:449
#2  0xffffffff8090d987 in panic (fmt=0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:637
#3  0xffffffff80cf20d0 in trap_fatal (frame=0xc, eva=<value optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:879
#4  0xffffffff80cf2431 in trap_pfault (frame=0xffffff80002ce870, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:795
#5  0xffffffff80cf29e4 in trap (frame=0xffffff80002ce870)
    at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80cdbd13 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff80923b6f in callout_tickstofirst (limit=250)
    at /usr/src/sys/kern/kern_timeout.c:381
#8  0xffffffff80d68721 in getnextcpuevent (event=0xffffff80002ce9b0, idle=1)
    at /usr/src/sys/kern/kern_clocksource.c:282
#9  0xffffffff80d6a731 in cpu_idleclock ()
    at /usr/src/sys/kern/kern_clocksource.c:791
#10 0xffffffff80ce2b61 in cpu_idle (busy=0)
    at /usr/src/sys/amd64/amd64/machdep.c:804
#11 0xffffffff809382ea in sched_idletd (dummy=<value optimized out>)
    at /usr/src/sys/kern/sched_ule.c:2625
#12 0xffffffff808db67f in fork_exit (
    callout=0xffffffff809381d0 <sched_idletd>, arg=0x0,.
    frame=0xffffff80002ceb00) at /usr/src/sys/kern/kern_fork.c:992
#13 0xffffffff80cdc23e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:606
#14 0x0000000000000000 in ?? ()
(kgdb).
```


----------

