# Panic kernel FreeBSD 10.0



## kirush (May 4, 2015)

```
# uname -a
FreeBSD nas3.xxx.net 10.0-RELEASE-p9 FreeBSD 10.0-RELEASE-p9 #0: Thu Sep 25 04:23:16 MSK 2014
```
Please help to find problem why kernel panic?


```
# kgdb kernel.debug /var/crash/vmcore.7
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
instruction pointer    = 0x20:0xffffffff80a49825
stack pointer           = 0x28:0xfffffe011ea908a0
frame pointer           = 0x28:0xfffffe011ea908f0
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 0 (dummynet)
trap number        = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809321b0 at kdb_backtrace+0x60
#1 0xffffffff808f9c95 at panic+0x155
#2 0xffffffff80d167e2 at trap_fatal+0x3a2
#3 0xffffffff80d16ab9 at trap_pfault+0x2c9
#4 0xffffffff80d16246 at trap+0x5e6
#5 0xffffffff80cfd4e2 at calltrap+0x8
#6 0xffffffff809bf1ae at netisr_dispatch_src+0x5e
#7 0xffffffff80afd79c at dummynet_send+0x10c
#8 0xffffffff80afd394 at dummynet_task+0x2c4
#9 0xffffffff8093ff46 at taskqueue_run_locked+0xe6
#10 0xffffffff809407c8 at taskqueue_thread_loop+0xa8
#11 0xffffffff808cb05a at fork_exit+0x9a
#12 0xffffffff80cfda1e at fork_trampoline+0xe
Uptime: 1d0h35m18s
Dumping 570 out of 4045 MB:..3%..12%..23%..31%..43%..51%..62%..71%..82%..93%

Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ng_ipfw.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ipfw.ko.symbols
Reading symbols from /boot/kernel/ng_car.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_car.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_socket.ko.symbols
Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_tee.ko.symbols
Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko.symbols
Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_vjc.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219        __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff808f9910 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808f9cd4 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80d167e2 in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:882
#4  0xffffffff80d16ab9 in trap_pfault (frame=0xfffffe011ea907f0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699
#5  0xffffffff80d16246 in trap (frame=0xfffffe011ea907f0) at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80cfd4e2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff80a49825 in ip_input (m=0xfffff80008727700) at /usr/src/sys/netinet/ip_input.c:623
#8  0xffffffff809bf1ae in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0x4000000001) at /usr/src/sys/net/netisr.c:972
#9  0xffffffff80afd79c in dummynet_send (m=<value optimized out>) at /usr/src/sys/netpfil/ipfw/ip_dn_io.c:665
#10 0xffffffff80afd394 in dummynet_task (context=<value optimized out>, pending=<value optimized out>) at /usr/src/sys/netpfil/ipfw/ip_dn_io.c:625
#11 0xffffffff8093ff46 in taskqueue_run_locked (queue=0xfffff800103b6c00) at /usr/src/sys/kern/subr_taskqueue.c:333
#12 0xffffffff809407c8 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:535
#13 0xffffffff808cb05a in fork_exit (callout=0xffffffff80940720 <taskqueue_thread_loop>, arg=0xffffffff8151efc8, frame=0xfffffe011ea90ac0) at /usr/src/sys/kern/kern_fork.c:995
#14 0xffffffff80cfda1e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606
#15 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
```


----------



## Remington (May 4, 2015)

Have you tried upgrading to 10.1?  All new major releases have some sorts of bug.


----------



## kirush (May 4, 2015)

I have such 3 servers 10.0, all others OK, I think problem is software, but I can't read this dump (don't understand) and can't understand in which module problem is.


----------



## SirDice (May 4, 2015)

FreeBSD 10.0 is end-of-life since February 2015 and is not supported any more. Please upgrade to 10.1.

Topics about unsupported FreeBSD versions
https://www.freebsd.org/security/unsupported.html


----------



## Remington (May 4, 2015)

If the problem is still present in 10.1 then you can file PR to have this looked at.  It's not that difficult to upgrade to 10.1 with buildworld.  I don't think anyone is going to help until you upgrade to 10.1.  You may want to check the configs and compare software installed in all of your servers to see which one stands out that could be causing the problem.


----------



## kirush (May 6, 2015)

2 days ago as you advised I upgraded to 10.1:


```
# uname -a
FreeBSD nas3.XXX.net 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue May  5 00:50:29 MSK 2015     root@nas3.XXX.net:/usr/obj/usr/src/sys/GENERIC  amd64
```
Yesterday server restarted again:

```
root@nas3:~ # last -10
root       pts/1    10.1.2.2               Thu May  7 00:06   still logged in
root       pts/0    help.XXX.net      Wed May  6 23:56   still logged in
boot time                                  Wed May  6 18:56
root       pts/0    10.1.2.2               Tue May  5 11:54 - 13:44  (01:49)
```

In /var/crash have:

```
-rw-------  1 root  wheel  639356928 May  6 18:55 vmcore.8
```
kgdb:

```
root@nas3:/usr/obj/usr/src/sys/GENERIC # kgdb kernel.debug /var/crash/vmcore.8
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel/netgraph.ko.symbols
Reading symbols from /boot/kernel/ng_ipfw.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ipfw.ko.symbols
Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
Loaded symbols for /boot/kernel/ipfw.ko.symbols
Reading symbols from /boot/kernel/ng_car.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_car.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_socket.ko.symbols
Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_mppc.ko.symbols
Reading symbols from /boot/kernel/rc4.ko.symbols...done.
Loaded symbols for /boot/kernel/rc4.ko.symbols
Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ether.ko.symbols
Reading symbols from /boot/kernel/ng_pppoe.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_pppoe.ko.symbols
Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_tee.ko.symbols
Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_pptpgre.ko.symbols
Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ksocket.ko.symbols
Reading symbols from /boot/kernel/ng_iface.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_iface.ko.symbols
Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ppp.ko.symbols
Reading symbols from /boot/kernel/ng_netflow.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_netflow.ko.symbols
Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko.symbols
Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_bpf.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_vjc.ko.symbols
Reading symbols from /boot/kernel/ng_l2tp.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_l2tp.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219        __asm("movq %%gs:%1,%0" : "=r" (td)
```
`(kgdb) bt:`

```
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80927da2 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xffffffff80928164 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80d24f5f in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4  0xffffffff80d25278 in trap_pfault (frame=0xfffffe011a565310, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5  0xffffffff80d248da in trap (frame=0xfffffe011a565310) at /usr/src/sys/amd64/amd64/trap.c:440
#6  0xffffffff80d0a7c2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff81a466db in bpf_filter (pc=0x34646dc7, p=0xfffff8003e94380e "E", wirelen=40, buflen=40) at /usr/src/sys/modules/netgraph/bpf/../../../net/bpf_filter.c:193
#8  0xffffffff81a4645f in ng_bpf_rcvdata (hook=<value optimized out>, item=<value optimized out>) at /usr/src/sys/modules/netgraph/bpf/../../../netgraph/ng_bpf.c:476
#9  0xffffffff8195d0ca in ng_apply_item (node=0xfffff80119342a00, item=0xfffff8003e2c5400, rw=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2397
#10 0xffffffff8195cd1a in ng_snd_item (item=0xfffff8003e2c5400, flags=<value optimized out>) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2314
#11 0xffffffff81a31d59 in ng_iface_send (ifp=0xfffff80119d61800, m=<value optimized out>, sa=<value optimized out>) at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:473
#12 0xffffffff81a318b8 in ng_iface_output (ifp=0xfffff80119d61800, m=0xfffff8003ec6da00, dst=<value optimized out>, ro=0x0)
    at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:408
#13 0xffffffff80a59024 in ip_output (m=0xfffff8003ec6da00, opt=<value optimized out>, flags=<value optimized out>, imo=<value optimized out>, inp=<value optimized out>)
    at /usr/src/sys/netinet/ip_output.c:638
#14 0xffffffff80a558ff in ip_forward (m=0xfffff8003ec6da00, srcrt=<value optimized out>) at /usr/src/sys/netinet/ip_input.c:1472
#15 0xffffffff80a5557d in ip_input (m=0xfffff8003ec6da00) at /usr/src/sys/netinet/ip_input.c:681
#16 0xffffffff809f44e2 in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0x0) at /usr/src/sys/net/netisr.c:972
#17 0xffffffff809eb996 in ether_demux (ifp=<value optimized out>, m=0xfffff8003ec6da00) at /usr/src/sys/net/if_ethersubr.c:851
#18 0xffffffff809ec63e in ether_nh_input (m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:646
#19 0xffffffff809f44e2 in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0x0) at /usr/src/sys/net/netisr.c:972
#20 0xffffffff804d45db in em_rxeof (count=99) at /usr/src/sys/dev/e1000/if_em.c:4532
#21 0xffffffff804d3d3c in em_handle_que (context=0xfffffe0000a4e000, pending=<value optimized out>) at /usr/src/sys/dev/e1000/if_em.c:1534
#22 0xffffffff80971475 in taskqueue_run_locked (queue=0xfffff80002ba4d00) at /usr/src/sys/kern/subr_taskqueue.c:342
#23 0xffffffff80971f08 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:563
#24 0xffffffff808f8b6a in fork_exit (callout=0xffffffff80971e60 <taskqueue_thread_loop>, arg=0xfffffe0000a50738, frame=0xfffffe011a565ac0) at /usr/src/sys/kern/kern_fork.c:996
#25 0xffffffff80d0acfe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606
#26 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
```
Please help to cure it.


----------



## SirDice (May 7, 2015)

It looks like it crashed somewhere in the networking code. I also saw some references to dummynet(4), is there a firewall running? Does it use traffic shaping?


----------



## kpa (May 7, 2015)

Looks like there is ipfw(8) involved. Try asking on the freebsd-net and/or freebsd-ipfw mailing lists.


----------



## kirush (May 7, 2015)

Yes, iI use a firewall, only one pipe:

```
00400    161854      8289970 fwd X.X.X.X,3128 tcp from table(1) to any dst-port 80
00401    199967     10465560 pipe tablearg ip from any to table(1)
00401    187811     10188461 pipe tablearg ip from table(2) to any
```


```
${fwcmd} pipe 1 config bw 256Kbit/s mask dst-ip 0xffffffff
${fwcmd} pipe 2 config bw 256Kbit/s mask src-ip 0xffffffff
```


----------



## junovitch@ (May 9, 2015)

All the Netgraph modules as well make it seem like you are doing some fairly advanced networking and could be running into an edge case when it comes to the peculiarities of your configuration.  A mailing list post as kpa mentioned would be a good idea.  While waiting on responses, removing some of the variables like Netgraph or specific firewall rules and testing again could help narrow down where the issues is through some trial and error.


----------



## kirush (May 10, 2015)

Netgraph I used time ago, I don't need it now. I will try to compile kernel without netgraph modules. Thanks.


----------



## junovitch@ (May 10, 2015)

There's no guarantee it will help.  The first panic didn't list any Netgraph related functions in the backtrace but the later panic did.  So starting off simplifying things to reduce the number of variables it could be makes sense.


----------



## kirush (May 16, 2015)

I made upgrade and now I have:
`root@nas:~ # uname -a`

```
FreeBSD nas.XXX.net 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #2: Wed May 13 22:49:38 MSK 2015     root@nas.XXX.net:/usr/obj/usr/src/sys/nas  amd64
```
in kernel I added only:

```
options         IPFIREWALL
options         DUMMYNET
```
Today 2 servers on 10.1 rebooted in different time.
First server:
`root@nas:/usr/obj/usr/src/sys/nas # kgdb kernel.debug /var/crash/vmcore.1`

```
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Cannot access memory at address 0x0
(kgdb) bt
#0  0x0000000000000000 in ?? ()
Cannot access memory at address 0x0
```

Second:
`root@nas3:/usr/obj/usr/src/sys/nas # kgdb kernel.debug /var/crash/vmcore.last`

```
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80963e70 at kdb_backtrace+0x60
#1 0xffffffff80928f95 at panic+0x155
#2 0xffffffff80d3840f at trap_fatal+0x38f
#3 0xffffffff80d38728 at trap_pfault+0x308
#4 0xffffffff80d37d8a at trap+0x47a
#5 0xffffffff80d1dc72 at calltrap+0x8
#6 0xffffffff809f5352 at netisr_dispatch_src+0x62
#7 0xffffffff80b1433c at dummynet_send+0x10c
#8 0xffffffff80b13f54 at dummynet_task+0x2c4
#9 0xffffffff809722e5 at taskqueue_run_locked+0xe5
#10 0xffffffff80972d78 at taskqueue_thread_loop+0xa8
#11 0xffffffff808f99da at fork_exit+0x9a
#12 0xffffffff80d1e1ae at fork_trampoline+0xe
Uptime: 4d21h21m20s
Dumping 555 out of 4044 MB:..3%..12%..21%..32%..41%..52%..61%..72%..81%..93%

Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel/netgraph.ko.symbols
Reading symbols from /boot/kernel/ng_ipfw.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ipfw.ko.symbols
Reading symbols from /boot/kernel/ng_car.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_car.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_socket.ko.symbols
Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_mppc.ko.symbols
Reading symbols from /boot/kernel/rc4.ko.symbols...done.
Loaded symbols for /boot/kernel/rc4.ko.symbols
Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ether.ko.symbols
Reading symbols from /boot/kernel/ng_pppoe.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_pppoe.ko.symbols
Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_tee.ko.symbols
Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_pptpgre.ko.symbols
Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ksocket.ko.symbols
Reading symbols from /boot/kernel/ng_iface.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_iface.ko.symbols
Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ppp.ko.symbols
Reading symbols from /boot/kernel/ng_netflow.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_netflow.ko.symbols
Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko.symbols
Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_bpf.ko.symbols
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_vjc.ko.symbols
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219        __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80928c12 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xffffffff80928fd4 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80d3840f in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4  0xffffffff80d38728 in trap_pfault (frame=0xfffffe011e99c800, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5  0xffffffff80d37d8a in trap (frame=0xfffffe011e99c800) at /usr/src/sys/amd64/amd64/trap.c:440
#6  0xffffffff80d1dc72 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff80a5627d in ip_input (m=0xfffff8004b041400) at /usr/src/sys/netinet/ip_input.c:594
#8  0xffffffff809f5352 in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0x40000000c1)
    at /usr/src/sys/net/netisr.c:972
#9  0xffffffff80b1433c in dummynet_send (m=<value optimized out>) at /usr/src/sys/netpfil/ipfw/ip_dn_io.c:665
#10 0xffffffff80b13f54 in dummynet_task (context=<value optimized out>, pending=<value optimized out>)
    at /usr/src/sys/netpfil/ipfw/ip_dn_io.c:625
#11 0xffffffff809722e5 in taskqueue_run_locked (queue=0xfffff800193b2a00) at /usr/src/sys/kern/subr_taskqueue.c:342
#12 0xffffffff80972d78 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:563
#13 0xffffffff808f99da in fork_exit (callout=0xffffffff80972cd0 <taskqueue_thread_loop>, arg=0xffffffff815e6800,
    frame=0xfffffe011e99cac0) at /usr/src/sys/kern/kern_fork.c:996
#14 0xffffffff80d1e1ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606
#15 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
```

I have a suspicion of a dummynet module.

`root@nas:/var/crash # sysctl -a | grep dummynet`

```
net.inet.ip.dummynet.hash_size: 64
net.inet.ip.dummynet.pipe_slot_limit: 100
net.inet.ip.dummynet.pipe_byte_limit: 1048576
net.inet.ip.dummynet.io_fast: 0
net.inet.ip.dummynet.debug: 0
net.inet.ip.dummynet.red_lookup_depth: 256
net.inet.ip.dummynet.red_avg_pkt_size: 512
net.inet.ip.dummynet.red_max_pkt_size: 1500
net.inet.ip.dummynet.tick_delta: 0
net.inet.ip.dummynet.tick_delta_sum: -233
net.inet.ip.dummynet.tick_adjustment: 5593
net.inet.ip.dummynet.tick_diff: -4366
net.inet.ip.dummynet.tick_lost: 0
net.inet.ip.dummynet.expire: 1
net.inet.ip.dummynet.expire_cycle: 0
net.inet.ip.dummynet.schk_count: 60
net.inet.ip.dummynet.si_count: 0
net.inet.ip.dummynet.fsk_count: 30
net.inet.ip.dummynet.queue_count: 0
net.inet.ip.dummynet.io_pkt: 1038380
net.inet.ip.dummynet.io_pkt_fast: 0
net.inet.ip.dummynet.io_pkt_drop: 66254
```
I will try:
`root@nas:/var/crash # sysctl -w net.inet.ip.dummynet.io_fast=1`


----------

