# I get fatal trap 12 when starting dahdi



## sergling (Jan 15, 2022)

Hi, 
after upgrade to FreeBSD 12.3 , wctdm24xxp.ko module broke down. On FreeBSD 11.4 all work fine.

I have <Wildcard TDM410P> card.

/etc/rc.conf

```
dahdi_enable="YES"
dahdi_modules="wctdm24xxp"
```
`service dahdi start` -> panic: page fault
`kldload /usr/local/lib/dahdi/wctdm24xxp.ko` -> panic: page fault

I'm install two new VM systems - all starting fine on it. But there is no equipment there.

Either something broke only in my working system, or everyone should have a failure on this board.

I'm build gdb and generic kernel with debug. I'm get dump.


```
sergling@office:~# ./debug.sh
GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD]
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.3".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.debug...

Unread portion of the kernel message buffer:
wctdm24xxp0: vendor=d161 device=8005 subvendor=ffffffff
wctdm24xxp0: <Wildcard TDM410P> port 0xe800-0xe8ff mem 0xdffefc00-0xdffeffff irq 21 at device 1.0 on pci4
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xfffffe0026a7bd60
fault code              = supervisor read instruction, protection violation
instruction pointer     = 0x20:0xfffffe0026a7bd60
stack pointer           = 0x28:0xfffffe001f12aa08
frame pointer           = 0x28:0xfffffe001f12aa20
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (idle: cpu0)
trap number             = 12
panic: page fault
cpuid = 0
time = 1642242487
KDB: stack backtrace:
#0 0xffffffff80c2d465 at kdb_backtrace+0x65
#1 0xffffffff80be15fb at vpanic+0x17b
#2 0xffffffff80be1473 at panic+0x43
#3 0xffffffff810fd961 at trap_fatal+0x391
#4 0xffffffff810fd9bf at trap_pfault+0x4f
#5 0xffffffff810fd006 at trap+0x286
#6 0xffffffff810d5098 at calltrap+0x8
#7 0xffffffff82a57cdf at vb_isr+0x19f
#8 0xffffffff80ba5822 at intr_event_handle+0x92
#9 0xffffffff812a4622 at intr_execute_handlers+0x52
#10 0xffffffff810d63ac at Xapic_isr1+0xdc
#11 0xffffffff8049b6c0 at acpi_cpu_idle+0x2e0
#12 0xffffffff812a0bee at cpu_idle_acpi+0x3e
#13 0xffffffff812a0c9f at cpu_idle+0x9f
#14 0xffffffff80c14f56 at sched_idletd+0x326
#15 0xffffffff80ba29ce at fork_exit+0x7e
#16 0xffffffff810d60ce at fork_trampoline+0xe
Uptime: 4m23s
Dumping 304 out of 3026 MB:..6%..11%..22%..32%..43%..53%..64%..74%..85%..95%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
```



```
(kgdb) list 0x20:0xfffffe0026a7bd60
No source file named 0x20.
```

I think I need to build the module into debug mode, but I can't do it. I have no experience in solving such problems. 
Either go back to the old version of the system, or wait for help.


----------



## sergling (Jan 15, 2022)

```
(kgdb) backtrace
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff80be1215 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452
#3  0xffffffff80be1653 in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:881
#4  0xffffffff80be1473 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:808
#5  0xffffffff810fd961 in trap_fatal (frame=0xfffffe001f12a940, eva=18446741875334823264)
    at /usr/src/sys/amd64/amd64/trap.c:921
#6  0xffffffff810fd9bf in trap_pfault (frame=0xfffffe001f12a940, usermode=<optimized out>, signo=<optimized out>,
    ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:739
#7  0xffffffff810fd006 in trap (frame=0xfffffe001f12a940) at /usr/src/sys/amd64/amd64/trap.c:405
#8  <signal handler called>
#9  0xfffffe0026a7bd60 in ?? ()
#10 0xffffffff82a59d84 in vb_schedule_deferred (vb=0xfffffe0026a7bd60) at ../../drivers/dahdi/voicebus/voicebus.c:1450
#11 0xffffffff82a57cdf in vb_isr (dev_id=0xfffffe0026a7bd60) at ../../drivers/dahdi/voicebus/voicebus.c:1887
#12 0xffffffff80ba5822 in intr_event_handle (ie=0xfffff800031d5e00, frame=0xfffffe001f12aaf0)
    at /usr/src/sys/kern/kern_intr.c:1318
#13 0xffffffff812a4622 in intr_execute_handlers (isrc=0xfffff800031c6e90, frame=0xfffffe001f12aaf0)
    at /usr/src/sys/x86/x86/intr_machdep.c:357
#14 <signal handler called>
#15 acpi_cpu_c1 () at /usr/src/sys/x86/x86/cpu_machdep.c:216
#16 0xffffffff8049b6c0 in acpi_cpu_idle (sbt=<optimized out>) at /usr/src/sys/dev/acpica/acpi_cpu.c:1187
#17 0xffffffff812a0bee in cpu_idle_acpi (sbt=12061023) at /usr/src/sys/x86/x86/cpu_machdep.c:510
#18 0xffffffff812a0c9f in cpu_idle (busy=0) at /usr/src/sys/x86/x86/cpu_machdep.c:630
#19 0xffffffff80c14f56 in sched_idletd (dummy=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2862
#20 0xffffffff80ba29ce in fork_exit (callout=0xffffffff80c14c30 <sched_idletd>, arg=0x0, frame=0xfffffe001f12ad40)
    at /usr/src/sys/kern/kern_fork.c:1080
#21 <signal handler called>
(kgdb) list *0xffffffff82a57cdf
0xffffffff82a57cdf is in vb_isr (../../drivers/dahdi/voicebus/voicebus.c:1888).
1883                       (TX_COMPLETE_INTERRUPT|RX_COMPLETE_INTERRUPT))) {
1884                    /* ******************************************************** */
1885                    /* NORMAL INTERRUPT CASE                                    */
1886                    /* ******************************************************** */
1887                    vb_schedule_deferred(vb);
1888                    __vb_setctl(vb, SR_CSR5, TX_COMPLETE_INTERRUPT|RX_COMPLETE_INTERRUPT);
1889            } else {
1890                    if (int_status & FATAL_BUS_ERROR_INTERRUPT)
1891                            dev_err(&vb->pdev->dev, "Fatal Bus Error detected!\n");
1892
(kgdb) list *0xffffffff82a59d84
0xffffffff82a59d84 is in vb_schedule_deferred (../../drivers/dahdi/voicebus/voicebus.c:1452).
1447    #if !defined(CONFIG_VOICEBUS_INTERRUPT)
1448            tasklet_hi_schedule(&vb->tasklet);
1449    #else
1450            vb->tasklet.func(vb->tasklet.data);
1451    #endif
1452    }
1453
1454    /**
1455     * vb_tasklet_boot() - When vb->mode == BOOT
1456     *
```

vb->tasklet.func(vb->tasklet.data); - It looks like everything is falling down here


----------



## _martin (Jan 15, 2022)

This seems to be 3rd party module -- I don't see it neither in 12 nor in 13 base. Quick search pointed me dahdi github ; I'm assuming it's that. You should ask developers about the issue there. 

In the backtrace you provided frame 10 is the interesting one. So in kgdb 
	
	



```
f 10
x/12i $pc
```
 would be more interesting. 
At first glance it may be that this is a ro area of the kernel where write was attempted. But that x/12i command would tell more. But true or not still you'd need to ask somebody who is familair with this 3rd party module.


----------



## sergling (Jan 19, 2022)

The FreeBSD forum broke down, sadly.

I found out that the problem is here: vb->tasklet.func(vb->tasklet.data);

I had to make an edit to the code, rebuild the module.


```
(kgdb) f 18
#18 0xffffffff82a59de1 in vb_schedule_deferred (vb=0xfffffe0026af3d60) at ../../drivers/dahdi/voicebus/voicebus.c:1453
1453                vb_tasklet_normal(vb); /*vb_tasklet_normal(vb->tasklet.data);*/
(kgdb) list +
1448            tasklet_hi_schedule(&vb->tasklet);
1449    #else
1450            /*vb->tasklet.func(vb->tasklet.data);*/
1451            switch(vb->mode){
1452            case NORMAL:
1453                vb_tasklet_normal(vb); /*vb_tasklet_normal(vb->tasklet.data);*/
1454                break;
1455            case BOOT:
1456                vb_tasklet_boot(vb->tasklet.data);
1457                break;
(kgdb) p vb->tasklet
$1 = {task = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff82a55800 <vb_tasklet_normal>, ta_context = 0x0}, func = 0xfffffe0026af3d60, data = 0,
  disable_count = 0}
(kgdb)
```

The structure tasklet contains incorrect data. The func refers to the forbidden area, data contains 0, although it should contain a pointer to vb

I have found the initialization procedure:

```
void
tasklet_init(struct tasklet_struct *t, void (*func)(unsigned long), unsigned long data)
{
    TASK_INIT(&t->task, PI_REALTIME, tasklet_run, t);
    t->func = func;
    t->data = data;
    t->disable_count = 0;
}
```

It seems to me that something broke even earlier. I fixed the code as best I could, but I got a crash again in new place. The module developer is lost, the current mantainer can't help anything. As a result, my asterisk stopped working with the fxs line. I tried to deploy a clean system on 11.4, but after the build I got a crash again. It looks like the module broke earlier, I just haven't assembled it for a long time. Either go to version 11.1 or 10, but this is already sad. What's scariest of all, I don't want to go to Linux.

I would like to try debugging again, but I don't understand how I can put a breakpoint when loading the kernel module. It would be interesting to see the execution of the initialization procedure of the problematic structure. And in fact, either I was very unlucky, or the module does not work and is not needed by anyone, then it must be removed from the ports. And welcome to linux.


----------



## SirDice (Jan 19, 2022)

sergling said:


> I would like to try debugging again, but I don't understand how I can put a breakpoint when loading the kernel module.


It's probably easier to do if you use remote debugging. 








						Chapter 10. Kernel Debugging
					

FreeBSD Kernel Debugging




					docs.freebsd.org


----------



## _martin (Jan 19, 2022)

You didn't share the output I mentioned. I'm not familiar with this module, I don't know what data it's working with. At first glance it seems more like a problem with `func`. 

You can use two FreeBSD VMs, one for running the code and one for debugging. Compile the kernel to support the online debugger, include gdb stubs. Then when system is booted break to debugger, switch to gdb, set the breakpoint and continue.


----------



## zirias@ (Jan 19, 2022)

Did you compile dahdi-kmod26 yourself or install it from packages? If the latter, I'd bet there's some incompatible in-kernel change between 12.2 and 12.3. As 12.2 is still supported, packages for 12.x are built with 12.2.


----------



## sergling (Jan 19, 2022)

SirDice said:


> It's probably easier to do if you use remote debugging.
> 
> 
> 
> ...


tnx, I was hoping there was something simpler for the module )


----------



## sergling (Jan 19, 2022)

Zirias said:


> Did you compile dahdi-kmod26 yourself or install it from packages? If the latter, I'd bet there's some incompatible in-kernel change between 12.2 and 12.3. As 12.2 is still supported, packages for 12.x are built with 12.2.


only from port, 
I've always worked with dahdi_kmod. After the update, I started trying both, both give an error.


----------



## SirDice (Jan 19, 2022)

Zirias said:


> f the latter, I'd bet there's some incompatible in-kernel change between 12.2 and 12.3. As 12.2 is still supported, packages for 12.x are built with 12.2.


There shouldn't be any but it can't be ruled out, definitely build misc/dahdi-kmod from ports to make 100% sure the module is specifically built for the kernel you currently have running.


----------



## sergling (Jan 19, 2022)

_martin said:


> You didn't share the output I mentioned. I'm not familiar with this module, I don't know what data it's working with. At first glance it seems more like a problem with `func`.
> 
> You can use two FreeBSD VMs, one for running the code and one for debugging. Compile the kernel to support the online debugger, include gdb stubs. Then when system is booted break to debugger, switch to gdb, set the breakpoint and continue.


During the time the forum was not working, I made corrections and rebuilt the module several times. It's hard for me to restore that state. As a result, now the execution goes further and falls already inside the kernel modules. Maybe my edits were incorrect, I can provide more detailed information on the current state.

It won't work with a virtual machine, I've already built a real one. The problem is only with the installed board when the interrupt processing occurs.


----------



## SirDice (Jan 19, 2022)

sergling said:


> It won't work with a virtual machine, I've already built a real one. The problem is only with the installed board when the interrupt processing occurs.


Yeah, the module probably won't attach if the hardware itself isn't there. You can certainly load the module but it won't be active.

But I see that the port itself is marked as BROKEN:

```
BROKEN=		does not compile: use of undeclared identifier 'SX_NOADAPTIVE'
```
That was done a while ago: https://cgit.freebsd.org/ports/comm...e?id=4820f0aa604558a87abdc82299b67c3181a52fbd


Does misc/dahdi-kmod26 work for you?


----------



## sergling (Jan 19, 2022)

SirDice said:


> Yeah, the module probably won't attach if the hardware itself isn't there. You can certainly load the module but it won't be active.
> 
> But I see that the port itself is marked as BROKEN:
> 
> ...








						252907 – [patch] misc/dahdi-kmod and misc/dahdi-kmod26 need -Wmisleading-indentation removed.
					






					bugs.freebsd.org
				




I simple comment BROKEN in make-file and all good compile.

I talked to the fjoe@FreeBSD.org last week, he confirmed that his port was going to be 3 months ago, but they just corrected the assembly, I understand that no one tested with the board.


----------



## sergling (Jan 19, 2022)

SirDice said:


> It's probably easier to do if you use remote debugging.
> 
> 
> 
> ...


Should the versions of both systems match?


----------



## covacat (Jan 19, 2022)

try to just return from vb_schedule_deferred if vb->tasklet.data is zero
i think an interrupt comes before voicebus_start is called so data is zero


----------



## sergling (Jan 19, 2022)

covacat said:


> try to just return from vb_schedule_deferred if vb->tasklet.data is zero
> i think an interrupt comes before voicebus_start is called so data is zero


Good idea, I'll try it tomorrow. Tell me how to write debugging information from the kernel module to the log correctly? I want to make some calls.


----------



## covacat (Jan 19, 2022)

printf(9) works
or look at device_printf(9)


----------



## _martin (Jan 19, 2022)

What version of dahdi are you using and where from ? The one I installed from ports doesn't match your offsets in src.

You can still do the remote debugging of the physical machine but for gdb you need a serial port or firewire . If the board doesn't have serial port (or firewire) you could still use the DDB as a native FreeBSD debugger. I use it very seldom, basically I don't know how to use it properly. Once enabled though ctrl-alt-esc from terminal throws you into debugger where you can debug the system on fly. 

I looked at your traces and code you provided -- I'm a bit confused which output in the reply is from what source. The function around the crash 
	
	



```
static void vb_schedule_deferred(struct voicebus *vb)
{
#if !defined(CONFIG_VOICEBUS_INTERRUPT)
        tasklet_hi_schedule(&vb->tasklet);
#else
        vb->tasklet.func(vb->tasklet.data);
#endif
}
```
Which from the output you provided the #if is true and `tasklet_hi_schedule` is used, so tasklet->task  is enqueued in taskqueue_fast.

What is interesting the tasklet addr is `0xfffffe0026a7bd60` and it's the code where it jumped. If you had a crash and these modules you crashed it with you could debug this a bit better too.


----------

