# "kgdb: command not found"



## PMc (Nov 20, 2022)

As the issue of the iwlwifi network driver is apparently not moving forward from the state of spring, and the bugreports are elegies of anything from dhcp to wifibox and not much useful, I decided to have a closer look myself.

So I've already killed the first nasty vermin 267869, which was simple and didn't need debugging (and encountered a hive of other who are happily celebrating party therein), but then found there is no means to obtain a kernel dump.

This is because suspend/resume does not work without DRM anymore, and panic does not work with DRM anymore, because DRM trausforms a panic into an endless loop of panics, and there will be no dump or reboot.

I finally managed to obtain a dump by jumping into the debugger and entering "dump" there. And now I would like to look into that dump, so I opened the "Developers Handbook", which once had a section about kernel debugging, and found that the commands there don't work anymore, Then I recognized that these have to be installed from ports nowadays, and it came to my mind that we do not use gcc to build the kernel, for quite a while now. I think this is boring - if you fight a monster (or a bug) you want a quick kill, you don't want to wait for boost-libs and texmf and whatever else to download and compile first.

So I wondered if it would be possible to look into a vmcore with the tools that are by default installed on the system, and I vaguely remember that I might once have achieved to do so. But it didn't work anymore.

Then I searched for some description on how it might work now. And indeed I found - tons of hits in the search engine, everywhere! Lots of lengthy papers about debugging FreeBSD with the LLDB debugger.

And they all tell the same: that some moritz(wtf?) is proud to have been engaged by the foundation to develop kernel debugging with LLDB. And then pages and pages of management bullshit with nothing useful at all.

This is just what I found in professional IT: there are shelves and shelves of paper, and they all contain zeroskill. They are there to be moved around in bullshit-bingo meetings, where you can have lengthy talks and then state that there is progress when there is none at all (and obviosely everybody is expected to applaude to the emperor's new clothes).

I always was happy that FreeBSD is different, is from and for the people who want to do something. And I'm starting to wonder if that foundation might start to belong to the _other_ people - those people just moving money, those people who do not have to hunger and freeze...


----------



## _martin (Nov 20, 2022)

I'll answer the topic question (assuming it is one): kgdb comes from the package devel/gdb. I don't see why would you lock yourself out and won't use it just because it's not in base. Few seconds and you have it available for you. Also official foundation's link: debugging support.

For core dump analysis you can use e.g. `kgdb /boot/kernel/kernel /var/crash/vmcore.0` where kgdb is a nice addon to gdb itself. For your attempt to take a look at this though static post-mortem analysis may not be enough and live debugging is a better option. Unfortunately FreeBSD generic kernel doesn't have gdb stub (why? I don't get that..), you do need to compile custom kernel for that. But once you do that you can trap to debugger and switch to gdb stub. This is very nice approach when debugging stuff in VM. 

It's been some time since I debugged physical box directly and even then it was over segger. If you have serial port you could leverage that. Or maybe usb2rs232.


----------



## PMc (Nov 20, 2022)

_martin said:


> I'll answer the topic question (assuming it is one):


You're welcome ("freebsd development" can have at least two meanings, and I thought I'll cover both).



_martin said:


> kgdb comes from the package devel/gdb. I don't see why would you lock yourself out and won't use it just because it's not in base.


Because it's not necessary.
I now figured what I did last time: I entered
`cd /usr/src/sys/amd64/compile/MYKERN
lldb kernel.full --core /var/crash/vmcore.2`
and I get what I want: the sources for the program counter positions and the structure and content of the variables.



_martin said:


> For core dump analysis you can use e.g. `kgdb /boot/kernel/kernel /var/crash/vmcore.0` where kgdb is a nice addon to gdb itself. For your attempt to take a look at this though static post-mortem analysis may not be enough and live debugging is a better option. Unfortunately FreeBSD generic kernel doesn't have gdb stub (why? I don't get that..), you do need to compile custom kernel for that. But once you do that you can trap to debugger and switch to gdb stub.


What's that? Do you mean the Ctrl-Alt-Esc trick? I have that compiled in always, and it's the only way to actually obtain a dump in this case.
But I didn't find a documentation on what else could be done at that point, and as far as I can see, that is all hex, and one would need to know the structures and pointers and basically work on the assembler code level.



_martin said:


> This is very nice approach when debugging stuff in VM.


Hm, that would need to move that iwlwifi beast into a VM. Could probably be done...


----------



## _martin (Nov 20, 2022)

You may hit a problem with lldb kernel debugging down the line as mentioned in the foundation's link. But feel free to chose your weapon.



PMc said:


> What's that? Do you mean the Ctrl-Alt-Esc trick?


No. I was talking about the `options GDB`. Once you enter debugger you can change to the (remote) gdb and continue in 2nd FreeBSD that is attached to the one you entered debugger in. For physical boxes typical way of doing this is over serial port (gdb's target remote command).

Depending where the problem is you may have problems debugging this in VM as, brainstorming here, fw will not get loaded as there's no physical device connected to the machine.


----------



## PMc (Nov 20, 2022)

_martin said:


> No. I was talking about the `options GDB`.


Indeed, what I have added is just `options DDB`. 
So it finally gets clear what these individual options do. KDB is always included, and DDB is the primitive ad-hoc look-into.



_martin said:


> Once you enter debugger you can change to the (remote) gdb and continue in 2nd FreeBSD that is attached to the one you entered debugger in. For physical boxes typical way of doing this is over serial port (gdb's target remote command).


Thank You. That could actually be done if it becomes necessary. I think I have the needed hardware somewhere...



_martin said:


> Depending where the problem is you may have problems debugging this in VM as, brainstorming here, fw will not get loaded as there's no physical device connected to the machine.


Agreed, there might be some fancy problems on that road.


----------



## _martin (Nov 21, 2022)

PMc said:


> Thank You. That could actually be done if it becomes necessary. I think I have the needed hardware somewhere...


I have not debugged machine that doesn't have uart (or better to say connector or pins available). It's not a problem for client (machine where kgdb is running) usb-to-rs232 saves the day. Problem is when server (machine to be debugged) doesn't have it. You could use firewire but I don't think that's an option in most cases.

I usually use 3rd, more powerful machine where I compile the kernel. I install kernel to different location: `make installkernel KERNCONF=MYCONF DESTDIR=/newkernel`. Then I copy the kernel to server and adjust /boot/loader.conf with `hint.uart.0.flags="0x90"`. More info at uart(4). Boot it, drop to debugger and change to gdb stub with the `gdb` db command.

I'm lazy. Easiest way (for me) is to boot client with the same kernel and adjust (or export via nfs) /usr/src /usr/lib/debug. But most certainly you don't need that. Having the same kernel booted avoids issues during symbols resolving. Something you can overwrite with `solib-search-path` in (k)gdb. Debugging session could then look like this:


```
# kgdb -q /newkernel/usr/lib/debug/boot/kernel/kernel.debug
Reading symbols from /newkernel/usr/lib/debug/boot/kernel/kernel.debug...
(kgdb) set serial baud 115200
(kgdb) set debug-file-directory /newkernel/usr/lib/debug
(kgdb) set solib-search-path /newkernel/
(kgdb) target remote /dev/cuau0
Remote debugging using /dev/cuau0
=> 0xffffffff80c70bb7 <kdb_enter+55>:    mov    QWORD PTR [rip+0x1275b4e],0x0        # 0xffffffff81ee6710 <kdb_why>
   0xffffffff80c70bc2 <kdb_enter+66>:    add    rsp,0x8
   0xffffffff80c70bc6 <kdb_enter+70>:    pop    rbx
...
..
(kgdb) info threads
..
  154  Thread 100145 "hello" (Sleeping)                  sched_switch (td=td@entry=0xfffffe01077311e0, flags=<optimized out>, flags@entry=260) at /usr/src/sys/kern/sched_ule.c:2245
(kgdb) p/x ((struct thread*)0xfffffe01077311e0)->td_proc
$6 = 0xfffff80002047000
(kgdb)
```


----------



## PMc (Nov 22, 2022)

_martin said:


> I have not debugged machine that doesn't have uart (or better to say connector or pins available). It's not a problem for client (machine where kgdb is running) usb-to-rs232 saves the day. Problem is when server (machine to be debugged) doesn't have it.


Ups, I didn't think that far. Indeed, having bootable USB doesn't mean the console could switch to an emulated serial. (This is something we would actually need to build!)


----------



## _martin (Nov 22, 2022)

Let I be forgiven for uploading pics. I have two odroids I can easily link together. I have bunch of ssd disks laying around so I can easily switch to whatever OS I like.
Was considering PXE boots as I once did but mentioned I'm lazy. 

With desktop MOBOs it should be still possible to find uart pins somewhere. Or go the firewire way. Not sure how I'd do that on notebook though.


----------

