# What is the best way to hack kernel?



## hamad_al_marri (Aug 11, 2020)

Hello everyone,
I have done some linux kernel hacking where the development process I used is as follow:

Install buildroot and make
Change some kernel code and compile it in host machine
run qemu with setting kernel path and buildroot directory with this command:


```
qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
-boot c -m 2049M -hda ../buildroot/output/images/rootfs.ext4 \
-append "root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr" \
-serial stdio -display none -smp $1
```



Now for FreeBSD, I am trying to do the same thing however what I am able to do so far:

I am running FreeBSD as host in my machine
I did some test changes, compile, and install the freebsd kernel on my host machine
I also installed FreeBSD in guest virtual machine and able to run it in qeme
What I am not able to do is to set the compiled kernel on my host to run on the guest freebsd.

I know already that FreeBSD is different than linux, you don't need to mention this here because all the search results show me a discussion on this topic instead of talking about an actual solution to this problem.

I need to know please what is the best way, best practice to develop the FreeBSD kernel. I need to create my own process scheduler on FreeBSD, I already did in linux it is not a matter of how to do your own scheduler, but how to debug or run your changes in a virtual machine for testing and development.

I need to know whether I can do with FreeBSD what I did with linux. If yes, how? If no, then what is the best way to do it in FreeBSD.
So far I have two options:

Either test the new kernel on my host, and restart my machine every time, which is not a good solution.
Or setup the guest FreeBSD to do the kernel development inside the guest (change code, compile, and install) all inside the guest machine. Which is not that good too but it is better than option 1
Please help.


----------



## SirDice (Aug 11, 2020)

Starting points: Handbook: Chapter 8. Configuring the FreeBSD Kernel, FreeBSD Developers' Handbook and development(7).


`make installkernel KERNCONF=MYKERNEL DESTDIR=/some/alternate/root`


```
DESTDIR               The directory hierarchy prefix where built objects
                           will be installed.  If not set, DESTDIR defaults to
                           the empty string.
```
See build(7)


----------



## a6h (Aug 11, 2020)

As a side note, get into the habit of working with DTrace: dtrace(1), Handbook Chapter 24. DTrace, FreeBSD Wiki: DTrace on FreeBSD
[EDIT] There're more learning benefits in troubleshooting a system with GENERIC kernel, than setting up a customised kernel.


----------



## hamad_al_marri (Aug 11, 2020)

Thank you guys for the help. Finally, I am able to do it by the following


```
# create the image
qemu-img create -f raw kernel_dev.img 8G

# install FreeBSD for the first time on the image
qemu-system-x86_64 -boot c -m 2049M -hda kernel_dev.img -cdrom FreeBSD-12.1-RELEASE-amd64-disc1.iso

# mount the image in /mnt/image, (md0p2) might be different in your machine
doas mdconfig -a -t vnode -f kernel_dev.img -u 0
doas mount /dev/md0p2 /mnt/image

# compile and install kernel
doas make buildkernel KERNCONF=MYKERNEL -j3
doas make installkernel KERNCONF=MYKERNEL DESTDIR=/mnt/image

# run qemu again
qemu-system-x86_64 -boot c -m 2049M -hda kernel_dev.img
```

I have another question, the make buildkernel keeps recompiling everything eventhough I changed one file
is there a way to keep make using old .o files?

Thank you


----------



## SirDice (Aug 11, 2020)

hamad_al_marri said:


> have another question, the make buildkernel keeps recompiling everything eventhough I changed one file
> is there a way to keep make using old .o files?


NO_CLEAN should do that. Just beware that it could lead to inconsistencies. 


```
NO_CLEAN               If set, no object tree files are cleaned at all.
                            This is the default when WITH_META_MODE is used
                            with filemon(4) loaded.  See src.conf(5) for more
                            details.  Setting NO_CLEAN implies NO_KERNELCLEAN,
                            so when NO_CLEAN is set no kernel objects are
                            cleaned either.
```
See build(7).


----------



## Mjölnir (Aug 11, 2020)

Install devel/ccache.  Better use bhyve(8) than emulators/qemu?  To test on bare metal, use ZFS boot environments (sysutils/beadm or bectl(8))


----------



## a6h (Aug 11, 2020)

mjollnir said:


> Install devel/ccache


ccache can make a difference.

Configuring devel/ccache to work with devel/poudriere:
`pkg install devel/ccache`
`mkdir -p /var/cache/ccache`

/usr/local/etc/poudriere.conf
`CCACHE_DIR=/var/cache/ccache`

Print the current statistics summary for the cache
`ccache -s`


----------



## ljboiler (Aug 11, 2020)

Myself, I've come to like the WITH_META_MODE method of incremental kernel and base building.
See src.conf(5)


----------



## didier (Oct 26, 2020)

mjollnir said:


> Install devel/ccache.  Better use bhyve(8) than emulators/qemu?  To test on bare metal, use ZFS boot environments (sysutils/beadm or bectl(8))


 bhyve : Nice !

May it be possible to connect guest with an usb device ?

Indeed, I wish debugging a USB wlan device's attachment.

Thank you again,


----------



## SirDice (Oct 26, 2020)

didier said:


> May it be possible to connect guest with an usb device ?


Soft of. You can pass-through PCIe devices: https://wiki.freebsd.org/bhyve/pci_passthru


----------

