# FreeBSD 11-STABLE VIRT Kernel under Linux Qemu with armkvm



## paulzhol (Jun 9, 2018)

Hi,

I'd like to document my successful attempt to run FreeBSD 11.2 virtualized on an Orange Pi Plus 2E (ARMv7 H3+, 2Gb RAM, 1Gb Ethernet, GICv2).
The reason for me using the VIRT target is incomplete hardware and driver support in 11.2 (the situation with 12-CURRENT might be better but I need to run on the upcoming 11.2).

My kernel configuration /sys/src/sys/arm/conf/VIRT-YVL:

```
include VIRT
ident VIRT-YVL

options ROOTDEVNAME=\"ufs:/dev/da0p2\"
options INCLUDE_CONFIG_FILE

options DDB
options BREAK_TO_DEBUGGER
options ALT_BREAK_TO_DEBUGGER

device  pci
device  pci_host_generic
options NEW_PCIB
options PCI_HP

device  scbus
device  da
device  pass

device  virtio_pci
device  virtio_scsi
device  virtio_balloon
device  virtio_console
device  virtio_random
```

The most important additions are `device  pci`, `device  pci_host_generic` and `device  virtio_pci`.
These are required to work around a qemu(?) bug with virtio-mmio networking described here,
by using `-device virtio-net-pci` instead of the default `-device virtio-net-device`.

Also attached to this post is a patch to enable working SMP under armkvm, without it `-smp 2` will fail to boot with the ULE scheduler panic
caused by a bad topology (because the second AP appears in the FDT but failes to activate) .
The patch enables virtual timer for arm >= 7 (similarly to aarch64), handles the `PSCI_RETVAL_ALREADY_ON` error in `virt_start_ap`
which is returned under armkvm (but not software-only emulation).
Additionally I had to disable `GIC_DEBUG_SPURIOUS` and increase the wait time in `check_ap`
(on my system it doesn't really take 30s for the AP to come up, but the 2000ms value used was too low).

The Linux host is a modified armbian with a 4.17.0 kernel compiled with KVM support `CONFIG_NO_HZ_FULL=y` and qemu-system-arm 2.12+dfsg-1+b1.
The UEFI payload being used is `QEMU_EFI_rel_16.02.fd` from Linaro.
Kernel cmdline to isolate cores 2,3 for virtualization only: `irqaffinity=0-1 isolcpus=2-3 nohz_full=2-3 rcu_nocbs=2-3`
qemu command line to run on cores 2,3 with RT priority:

```
chrt -r 1 taskset -c 2-3 qemu-system-arm \
        -enable-kvm -machine virt,accel=kvm,gic-version=2,kernel-irqchip=off \
        -cpu host -smp cores=2,threads=1,sockets=1 -m 1536 -nographic \
        -chardev tty,path=/dev/pts/3,id=pts3 -serial chardev:pts3 \
        -device virtio-scsi-pci,id=scsi \
        -drive if=none,file=./arm_freebsd_gpt.img,format=raw,cache=none,aio=native,id=hd0 \
        -device scsi-hd,drive=hd0,bus=scsi.0 \
        -netdev user,hostfwd=tcp::8022-:22,id=net0 -device virtio-net-pci,netdev=net0 \
        -object rng-random,filename=/dev/random,id=rng0 \
        -device virtio-rng-pci,max-bytes=1024,period=1000,rng=rng0 \
        -nodefaults -monitor stdio \
        -mem-prealloc -mem-path /dev/hugepages/arm-virt \
        -bios ./QEMU_EFI_rel_16.02.fd
```

`kernel-irqchip=off` seems mandatory, otherwise booting in SMP with 2 cores just spins at 100% cpu usage, single core does work.


----------

