# 10.3-BETA 2 zroot doesn't boot with VT-d enabled and vmm in loader.conf



## dehrmann (Feb 25, 2016)

I installed FreeBSD 10.3-BETA2 on a system with a Z170 chipset and Skylake CPU on a ZFS root device. When I have both VT-d enabled in the bios and vmm (for bhyve) enabled in loader.conf, sometime during the boot, there's an unrecoverable error with my SATA hard drive, it's reattched, but the boot was interrupted, and the boot loader can't find the ZFS root. If either VT-d is disabled or vmm isn't enabled in loader.conf, the system boots fine.

Here's a screenshot. The interesting part is probably

```
CAM status: CCB request was invalid
Error 22. Unretryable error.
<disk> detached
Cannot find the pool label for 'zroot'
Mounting from zfs:zroot/ROOT/default failed with error 5.
```


----------



## tobik@ (Feb 25, 2016)

Maybe loading vmm from /etc/rc.conf (i.e. later) helps:

```
kld_list="vmm"
```


----------



## dehrmann (Feb 25, 2016)

It might (actually, it almost has to). I'll give it a try. It's probably The Right Thing® to do, anyway:

```
kld_list     (str) A list of kernel    modules    to load    right after the    local
         disks are mounted.  Loading modules at    this point in the boot
         process is much faster    than doing it via /boot/loader.conf
         for those modules not necessary for mounting local disk.
```

rc.conf(5).


----------



## dehrmann (Feb 25, 2016)

It booted. Now I need to see if bhyve still starts.

But still, specifying it in loader.conf shouldn't cause this problem.


----------



## dehrmann (Feb 26, 2016)

It works if I don't user PCI passthrough. Once I try PCI passthrough, `bhyve` exits with an exit code of 1.

The PCI passthrough docs say


> Set up vmm.ko to be preloaded at boot-time.
> 
> edit /boot/loader.conf and put in a line
> 
> ...



So maybe it really does need to be in /boot/loader.conf.


----------



## tobik@ (Feb 26, 2016)

Good morning,



dehrmann said:


> It works if I don't user PCI passthrough.


Ah  I guess if you load it later vmm.ko doesn't get the chance to attach to PCI devices before other drivers do.

Will you open a bug for your boot error on https://bugs.freebsd.org?


----------



## dehrmann (Feb 29, 2016)

It turns out removing pptdevs="..." also prevents the problem, but I'm not sure if that PCI device is the culprit, or if having it at all triggers some code path. I'll try a different device.

It looks like my problem is an interrupt storm that triggers a a PCI controller reset (and that causes failures with the SSD).


```
root monut waiting for: usbus1 usbus0
uhub1: 4 ports with 4 removable, self powered
uhub0: 26 ports with 24 removable, self powered
root monut waiting for: usbus0
pci_interrupt: host controller halted
pci_interrupt: host controller halted
...
```

I have the IRQ of the device, so maybe I'll be able to see which might be causing this.


----------



## tingo (Feb 29, 2016)

Now we have BETA3 ...


----------



## dehrmann (Feb 29, 2016)

Bug created. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207602

I took out everything interesting, so it's just failing on PCI passthrough for an onboard ethernet controller.

Guess I should look at the changelog and try BETA3.


----------



## dehrmann (Mar 6, 2016)

The issue still exists on RC1, and the root fs type doesn't matter; it also happens with a UFS root.


----------



## Christoph- (Mar 11, 2016)

I'm trying to passthrough a VGA device and i'm getting the same error as you, have you found a solution?


----------



## dehrmann (Mar 12, 2016)

Nope. Out of curiosity, what chipset are you seeing this with?


----------



## Christoph- (Mar 12, 2016)

Z170 with Mainboard ASRock Z170 Extreme4


----------

