# pci passthrough bhyve usb xhci



## Ofloo (Mar 21, 2018)

FreeBSD version 11-STABLE 

svn info

```
Path: .
Working Copy Root Path: /usr/src
URL: https://svn0.eu.freebsd.org/base/stable/11
Relative URL: ^/stable/11
Repository Root: https://svn0.eu.freebsd.org/base
Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revision: 331092
Node Kind: directory
Schedule: normal
Last Changed Author: eadler
Last Changed Rev: 331092
Last Changed Date: 2018-03-17 07:44:39 +0100 (Sat, 17 Mar 2018)
```


once started this shows on console:

```
(ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 ff 40 40 29 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: CCB request was invalid
(ada0:ahcich0:0:0:0): Error 22, Unretryable error
(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid
(aprobe0:ahcich0:0:0:0): Error 22, Unretryable error
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST500DM002-1SB10A CC43> s/n ZA4292G6 detached
(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid
(aprobe0:ahcich0:0:0:0): Error 22, Unretryable error
ahcich0: Poll timeout on slot 28 port 0
ahcich0: is 20000000 cs 10000000 ss 00000000 rs 10000000 tfd 1d0 serr 00000800 cmd 00005c17
(aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
(ada0:ahcich0:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: CCB request was invalid
(ada0:ahcich0:0:0:0): Error 22, Unretryable error
ahcich0: Poll timeout on slot 30 port 0
ahcich0: is 20000000 cs 40000000 ss 00000000 rs 40000000 tfd 1d0 serr 00000800 cmd 00005e17
(aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
(ada0:ahcich0:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40 00 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: CCB request was invalid
(ada0:ahcich0:0:0:0): Error 22, Unretryable error
(ada0:ahcich0:0:0:0): Periph destroyed
em0: Watchdog timeout Queue[0]-- resetting
Interface is RUNNING and ACTIVE
em0: TX Queue 0 ------
em0: hw tdh = 428, hw tdt = 471
em0: Tx Queue Status = -2147483648
em0: TX descriptors avail = 981
em0: Tx Descriptors avail failure = 0
em0: RX Queue 0 ------
em0: hw rdh = 912, hw rdt = 908
em0: RX discarded packets = 0
em0: RX Next to Check = 909
em0: RX Next to Refresh = 908
xl0: watchdog timeout
em0: Watchdog timeout Queue[0]-- resetting
Interface is RUNNING and ACTIVE
em0: TX Queue 0 ------
em0: hw tdh = 0, hw tdt = 58
em0: Tx Queue Status = -2147483648
em0: TX descriptors avail = 966
em0: Tx Descriptors avail failure = 0
em0: RX Queue 0 ------
em0: hw rdh = 0, hw rdt = 1023
em0: RX discarded packets = 0
em0: RX Next to Check = 0
em0: RX Next to Refresh = 1023
em0: Watchdog timeout Queue[0]-- resetting
Interface is RUNNING and ACTIVE
em0: TX Queue 0 ------
em0: hw tdh = 0, hw tdt = 58
em0: Tx Queue Status = -2147483648
em0: TX descriptors avail = 966
em0: Tx Descriptors avail failure = 0
em0: RX Queue 0 ------
em0: hw rdh = 0, hw rdt = 1023
em0: RX discarded packets = 0
em0: RX Next to Check = 0
em0: RX Next to Refresh = 1023
xl0: watchdog timeout
em0: Watchdog timeout Queue[0]-- resetting
Interface is RUNNING and ACTIVE
em0: TX Queue 0 ------
em0: hw tdh = 0, hw tdt = 54
em0: Tx Queue Status = -2147483648
em0: TX descriptors avail = 970
em0: Tx Descriptors avail failure = 0
em0: RX Queue 0 ------
em0: hw rdh = 0, hw rdt = 1023
em0: RX discarded packets = 0
em0: RX Next to Check = 0
em0: RX Next to Refresh = 1023
```
boot loader:

```
pptdevs="0/20/0"
```
vm-bhyve configuration:

```
passthru0="0/20/0"
```


----------



## SirDice (Mar 21, 2018)

It would probably help if you told us what the problem is. We're good but we're not clairvoyant.


----------



## Ofloo (Mar 21, 2018)

Sorry I thought that it was obvious, .. everything freezes as you can see the network driver starts acting up em0 and apparently storage driver as well, once the bhyve virtualisation starts.

Recompiled the kernel to not support the usb controller as suggested
https://github.com/pr1ntf/iohyve/wiki/USB-3.0-PCI-Controller-Pass-through

when I run vm passthru it says 0/20/0 is ready but when i start the vm it freezes and outputs the above on the console screen.


----------



## SirDice (Mar 21, 2018)

I'd start by investigating the disks issues. Because those don't look good, it looks like the disk is about to die. Also note that when you start the first VM the machine will appear to 'hang' for a second or so. This is normal. The errors you're seeing on the network cards appear to be caused by a watchdog, turn it off in the BIOS.


----------



## Ofloo (Mar 21, 2018)

This behaviour only occours when doing pci passthrough right after I run vm start vmmachine ofcourse only when passthrough is enabled.


----------



## SirDice (Mar 21, 2018)

What device is 0/20/0 supposed to be?


----------



## Ofloo (Mar 21, 2018)

Sunrise Point-H USB 3.0 xHCI Controller


----------



## Ofloo (Mar 23, 2018)

Tried uPD720202 USB 3.0 Host Controller  same result



```
# vm passthru
DEVICE     BHYVE ID     READY        DESCRIPTION
hostb0     0/0/0        No           Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers
pcib1      0/1/0        No           Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16)
vgapci0    0/2/0        No           HD Graphics 530
none0      0/20/0       No           Sunrise Point-H USB 3.0 xHCI Controller
none1      0/20/2       No           Sunrise Point-H Thermal subsystem
none2      0/22/0       No           Sunrise Point-H CSME HECI
ahci0      0/23/0       No           Sunrise Point-H SATA controller [AHCI mode]
pcib2      0/28/0       No           Sunrise Point-H PCI Express Root Port
pcib4      0/28/6       No           Sunrise Point-H PCI Express Root Port
isab0      0/31/0       No           Sunrise Point-H LPC Controller
none3      0/31/2       No           Sunrise Point-H PMC
hdac0      0/31/3       No           Sunrise Point-H HD Audio
none4      0/31/4       No           Sunrise Point-H SMBus
em0        0/31/6       No           Ethernet Connection (2) I219-V
pcib3      2/0/0        No           XIO2001 PCI Express-to-PCI Bridge
xl0        3/0/0        No           3c905B 100BaseTX [Cyclone]
ppt0       4/0/0        Yes          uPD720202 USB 3.0 Host Controller
```


```
ppt0@pci0:4:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
```


----------



## szabig (Apr 15, 2018)

I have the same issue as well - same error message, same trigger. If any additional information about my case helps the investigation, let me know and I will post it.
Thanks!


----------



## Ofloo (Apr 15, 2018)

I have been thinking about  it I also use an older pci card extra network card, is that the case for you as well?


----------



## szabig (Apr 15, 2018)

No, this is a brand new laptop, and I tried several PCI devices to check whether the problem is with the given device - but no good, all of my attempts at different devices gave the same result of error messages + freezing.


----------



## Ofloo (Apr 16, 2018)

Well it's a brand new desktop as well, i just had a need for a second network interface and i still had that one lying around, ..


----------



## Ordoban (May 1, 2018)

Hi, I have a similar issue. The system crashes if I try to use PCI passthrough. I have tested this with both versions 11.2 12.0, and different PCI cards (1:0:0, 8:0:0, 9:0:0). Allways the same result:

```
xhci0: Resetting controller
uhub0: at usbus0, port 1 addr 1 (disconnected)
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 02 86 94 10 00 00 00 00
(da0:umass-sim0:0:0:0): CAM status CCB request completed with an error
ugen0.2: <vendor 0x8087 product 0x0a2a> at usbus0 (disconnected)
(da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
ubt0: at uhub0, port 7, addr 1 (disconnected)
ubt0: detached
ugen0.3 <vendor 0x138a product 0x8050> at usbus0 (disconnected)
ugen0.4 <asmedia ASMT1053> at usbus0 (disconnected)
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 02 86 94 10 00 00 00 00
(da0:umass-sim0:0:0:0): CAM status CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 2 more tries remain
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 02 86 94 10 00 00 00 00
(da0:umass-sim0:0:0:0): CAM status CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 1 more tries remain
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 02 86 94 10 00 00 00 00
(da0:umass-sim0:0:0:0): CAM status CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command, 0 more tries remain
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 02 86 94 10 00 00 00 00
(da0:umass-sim0:0:0:0): CAM status CCB request completed with an error
(da0:umass-sim0:0:0:0): Error 5, Retries exhausted
(((5 more read errors)))
da0: at umass-sim0 bus 0 scbus 5 target 0 tun 0
da0: <ASMT 2105 0> s/n .... detached
(da0:umass-simd0:0:0:0): Periph destroyed 
(((that was the root filesystem)))
umass0: detached
(((endless repeat of the following messages)))
xhci0: Run timeout
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 24 ports with 24 removable, self powerd
xhci0: Resetting controller
usb_alloc_device: device init 2 failed (USB_ERR_TIMEOUT, ignored)
ugen0.2: <Unknown > at usbus0 (disconnected)
uhub_reattach_port: could not allocate new device
usb_alloc_device: device init 2 failed (USB_ERR_TIMEOUT, ignored)
ugen0.2: <Unknown > at usbus0 (disconnected)
uhub_reattach_port: could not allocate new device
usb_alloc_device: device init 2 failed (USB_ERR_TIMEOUT, ignored)
ugen0.2: <Unknown > at usbus0 (disconnected)
uhub_reattach_port: could not allocate new device
uhub0: at usbus0, port 1, addr 1 (disconnected)
```

Maybe helpful: startup messages

```
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r333017: Thu Apr 26 16:47:20 UTC 2018
    root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 1920x1080
CPU: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (2592.13-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16375443456 (15616 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HPQOEM SLIC-MPC>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
Timecounter "TSC-low" frequency 1296064035 Hz quality 1000
random: entropy device external interface
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810a8470, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
netmap: loaded module
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <HPQOEM SLIC-MPC> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
acpi_ec0: <Embedded Controller: GPE 0x50> port 0x62,0x66 on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0x5000-0x507f mem 0x95000000-0x95ffffff,0x80000000-0x8fffffff,0x90000000-0x91ffffff irq 16 at device 0.0 on pci1
vgapci1: <VGA-compatible display> port 0x6000-0x603f mem 0x93000000-0x93ffffff,0xa0000000-0xafffffff irq 16 at device 2.0 on pci0
vgapci1: Boot video device
xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0x96200000-0x9620ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
xhci_interrupt: host controller halted
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point AHCI SATA controller> port 0x6080-0x6087,0x6088-0x608b,0x6060-0x607f mem 0x96228000-0x96229fff,0x96230000-0x962300ff,0x9622e000-0x9622e7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 4 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x4000-0x40ff mem 0x96104000-0x96104fff,0x96100000-0x96103fff irq 16 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x54000000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci3: <ACPI PCI bus> on pcib3
ppt0 mem 0x96000000-0x96001fff irq 17 at device 0.0 on pci3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pcib4: [GIANT-LOCKED]
pci4: <ACPI PCI bus> on pcib4
pci4: <unknown> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
hdac0: <Intel Sunrise Point HDA Controller> mem 0x96220000-0x96223fff,0x96210000-0x9621ffff irq 16 at device 31.3 on pci0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
acpi_tz2: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model Generic PS/2 mouse, device ID 0
battery0: <ACPI Control Method Battery> on acpi0
acpi_acad0: <AC Adapter> on acpi0
acpi_lid0: <Control Method Lid Switch> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
hdacc0: <Realtek ALC290 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC290 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC290 (Analog 2.0+HP/2.0)> at nid 20,21 and 18 on hdaa0
pcm1: <Realtek ALC290 (Right Analog Mic)> at nid 26 on hdaa0
hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
cam_periph_alloc: attempt to re-allocate valid device pass1 rejected flags 0 refcount 3
passasync: Unable to attach new device due to status 0x6: CCB request was invalid
ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
cd0 at ahcich2 bus 0 scbus2 target 0 lun 0
cd0:
cd0:
cd0: 150.000MB/s transfers (SATA 1.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0:
ada0:
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0:
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1:
ada1:
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1:
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from zfs:zroot/ROOT/default []...
Root mount waiting for: usbus0
uhub0: 24 ports with 24 removable, self powered
ACPI Error: No handler for Region [RCM0] (0xfffff800039d9e80) [SystemCMOS] (20180313/evregion-288)
ACPI Error: Region SystemCMOS (ID=5) has no handler (20180313/exfldio-428)
ACPI Error: Method parse/execution failed \134_SB.WMID.ESDT, AE_NOT_EXIST (20180313/psparse-677)
ACPI Error: Method parse/execution failed \134_SB.PCI0.LPCB.EC0._Q42, AE_NOT_EXIST (20180313/psparse-677)
acpi_ec0: evaluation of query method _Q42 failed: AE_NOT_EXIST
Root mount waiting for: usbus0
ugen0.2: <vendor 0x8087 product 0x0a2a> at usbus0
ugen0.3: <vendor 0x138a product 0x0050> at usbus0
Root mount waiting for: usbus0
ugen0.4: <asmedia ASMT1053> at usbus0
umass0 on uhub0
umass0: <asmedia ASMT1053, class 0/0, rev 3.00/1.00, addr 3> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:5:0: Attached to scbus5
da0 at umass-sim0 bus 0 scbus5 target 0 lun 0
da0: <ASMT 2105 0> Fixed Direct Access SPC-4 SCSI device
da0:
da0: 400.000MB/s transfers
da0: 238475MB (488397168 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
Solaris: NOTICE: Cannot find the pool label for 'zroot'
Mounting from zfs:zroot/ROOT/default failed with error 5; retrying for 3 more seconds
re0: link state changed to DOWN
re0: link state changed to UP
ubt0 on uhub0
ubt0: <vendor 0x8087 product 0x0a2a, class 224/1, rev 2.01/0.01, addr 1> on usbus0
WARNING: attempt to domain_add(bluetooth) after domainfinalize()
WARNING: attempt to domain_add(netgraph) after domainfinalize()
ubt0: ubt_ctrl_write_callback:782: control transfer failed: USB_ERR_TIMEOUT
ng_hci_process_command_timeout: ubt0hci - unable to complete HCI command OGF=0x3, OCF=0x3. Timeout
```

pciconf -a -v

```
hostb0@pci0:0:0:0:    class=0x060000 card=0x80ee103c chip=0x19108086 rev=0x07 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers'
    class      = bridge
    subclass   = HOST-PCI
pcib1@pci0:0:1:0:    class=0x060400 card=0x80ee103c chip=0x19018086 rev=0x07 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16)'
    class      = bridge
    subclass   = PCI-PCI
vgapci1@pci0:0:2:0:    class=0x030000 card=0x80ee103c chip=0x191b8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'HD Graphics 530'
    class      = display
    subclass   = VGA
xhci0@pci0:0:20:0:    class=0x0c0330 card=0x80ee103c chip=0xa12f8086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
none0@pci0:0:20:2:    class=0x118000 card=0x80ee103c chip=0xa1318086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H Thermal subsystem'
    class      = dasp
none1@pci0:0:21:0:    class=0x118000 card=0x80ee103c chip=0xa1608086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H Serial IO I2C Controller'
    class      = dasp
none2@pci0:0:21:1:    class=0x118000 card=0x80ee103c chip=0xa1618086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H Serial IO I2C Controller'
    class      = dasp
none3@pci0:0:22:0:    class=0x078000 card=0x80ee103c chip=0xa13a8086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H CSME HECI'
    class      = simple comms
ahci0@pci0:0:23:0:    class=0x010601 card=0x80ee103c chip=0xa1038086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
pcib2@pci0:0:28:0:    class=0x060400 card=0x80ee103c chip=0xa1148086 rev=0xf1 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:5:    class=0x060400 card=0x80ee103c chip=0xa1158086 rev=0xf1 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:28:6:    class=0x060400 card=0x80ee103c chip=0xa1168086 rev=0xf1 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:    class=0x060100 card=0x80ee103c chip=0xa14e8086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
none4@pci0:0:31:2:    class=0x058000 card=0x80ee103c chip=0xa1218086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H PMC'
    class      = memory
hdac0@pci0:0:31:3:    class=0x040300 card=0x80ee103c chip=0xa1708086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H HD Audio'
    class      = multimedia
    subclass   = HDA
none5@pci0:0:31:4:    class=0x0c0500 card=0x80ee103c chip=0xa1238086 rev=0x31 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sunrise Point-H SMBus'
    class      = serial bus
    subclass   = SMBus
vgapci0@pci0:1:0:0:    class=0x030200 card=0x80ee103c chip=0x139a10de rev=0xa2 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GM107M [GeForce GTX 950M]'
    class      = display
    subclass   = 3D
re0@pci0:7:0:0:    class=0x020000 card=0x80ee103c chip=0x816810ec rev=0x15 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
ppt0@pci0:8:0:0:    class=0x028000 card=0x50108086 chip=0x095a8086 rev=0x61 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Wireless 7265'
    class      = network
none6@pci0:9:0:0:    class=0xff0000 card=0x80ee103c chip=0x522a10ec rev=0x01 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTS522A PCI Express Card Reader'
```

bhyve startup command

```
bhyve -c 1 \
-s 7,fbuf,tcp=0.0.0.0:5900 \
-s 20,xhci,tablet \
-s 0,hostbridge \
-s 3,ahci-hd,/root/windisk.bin \
-s 31,lpc -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-s 8,passthru,8/0/0 \
-m 4G -H -w -P -S windows7
```


----------



## grehan@ (May 2, 2018)

I think this is a bug in PCI passthru where a guest is attempting to enable legacy interrupts on a device (particularly Win7, not much MSI device support there). The legacy interrupt is still enabled on the device itself, but conflicts with other devices, resulting in false interrupts being delivered on the host.

The fix for bhyve would be to squash these interrupts, though that won't help a guest that is expecting to be able to use legacy interrupts on a device (only MSI/MSIx can be used).


----------



## Ordoban (May 2, 2018)

grehan@ said:


> I think this is a bug in PCI passthru where a guest is attempting to enable legacy interrupts on a device (particularly Win7, not much MSI device support there).



/root/windisk.bin was empty (filled with zero) in the case of my test with 12.0, so we can not blame Windows 7 for this. Does the UEFI bootrom also touch the legacy interupts?


----------



## grehan@ (May 2, 2018)

UEFI only uses polled-mode for it's device drivers, and shouldn't be touching stuff like XHCI (the drivers for that aren't built int). Still a mystery


----------



## Ordoban (May 4, 2018)

I have done a little debugging, and found out:
If I place an endless loop into *iommu.c --> iommu_init* before the IOMMU_ENABLE macro, bhyve get stuck but the system stay alife. If I put the loop after, the system crash like described above.
So the failure must happens somewhere in *vtd.c --> vtd_init*. I don't understand all of the vtd_init, but I see some interactions with the ACPI tables, and I know the ACPI is not correct on this laptop.
Seems I have to learn something about ACPI to get PCI passthrough working on this machine.


----------



## Ofloo (May 9, 2018)

Is there a way to boot Win7 in bhyve without uefi?


----------



## CallumA (Jul 10, 2018)

I believe I'm affected by this same issue on 11.2-RELEASE. It's 100% not UEFI emulation since I can reproduce the same issue without UEFI (FreeBSD guest).

As soon as `bhyve` is run, with the passthrough device `-s n:0,passthru,x/y/z`, it appears the PCIe bus just completely screws up and nothing works anymore (disks hang, NICs hang, USB hangs, etc.)

Something that's interesting is Ofloo, Ordoban and I all seem to be running on "Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor". In my case this is an E3-1275 v5 (on a C236 chipset motherboard). I'd hazard a guess this may be an annoying hardware-specific issue.

Unfortunately passthrough has become quite important to my application, and VT-d is just magic to me so I wouldn't know where to start debugging this!


----------



## Ordoban (Jul 10, 2018)

CallumA said:


> Something that's interesting is Ofloo, Ordoban and I all seem to be running on "Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor".


Wrong! Mine is an i7-6700HQ, but you are also right! Seems we share the "Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor *Host Bridge/DRAM Registers*" and "Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor *PCIe Controller (x16)*".
I have already done some debugging, but give up. All I can say is: 
- the vtd_init function does some re-organizing on the memory mappings with the help of one ACPI table, but does something wrong.


----------



## CallumA (Jul 10, 2018)

Well, I've spent a long day trying to work out what's going on but can confirm I'm well out of my depth. What I can say is that the issue arrises after the second (out of two) `vtd_translation_enable(vtdmap);` (`vtdmap = vtdmaps[1]`) is executed in `vtd_enable()`. The system remains stable if this is conditionally executed for i < 1 (leaving only the first vtdmap enabled) but booting the VM then does not work.

Why the second enable breaks things I don't know. Perhaps all other devices aren't properly added to the host domain before enabling translation, although I have confirmed `iommu_add_device(host_domain, pci_get_rid(dev));` is executed on all non-passthrough devices perhaps something prevents that from working on this model. I would hazard a guess that, as you say, the issue exists in `vtd_init`.

As I say, I'm well out of my depth here but any suggestions (known differences in 6th Gen processors perhaps) from someone who knows better would definitely be welcome.


----------



## Ordoban (Jul 13, 2018)

Got it working! The VM runs and have access to the pci device.

Seems our chipset have 2 VTd units. FreeBSD uses the first (or both?), but only the second works.
I disabled the first one by inserting

```
vtdmaps[0] = vtdmaps[1];
units = 1;
```
into the vtd.c line 299 (vtd_init after the DMAR copy loop)

Just to lower our mood:
The VM run is a one-shoot. There is a 
	
	



```
vm_reinit: Device busy
```
 error on second VM start. Need to restart the host too...


----------



## CallumA (Jul 15, 2018)

Mate, you're a legend!

That led me to discover that, for devices on the second VTd unit, the translation context entry isn't configured properly. On our models each VTd remapping unit has different capabilities, the problem here being DT in the Extended Capabilities Register which declares whether or not device-IOTLBs are supported. Unit 0 supports this but unit 1 doesn't. Because, when a device is mapped, the capabilities of _only the first_ VTd unit is checked it is assumed that the second unit also supports device-IOTLBs (which it doesn't) and the wrong translation type is set for device contexts attached to the second unit. According to the spec it's actually the last unit which is used as a sort of default for PCI devices not attached to other units so this is actually the opposite to the "best guess" - although neither are the ideal solution.

A quick fix is to replace 
	
	



```
if (VTD_ECAP_DI(vtdmap->ext_cap))
```
 with 
	
	



```
if (0)
```
 on line 392 of vtd.c.

Technically this isn't to spec for the first translation unit, but it does work and doesn't seem to cause issues.

I've written a patch to properly correct this on a per-device basis, rather than the above bodge, as well as fix some other oversights in the implementation of capabilities for multiple VTd units. I'll submit a bug report and patch for this in the morning (well, later in the day given it's 05:30 now ).


----------



## Ordoban (Jul 17, 2018)

Very cool to know the real reason. Your quick fix works on my machine too.

The `vm_reinit: Device busy` issue only appears if I not shutdown the VM properly (Strg+C'ed bhyve). I think this is a issue for the Bhyve developer.

Could you hint me when you post your bug report and patch?


----------



## CallumA (Jul 18, 2018)

Ordoban said:


> Could you hint me when you post your bug report and patch?



Bug reported: PR 229852
Full patch available at: https://bugs.freebsd.org/bugzilla/attachment.cgi?id=195225


----------



## Siffredinho (Jul 29, 2018)

I just wanted to say "thank you". I had this problem with 3 different boards and wanted to use PCI Passthrough with a NIC which always ended in a panic situation (doesn't matter what Interface I entered in pptdevs).

This patch also works properly with an actual FreeBSD 12.0-CURRENT.  The actual CPU I use is a i7-7700. I had the issue also with older generations.


----------



## Ofloo (Aug 4, 2018)

I've applied the patch to release 11.2 and everything works now. I'll let you know if I got any stability issues.


----------



## Botanic (Aug 5, 2018)

I have a problem with PCI passthrough with OpenBSD and NetBSD guests. FreeBSD and Linux guests work without a problem.
My CPU is a E3-1220 v5 with the C236 chipset, just as reported by others in this thread.
I have the same problem running bhyve with both 11.1-Release and 11.2-Release on the host (hypervisor).

The NIC that I do passthrough with, will not work and dmesg shows this:

```
wm0 at pci0 dev 5 function 0: 82576 quad-1000BaseT Ethernet (rev. 0x01)
pci_intr_map: no mapping for pin B (line=ff)
wm0: unable to map interrupt
bge0 at pci0 dev 6 function 0: Broadcom BCM5720 Gigabit Ethernet
pci_intr_map: no mapping for pin B (line=ff)
bge0: couldn't map interrupt
```

Note that I have tried with two different NICs.
The hardware works fine booting OpenBSD without bhyve.

Is this a related or the same problem as discussed in this thread?
The server is a production server so I would not want to divert from Release.


----------



## drhowarddrfine (Aug 5, 2018)

Botanic said:


> FreeBSD and Linux guests work without a problem.





Botanic said:


> I have the same problem with both 11.1-Release and 11.2-Release.


Contradiction


----------



## Botanic (Aug 5, 2018)

drhowarddrfine said:


> Contradiction


No it's not. Maybe I was a bit unclear so I edited my post to clarify.

I discovered the problem running 11.1-Release on my server, running bhyve with a FreeBSD and a OpenBSD guest.
PCI passthrough on the virtual FreeBSD worked. PCI passthrough on the virtual OpenBSD did not work.
I have since upgraded to 11.2-Release on the hardware and tried with a virtual CentOS and virtual NetBSD.
CentOS works. NetBSD does not work.


----------



## Ofloo (Sep 26, 2018)

Up until today still running without issues releng-11.2


----------



## CallumA (Oct 23, 2018)

Botanic said:


> Is this a related or the same problem as discussed in this thread?



Potentially, but the symptoms seem different so I wouldn't get your hopes up. You can try using the patch from:


CallumA said:


> Bug reported: PR 229852
> Full patch available at: https://bugs.freebsd.org/bugzilla/attachment.cgi?id=195225



Simple procedure for building and installing patched vmm module:
`cd /usr/src/sys/amd64/vmm/intel
fetch "https://bugs.freebsd.org/bugzilla/attachment.cgi?id=195225" -o vmm.patch
patch < ./vmm.patch
cd /usr/src/sys/modules/vmm
make
make install clean
mv /boot/kernel/vmm.ko /boot/kernel/vmm_unpatched.ko`


----------



## Ofloo (Jan 27, 2019)

Up until today haven't had one crash and everything functions fine. Well it only crashed once but that was due to the fact that i accidentally had passed through the PCI device to 2 machines.


----------



## chk.jxcn (Oct 24, 2019)

Try to upgrade from 11.0-RELEASE to 12.0-RELEASE to resolve the same issue, unlucky until apply this patch.
My CPU is 3215U and it make pfsense runing faster on bhyve.
Appreciate!!!


----------



## free-and-bsd (Feb 26, 2020)

The patch seems to be in STABLE now. Thank you all


----------



## free-and-bsd (Feb 28, 2020)

OK, so far haven't applied the patch, the results are different but still negative: system doesn't freeze, but bhyve (Win 10 guest) crashes shortly after start. Will see now if the patch helps that issue.
I'm also trying the xhci passthrough following the same idea. So, pptdevs work fine, `pciconf -vl | grep ppt`:

```
ppt0@pci0:6:0:0:        class=0x0c0330 card=0x34831106 chip=0x34831106 rev=0x01 hdr=0x00
ppt1@pci0:7:0:0:        class=0x020000 card=0x012310ec chip=0x816810ec rev=0x0c hdr=0x00
```
But bhyve starts, I get connected showing Win 10 booting for a short while, then crashes:

```
Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
```


----------



## free-and-bsd (Feb 28, 2020)

free-and-bsd said:


> OK, so far haven't applied the patch, the results are different but still negative: system doesn't freeze, but bhyve (Win 10 guest) crashes shortly after start. Will see now if the patch helps that issue.
> I'm also trying the xhci passthrough following the same idea. So, pptdevs work fine, `pciconf -vl | grep ppt`:
> 
> ```
> ...


EDIT: Oops, this seems to be a different issue. This happens to a card with multiple bars, it seems.


----------



## free-and-bsd (Feb 28, 2020)

free-and-bsd said:


> EDIT: Oops, this seems to be a different issue. This happens to a card with multiple bars, it seems.


Yes, this is a different issue, unrelated to the one with freezing -- that one I evidently don't have.
But this one is bhyve shutting down when passing through a PCI card with multiple bars (read the link above). The patch provided in that thread at freebsd-virtualization mailing list _works fin_e, but has not yet been committed.
I'm not sure if this should be put in a separate thread, but it looks like worth it.


----------

