# upgrade from 12.0 to 12.1 problems with pci passthrough



## Ofloo (Mar 7, 2020)

Did upgrade from 12.0 to 12.1p2

When i compiled kernel and installed it no issues. Even pci passthrough worked. Trouble only started after I ran installworld and mergemaster -iU (and ofcourse rebooted)

error:

```
Mar 07 10:16:51: initialising
Mar 07 10:16:51:  [loader: bhyveload]
Mar 07 10:16:51:  [cpu: 4]
Mar 07 10:16:51:  [memory: 2048M]
Mar 07 10:16:51:  [hostbridge: standard]
Mar 07 10:16:51:  [com ports: com1]
Mar 07 10:16:51:  [uuid: ae337462-f766-11e7-8028-ac1f6b45bb3c]
Mar 07 10:16:51:  [utctime: yes]
Mar 07 10:16:51:  [debug mode: no]
Mar 07 10:16:51:  [primary disk: disk0]
Mar 07 10:16:51:  [primary disk dev: sparse-zvol]
Mar 07 10:16:53: fatal; pci passthrough not supported on this system (no VT-d or amdvi)
```

dmesg:


```
---<<BOOT>>---
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p2 r357267 OFL amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 1024x768
CPU: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz (2200.08-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506f1  Family=0x6  Model=0x5f  Stepping=1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>
  Structured Extended Features3=0x2c000000<IBPB,STIBP,ARCH_CAP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x1<RDCL_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16557076480 (15790 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 3 2 1
Timecounter "TSC-low" frequency 1100038885 Hz quality 1000
random: entropy device external interface
000.000017 [4335] netmap_init               netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff811630f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd0 at kbdmux0
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256> on motherboard
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <processor> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> mem 0x7b7ff40000-0x7b7ff5ffff at device 9.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ppt0 mem 0xded00000-0xded7ffff,0xded8c000-0xded8ffff at device 0.0 on pci2
ppt1 mem 0xdec00000-0xdec7ffff,0xded88000-0xded8bfff at device 0.1 on pci2
ppt2 mem 0xdeb80000-0xdebfffff,0xded84000-0xded87fff at device 0.2 on pci2
ppt3 mem 0xdeb00000-0xdeb7ffff,0xded80000-0xded83fff at device 0.3 on pci2
pcib3: <ACPI PCI-PCI bridge> mem 0x7b7ff20000-0x7b7ff3ffff at device 16.0 on pci0
pci3: <ACPI PCI bus> on pcib3
nvme0: <Generic NVMe Device> mem 0xdee00000-0xdee03fff,0xdee04000-0xdee05fff at device 0.0 on pci3
pcib4: <ACPI PCI-PCI bridge> mem 0x7b7ff00000-0x7b7ff1ffff at device 17.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.0 on pci4
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> port 0xd000-0xd07f mem 0xdd000000-0xddffffff,0xde000000-0xde01ffff at device 0.0 on pci5
ahci0: <Intel Denverton AHCI SATA controller> port 0xe050-0xe057,0xe040-0xe043,0xe020-0xe03f mem 0xdf004000-0xdf005fff,0xdf007000-0xdf0070ff,0xdf006000-0xdf0067ff at device 20.0 on pci0
ahci0: AHCI v1.31 with 4 6Gbps ports, Port Multiplier not supported
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich6: <AHCI channel> at channel 6 on ahci0
ahciem0: <AHCI enclosure management bridge> at channel 2147483647 on ahci0
device_attach: ahciem0 attach returned 6
xhci0: <Intel Denverton USB 3.0 controller> mem 0x7b7ff60000-0x7b7ff6ffff at device 21.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pcib6: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci6: <ACPI PCI bus> on pcib6
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7b7fc00000-0x7b7fdfffff,0x7b7fe04000-0x7b7fe07fff at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: ac:1f:6b:45:bb:xx
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7b7fa00000-0x7b7fbfffff,0x7b7fe00000-0x7b7fe03fff at device 0.1 on pci6
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: ac:1f:6b:45:bb:xx
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
pcib7: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci7: <ACPI PCI bus> on pcib7
ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7b7f600000-0x7b7f7fffff,0x7b7f804000-0x7b7f807fff at device 0.0 on pci7
ix2: Using 2048 TX descriptors and 2048 RX descriptors
ix2: Using 4 RX queues 4 TX queues
ix2: Using MSI-X interrupts with 5 vectors
ix2: allocated for 4 queues
ix2: allocated for 4 rx queues
ix2: Ethernet address: ac:1f:6b:45:bb:xx
ix2: netmap queues/slots: TX 4/2048, RX 4/2048
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7b7f400000-0x7b7f5fffff,0x7b7f800000-0x7b7f803fff at device 0.1 on pci7
ix3: Using 2048 TX descriptors and 2048 RX descriptors
ix3: Using 4 RX queues 4 TX queues
ix3: Using MSI-X interrupts with 5 vectors
ix3: allocated for 4 queues
ix3: allocated for 4 rx queues
ix3: Ethernet address: ac:1f:6b:45:bb:xx
ix3: netmap queues/slots: TX 4/2048, RX 4/2048
pci0: <simple comms> at device 24.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
ichsmb0: <Intel Denverton SMBus controller> port 0xe000-0xe01f mem 0x7b7ff70000-0x7b7ff700ff at device 31.4 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: console (115200,n,8,1)
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat loadable, default to accept, logging disabled
DUMMYNET 0xfffff800040626c0 with IPv6 initialized (100409)
load_dn_sched dn_sched FQ_PIE loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
nvd0: <ADATA SX6000NP> NVMe namespace
nvd0: 244198MB (500118192 512 byte sectors)
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
ipmi0: IPMI device rev. 1, firmware rev. 3.60, version 2.0, device support mask 0xbf
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
ada0 at ahcich5 bus 0 scbus2 target 0 lun 0
ada0: <CT1000MX500SSD1 M3CR010> ACS-3 ATA SATA 3.x device
ada0: Serial Number 1806E10D2D36
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors)
ada1 at ahcich6 bus 0 scbus3 target 0 lun 0
ada1: <HGST HTS545050A7E380 GG2OACY0> ATA8-ACS SATA 2.x device
ada1: Serial Number 131201TM85A3PY22WBSM
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 476940MB (976773168 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 1> on usbus0
uhub1: MTT enabled
Root mount waiting for: usbus0
uhub1: 4 ports with 4 removable, self powered
ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
uhub2 on uhub1
uhub2: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 2> on usbus0
Root mount waiting for: usbus0
uhub2: 4 ports with 3 removable, self powered
ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
ukbd0 on uhub2
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
kbd1 at ukbd0
Root mount waiting for: usbus0
ugen0.5: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub3 on uhub1
uhub3: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 4> on usbus0
uhub3: MTT enabled
Root mount waiting for: usbus0
uhub3: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.6: <SanDisk Ultra Fit> at usbus0
umass0 on uhub0
umass0: <SanDisk Ultra Fit, class 0/0, rev 3.00/1.00, addr 5> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:4:0: Attached to scbus4
da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
da0: <SanDisk Ultra Fit 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C530001100414118101
da0: 400.000MB/s transfers
da0: 14663MB (30031250 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
GEOM_ELI: Device zvol/zroot/swap.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
lo0: link state changed to UP
ix0: link state changed to UP
ix1: link state changed to UP
ix2: link state changed to UP
ix3: link state changed to UP
ums0 on uhub2
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
```


```
acpidump -t DMAR
/*
  RSD PTR: OEM=SUPERM, ACPI_Rev=2.0x (2)
    XSDT=0x000000007e172098, length=36, cksum=143
*/
/*
  XSDT: Length=180, Revision=1, Checksum=178,
    OEMID=SUPERM, OEM Table ID=SMCI--MB, OEM Revision=0x1072009,
    Creator ID=AMI, Creator Revision=0x10013
    Entries={ 0x000000007e176b58, 0x000000007e176c70, 0x000000007e176cb8, 0x000000007e176d58, 0x000000007e176da0, 0x000000007e176de0, 0x000000007e176f90, 0x000000007e177008, 0x000000007e177038, 0x000000007e177070, 0x000000007e1770b8, 0x000000007e1789c0, 0x000000007e178a30, 0x000000007e178a68, 0x000000007e178b10, 0x000000007e178b40, 0x000000007e178d70, 0x000000007e178ec0 }
*/
/*
  FACP: Length=276, Revision=6, Checksum=154,
    OEMID=SUPERM, OEM Table ID=SMCI--MB, OEM Revision=0x1072009,
    Creator ID=AMI, Creator Revision=0x10013
     FACS=0x7e1be080, DSDT=0x0
    INT_MODEL=APIC
    Preferred_PM_Profile=Enterprise Server (4)
    SCI_INT=9
    SMI_CMD=0xb2, ACPI_ENABLE=0xa0, ACPI_DISABLE=0xa1, S4BIOS_REQ=0x0
    PSTATE_CNT=0x0
    PM1a_EVT_BLK=0x0-0x3
    PM1a_CNT_BLK=0x0-0x1
    PM_TMR_BLK=0x0-0x3
    P_LVL2_LAT=101 us, P_LVL3_LAT=1001 us
    FLUSH_SIZE=1024, FLUSH_STRIDE=16
    DUTY_OFFSET=1, DUTY_WIDTH=3
    DAY_ALRM=13, MON_ALRM=0, CENTURY=0
    IAPC_BOOT_ARCH={LEGACY_DEVICES}
    Flags={WBINVD,C1_SUPPORTED,SLEEP_BUTTON,S4_RTC_WAKE,RESET_REGISTER,PLATFORM_CLOCK,REMOTE_POWER_ON}
    RESET_REG=0xcf9:0[8] (IO), RESET_VALUE=0xe
*/
/*
  FACS:    Length=64, HwSig=0x00000060, Firm_Wake_Vec=0x00000000
    Global_Lock=
    Flags=
    Version=2
*/
acpidump: DSDT is corrupt
```

Did rollback to previous state and all worked again?


----------

