# Marvell 88SE9230  passthrough fails for combination FreeBSD as Xen guest



## eiselekd (Jul 27, 2018)

I'm trying to setup FreeBSD (FreeNAS 11.2-beta1) inside a Xen guest. I want to us a 4 port Marvell 88SE9230 SATA controller PCI card in passthrough mode.
Host is a Ubuntu 18.04 Xen dom0. I have the IOMMU enabled, board is a GA-970A-DS3P.
The device seems to be present when booting FreeBSD when examining via lspci and pciconf.

However I have the strange behaviour that disks are not detected reliably. For instance I encountered this situation:

```
sysctl kern.disks
kern.disks: ada1 ada0 cd0 ada0
```
So it seems that name ada0 is detected 2 times. One instance of ada0 is the image based IDE root device. The other ada0 and ada1 are two
disks connected to the the Marvell controller.

I tried these combinations:
- Linux KVM as hypervisor with FreeNAS as Guest and 88SE9230 passthrough:  88SE9230 works with all connected disks detected
- Normal FreeNAS install : 88SE9230 works with all connected disks detected
- XEN-4.9 as hypervisor with Ubuntu 18.04 as Guest and 88SE9230 passthrough:  88SE9230 works with all connected disks detected
- XEN-4.9 as hypervisor with FreeNAS as Guest and 88SE9230 passthrough:  88SE9230 is passed through however only the last disk is detected or none.

So it seems that for the combination FreeBSD running inside Xen the 88SE9230 passthrough failes.

Any hists how to debug this?

dmsg:

```
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-STABLE #0 r325575+b31fa24ea5b(freenas/11-stable): Thu Jul  5 16:57:13 EDT 2018
    root@nemesis.tn.ixsystems.com:/freenas-11.2-releng/freenas/_BE/objs/freenas-11.2-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): text 80x25
XEN: Hypervisor version 4.9 detected.
CPU: AMD FX(tm)-8350 Eight-Core Processor            (4018.87-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x600f20  Family=0x15  Model=0x2  Stepping=0
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xbe983203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x2189f1<LAHF,CR8,ABM,SSE4A,MAS,Prefetch,XOP,LWP,FMA4,TBM>
  Structured Extended Features=0x8<BMI1>
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 8380219392 (7992 MB)
avail memory = 8067649536 (7693 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s)
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
SMP: AP CPU #1 Launched!
random: entropy device external interface
module_register_init: MOD_LOAD (vesa, 0xffffffff81040d10, 0) error 19
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc220-0xc22f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 24 at device 2.0 on pci0
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf30a0000-0xf30a0fff at device 3.0 on pci0
vgapci0: Boot video device
ahci0: <Marvell 88SE9230 AHCI SATA controller> port 0xc230-0xc237,0xc240-0xc243,0xc238-0xc23f,0xc244-0xc247,0xc200-0xc21f mem 0xf30a1000-0xf30a17ff irq 36 at device 5.0 on pci0
ahci0: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported
ahci0: quirks=0x900<NOBSYRES,ALTSIG>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich6: <AHCI channel> at channel 6 on ahci0
ahcich7: <AHCI channel> at channel 7 on ahci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xec800-0xeffff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
usb_needs_explore_all: no devclass
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 00:16:3e:a1:85:b1
xenbusb_back0: <Xen Backend Devices> on xenstore0
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xn0: backend features: feature-sg feature-gso-tcp4
xbd0: 30720MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd0: features: flush
xbd0: synchronize cache commands enabled.
ada0 at ahcich1 bus 0 scbus3 target 0 lun 0
cd0 at ata0 bus 0 scbus0 target 1 lun 0
ada0: <ST4000DM004-2CV104 0001> ACS-3 ATA SATA 3.x device
ada0: Serial Number ZFN0EJAG
ada0: 600.000MB/s transferscd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00002
cd0: 16.700MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
(WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
can't re-use a leaf (led)!
ada0: 3815447MB (7814037168 512 byte sectors)
ada0: quirks=0x1<4K>
pass2 at ahcich7 bus 0 scbus9 target 0 lun 0
pass2: <Marvell Console 1.01> Removable Processor SCSI device
pass2: Serial Number HKDP221516WL
pass2: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)
random: unblocking device.
Trying to mount root from zfs:freenas-boot/ROOT/default []...
GEOM: ada0: the secondary GPT table is corrupt or invalid.
GEOM: ada0: using the primary only -- recovery suggested.
g_dev_taste: make_dev_p() failed (gp->name=ada0, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0p1, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0p2, error=17)
SVM: not available.
module_register_init: MOD_LOAD (vmm, 0xffffffff82a24470, 0) error 6
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> K8/4/48/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA>
xn0: 2 link states coalesced
xn0: link state changed to UP
```
uname -a

```
FreeBSD freenas.8.8.8.8 11.1-STABLE FreeBSD 11.1-STABLE #0 r325575+b31fa24ea5b(freenas/11-stable): Thu Jul  5 16:57:13 EDT 2018     root@nemesis.tn.ixsystems.com:/freenas-11.2-releng/freenas/_BE/objs/freenas-11.2-releng/freenas/_BE/os/sys/FreeNAS.amd64  amd64
```
pciconf

```
ahci0@pci0:0:5:0:    class=0x010601 card=0x92301b4b chip=0x92301b4b rev=0x11 hdr=0x00
    vendor     = 'Marvell Technology Group Ltd.'
    device     = '88SE9230 PCIe SATA 6Gb/s Controller'
    class      = mass storage
    subclass   = SATA
    bar   [10] = type I/O Port, range 32, base 0xc230, size 8, enabled
    bar   [14] = type I/O Port, range 32, base 0xc240, size 4, enabled
    bar   [18] = type I/O Port, range 32, base 0xc238, size 8, enabled
    bar   [1c] = type I/O Port, range 32, base 0xc244, size 4, enabled
    bar   [20] = type I/O Port, range 32, base 0xc200, size 32, enabled
    bar   [24] = type Memory, range 32, base 0xf30a1000, size 2048, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message enabled with 1 message
    cap 10[70] = PCI-Express 2 legacy endpoint max data 128(512) RO NS
                 link x2(x2) speed 5.0(5.0) ASPM disabled(L0s/L1)
```
lspci

```
00:05.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
    Subsystem: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 36
    Region 0: I/O ports at c230
    Region 1: I/O ports at c240
    Region 2: I/O ports at c238
    Region 3: I/O ports at c244
    Region 4: I/O ports at c200
    Region 5: Memory at f30a1000 (32-bit, non-prefetchable)
    Expansion ROM at f3090000 [disabled]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Address: fee00000  Data: 0034
    Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn-
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v15] #ffff
    Capabilities: [ffc v15] #ffff
    Capabilities: [ffc v15] <chain looped>
```
/dev shows only ada0 which is the image based IDE device, no 88SE9230 attached disk seem to have being detected.

```
total 11
dr-xr-xr-x  12 root  wheel      512 Jul 27 12:22 .
drwxr-xr-x  20 root  wheel       28 Jul 27 12:23 ..
crw-r--r--   1 root  wheel     0x2a Jul 27 12:22 acpi
crw-r-----   1 root  operator  0x42 Jul 27 12:22 ada0
crw-r-----   1 root  operator  0x43 Jul 27 12:22 ada0p1
crw-r-----   1 root  operator  0x44 Jul 27 12:22 ada0p2
crw-r-----   1 root  operator  0x5f Jul 27 12:22 ada0p9
crw-rw-r--   1 root  operator  0x2c Jul 27 12:22 apm
crw-rw----   1 root  operator  0x2b Jul 27 12:22 apmctl
crw-------   1 root  wheel     0x2d Jul 27 12:22 atkbd0
crw-------   1 root  kmem       0xf Jul 27 12:22 audit
crw-------   1 root  wheel      0xe Jul 27 12:22 auditpipe
crw-------   1 root  wheel     0x16 Jul 27 12:22 bpf
lrwxr-xr-x   1 root  wheel        3 Jul 27 12:22 bpf0 -> bpf
crw-rw-rw-   1 root  wheel     0x30 Jul 27 12:22 bpsm0
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:22 cam
crw-r-----   1 root  operator  0x47 Jul 27 12:22 cd0
crw-------   1 root  wheel      0x7 Jul 27 12:24 console
crw-------   1 root  wheel     0x22 Jul 27 12:22 consolectl
crw-r-----   1 root  kmem      0x1e Jul 27 12:22 cpuctl0
crw-r-----   1 root  kmem      0x1f Jul 27 12:22 cpuctl1
crw-rw-rw-   1 root  wheel     0x3d Jul 27 12:22 crypto
crw-rw-rw-   1 root  wheel     0x10 Jul 27 12:22 ctty
crw-rw----   1 uucp  dialer    0x34 Jul 27 12:22 cuau0
crw-rw----   1 uucp  dialer    0x35 Jul 27 12:22 cuau0.init
crw-rw----   1 uucp  dialer    0x36 Jul 27 12:22 cuau0.lock
crw-------   1 root  wheel      0x8 Jul 27 12:22 devctl
crw-------   1 root  wheel      0x9 Jul 27 12:22 devctl2
cr--r--r--   1 root  wheel     0x41 Jul 27 12:22 devstat
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:23 dtrace
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:22 fd
crw-------   1 root  wheel     0x25 Jul 27 12:22 fido
crw-rw-rw-   1 root  wheel     0x11 Jul 27 12:22 full
crw-r-----   1 root  operator   0x6 Jul 27 12:22 geom.ctl
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:22 gpt
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:22 gptid
crw-------   1 root  wheel     0x20 Jul 27 12:22 io
crw-------   1 root  wheel     0x23 Jul 27 12:22 iscsi
lrwxr-xr-x   1 root  wheel        6 Jul 27 12:22 kbd0 -> atkbd0
lrwxr-xr-x   1 root  wheel        7 Jul 27 12:22 kbd1 -> kbdmux0
crw-------   1 root  wheel     0x27 Jul 27 12:22 kbdmux0
crw-------   1 root  wheel      0xd Jul 27 12:22 klog
crw-r-----   1 root  kmem       0xc Jul 27 12:22 kmem
crw-------   1 root  wheel      0xa Jul 27 12:22 mdctl
crw-r-----   1 root  kmem       0xb Jul 27 12:22 mem
crw-r-----   1 root  operator  0x58 Jul 27 12:22 mlx5ctl
crw-------   1 root  kmem      0x26 Jul 27 12:22 nfslock
crw-rw-rw-   1 root  wheel     0x12 Jul 27 14:55 null
crw-------   1 root  operator  0x48 Jul 27 12:22 pass0
crw-------   1 root  operator  0x49 Jul 27 12:22 pass1
crw-------   1 root  operator  0x4a Jul 27 12:22 pass2
crw-r--r--   1 root  wheel     0x14 Jul 27 12:22 pci
crw-rw-rw-   1 root  wheel     0x2f Jul 27 12:22 psm0
crw-rw-rw-   1 root  wheel     0x15 Jul 27 12:22 ptmx
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:32 pts
crw-r--r--   1 root  wheel      0x4 Jul 27 12:23 random
crwx------   1 root  wheel     0x57 Jul 27 12:22 rdma_cm
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:22 reroot
crw-------   1 root  wheel     0x21 Jul 27 12:22 snp
lrwxr-xr-x   1 root  wheel        4 Jul 27 12:22 stderr -> fd/2
lrwxr-xr-x   1 root  wheel        4 Jul 27 12:22 stdin -> fd/0
lrwxr-xr-x   1 root  wheel        4 Jul 27 12:22 stdout -> fd/1
crw-------   1 root  wheel     0x24 Jul 27 12:22 sysmouse
crw-------   1 root  wheel     0x31 Jul 27 12:22 ttyu0
crw-------   1 root  wheel     0x32 Jul 27 12:22 ttyu0.init
crw-------   1 root  wheel     0x33 Jul 27 12:22 ttyu0.lock
crw-------   1 root  wheel     0x4b Jul 27 12:30 ttyv0
crw-------   1 root  wheel     0x4c Jul 27 12:24 ttyv1
crw-------   1 root  wheel     0x4d Jul 27 12:24 ttyv2
crw-------   1 root  wheel     0x4e Jul 27 12:24 ttyv3
crw-------   1 root  wheel     0x4f Jul 27 12:24 ttyv4
crw-------   1 root  wheel     0x50 Jul 27 12:24 ttyv5
crw-------   1 root  wheel     0x51 Jul 27 12:24 ttyv6
crw-------   1 root  wheel     0x52 Jul 27 12:24 ttyv7
crw-------   1 root  wheel     0x53 Jul 27 12:22 ttyv8
crw-------   1 root  wheel     0x54 Jul 27 12:22 ttyv9
crw-------   1 root  wheel     0x55 Jul 27 12:22 ttyva
crw-------   1 root  wheel     0x56 Jul 27 12:22 ttyvb
crw-------   1 root  wheel     0x39 Jul 27 12:22 ufssuspend
lrwxr-xr-x   1 root  wheel        6 Jul 27 12:22 urandom -> random
crw-rw----   1 root  uucp      0x3e Jul 27 12:22 usbctl
dr-xr-xr-x   2 root  wheel      512 Jul 27 12:22 xen
crw-------   1 root  operator  0x3f Jul 27 12:22 xpt0
crw-rw-rw-   1 root  wheel     0x13 Jul 27 12:22 zero
crw-rw-rw-   1 root  operator  0x3b Jul 27 12:22 zfs
```


----------



## eiselekd (Jul 28, 2018)

Given the output of `sysctl kern.disks` I was able to find a workaround. Instead of allocating a image based IDE root device I switched to XEN bus type which gives me disks names

```
sysctl kern.disks
kern.disks: ada1 ada0 cd0 xdb0
```
instead which doenst clash.
Double-allocation of ada0 device is not so urgent for me anymore, but is still there...


----------

