# Bhyve GPU pass-through



## Phishfry (May 12, 2019)

I saw on Reddit via Twitter that a user has GPU passthrough working with Bhyve.

So he is claiming that you can pass-through a Nvidia video card and assign it to an X Windows session using the nvidia-config application.
You must add the video device to your VM's Xorg config beforehand. Supposedly one line to add.
I have been saving old monitors for just this purpose. I want 4 headed VM each with their own display. Yipee.
Now we are getting closer to Xen functionality.

That way sounds like it might just work. Has anybody tried anything similar?


----------



## shkhln (May 12, 2019)

Sounds kinda weird, to be honest. This should have been tried already.


----------



## Phishfry (May 13, 2019)

Well I have a bhyve testing machine up checking this out. Using GeForce GTX650 half height card.
So far so good. I loaded Nvidia driver 390 and linux compat. on a FreeBSD-12 VM with Xorg and Xfce
So I am configuring it now.
I passed through the device and its child. Wrote a nvidia.conf with BusID included.

```
vgapci0@pci0:0:5:0:    class=0x030000 card=0x129119da chip=0x0fc610de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 [GeForce GTX 650]'
    class      = display
    subclass   = VGA
hdac0@pci0:0:5:1:    class=0x040300 card=0x129119da chip=0x0e1b10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 HDMI Audio Controller'
    class      = multimedia
    subclass   = HDA
```

I might have to ditch tmux to test this out.
Currently I start all my VM's from a rc.local script with a tmux session.


----------



## Phishfry (May 13, 2019)

Well I have to say, I have one VM with video passthru working.
I used a single EFI bhyve VM and I have xfce desktop over stdio on a HDMI monitor coming from my GTX650.
Now, will that same setup work for multiple video cards passed through.

I did nothing special. Normal nvidia driver and didn't use nvidia-xconfig or nvidia-settings. Just my own conf with BusID.
/usr/local/etc/X11/xorg.conf.d/20-nvidia.conf

```
Identifier      "Nvidia Card"
        Driver          "nvidia"
        VendorName      "NVIDIA Corporation"
        BoardName       "GeForce GTX 650"
        BusID           "PCI:0:5:0"
EndSection

Section "Screen"
        Identifier     "Screen0"
        Device         "Device0"
        Monitor        "Monitor0"
        Option         "metamodes" "nvidia-auto-select +0+0 {ForceCompositionPipeline=On, ForceFullCompositionPipeline=On}"
        Option         "AllowIndirectGLXProtocol" "off"
        Option         "TripleBuffer" "on"
EndSection
```

I have no mouse or keyboard control on the HDMI screen. I might need to add to /x11/xorg.conf.d/  file for input devices.


----------



## D-FENS (May 13, 2019)

This is really cool, could you share your bhyve arguments please? How do you do the passthrough?


----------



## Phishfry (May 13, 2019)

Here is my /etc/rc.local I use to start up my VM. Device 5:0 and 5:1 are my video card.

```
# cat /etc/rc.local
#!/bin/sh
bhyve -S -c 4 -m 3G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 3:0,passthru,2/0/0 -s 5:0,passthru,5/0/0 -s 5:1,passthru,5/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1 &
```
Notice with the ampersand it sends the process to the backround.
With no ampersand it boots to the VM and I can run `startxfce4`.
Host is unreachable when ran this way. This way is hacky but it does work. Probably a better way.

Still working on mouse input. Isn't that what xhci,tablet does?


----------



## Phishfry (May 13, 2019)

For the mouse- xhci,tablet pass-thru ends up as /dev/ums0 on the VM. I tried assigning that a xorg input with no luck.
This was more a vanity exercise. No mouse or keyboard all but makes this useless. A nice picture on a HDMI screen.
Wondering about passing through a USB mouse/keyboard for each desktop.


----------



## D-FENS (May 13, 2019)

I have not done a graphical VM so far. But in this post you can see what arguments I use: https://forums.freebsd.org/threads/...m-bhyve-uefi-bhyve_uefi-fd.69616/#post-421438


----------



## hukadan (May 13, 2019)

Very interesting indeed. While digging the Internet for more information on the subject, I came across this tweet : 



__ https://twitter.com/i/web/status/1127136891365658625He said he would post further details before Thursday (the tweet is from May, 11).


----------



## Phishfry (May 16, 2019)

I was able to get two NVidia video cards working with two monitors and two seperate VM's.
The hangup is input. No mouse or keyboard.
For testing setup I bought some USB3.0 PCIe cards and plan on passing the card thru. NEC USB3 chipset.
Curious to see how USB disks work and mouse and kb in a VM with their own dedicated host device.


----------



## Phishfry (May 20, 2019)

I am happy to say I have passed thru the USB3 controller and I have a ums0 mouse working on a VM's Xorg hdmi screen.
Bought 2 USB3 cards so checking on the second one now in a second VM.


----------



## Phishfry (May 20, 2019)

Awe shucks that won't work. When I mentioned above I have 2 video cards on 2 VM's, notice I did not use the word 'simultaneous'.
The only way I can get this to work is launching bhyve --in the foreground-- and with that method I can launch Xorg in for the foreground VM mapped onto a dedicated HDMI monitor..

To do this for testing maybe I run VM in the backround and do autologin and startxfce script to bring it up kiosk style.
Sure is hacky. I could do multiple Xorg VM's to multiple monitors that way believe.
I also have a virtio-console method in mind to test.


----------



## aragats (May 20, 2019)

Phishfry said:


> I am happy to say I have passed thru the USB3 controller and I have a ums0 mouse working on a VM's Xorg hdmi screen.


That's really cool! I never had luck to pass through a USB controller, always got errors with various controllers and PCs (reported a couple of times on these forums). I don't think I tried with 12.0 though.


----------



## Phishfry (May 20, 2019)

Got it working now with independent mouse and keyboard and monitor in the background.
https://forums.freebsd.org/threads/autologin-and-auto-run-x-server.6782/#post-184736
Made a user: xfce with no password.


----------



## Phishfry (May 20, 2019)

I was only testing with mouse. So I opened mousepad text editor and no keyboard, so I hotplugged a keyboard in my passthru USB3 controller and I had text in mousepad.
Mission complete on VM1

Time for the real test.


----------



## Phishfry (May 20, 2019)

I am pretty sure that any supported PCIe USB controller would work.
This is what I used. A $7 Dell adapter off ebay:
https://www.ebay.com/itm/192844099338
shows up as xhci0 before passthru


----------



## Phishfry (May 20, 2019)

I think the problem with using onboard USB is you would lose the xhci controller to the host system.
I need that (mouse/kybd) for host management.
The onboard USB controllers all hang off of one pci address.


----------



## Ordoban (May 21, 2019)

Do you know sysutils/synergy ?


----------



## Phishfry (May 22, 2019)

I have not tried that port yet but it sounds like something that might work.
Right now I like using a separate USB3 controller for each Xorg VM.
Problem is I only have 7 slot motherboard and I am running out of slots.
Trying to use 2 NVMe, Chelsio 10G, and 2 video cards and 2 USB cards. Maxed out at 2 VM's.
I could bifurcate my NVMe as a last resort.
Noticed that my onboard USB uses two PCI addresses so I need to suss that out.
It appears that the USB pin headers are on a separate PCI bus. So I can use those.

Does anybody know of any Video cards with USB inputs?
Hardware virtualization cards that work in FreeBSD?
Here is what I have seen on ebay;
TERADICI PCoIP cards:








						Teradici by Dell PCoIP Remote Workstation Host Card TERA2220 Dual Display for sale online | eBay
					

Find many great new & used options and get the best deals for Teradici by Dell PCoIP Remote Workstation Host Card TERA2220 Dual Display at the best online prices at eBay! Free shipping for many products!



					www.ebay.com
				



I doubt these would work but I like the idea. Multiple functions per slot.


----------



## aragats (May 22, 2019)

Just tested a USB-over-Ethernet adapter BB-UE204 with Windows bhyve guest as a keyboard and mouse interface. Works fine. The drawback is its high price ($200+). I got one at work place, but wouldn't buy if had to pay own money.


----------



## Phishfry (May 22, 2019)

Ahh yes. BlackBox makes adapter for everything (at a steep price).

What I was just wondering is 'does PCIe contain USB natively'.
I bought a 90 degree PCIe riser from an IBM box and it had a USB jack on it.
The thing I worry about with that approach (like USB with video card) is that it would probably not use separate PCI bus ID.
I dunno. All new to me.
EDIT: I see nothing in PCIe slot signaling for USB. Only miniPCIe adds USB2 for cellular modems and their sims.
Probably a special IBM X36xx riser for certain slot in that server.


----------



## Crivens (May 22, 2019)

hukadan said:


> He said he would post further details before Thursday (the tweet is from May, 11).


So I'd wager he got it working and now is deep in some red dead redemption session


----------



## Phishfry (May 22, 2019)

Has been crickets from that guy..

As far as I can tell this works with FreeBSD no special sauce needed. Just make a xorg.conf with the BusID.
Surprised no one has tried it before.
I am now launching my two Xorg VM's with tmux. So no issues there. I though maybe tmux would be an issue.
I am copying in some MPG videos now to test HDMI audio passthru to separate monitors.


----------



## Phishfry (May 22, 2019)

No HDMI audio without any work.
I did test USB Memory sticks and found that device discovery happens quick as normal, but the CAM support lags by a minute.
No device in /dev until CAM.


----------



## shkhln (May 22, 2019)

Does OpenGL actually work? And after multiple VM reboots?



Crivens said:


> So I'd wager he got it working and now is deep in some red dead redemption session



That game is kind of infamous for being console exclusive. (Am I ruining the joke?)


----------



## Crivens (May 23, 2019)

shkhln said:


> That game is kind of infamous for being console exclusive. (Am I ruining the joke?)


No, you just expose my complete ignorance of the gaming subuniverse from the last 10 years.


----------



## Phishfry (May 24, 2019)

shkhln said:


> Does OpenGL actually work? And after multiple VM reboots?


`glxgears -info`

```
303 frames in 5.0 seconds = 60.520 FPS
297 frames in 5.0 seconds = 59.379 FPS
301 frames in 5.0 seconds = 60.002 FPS
300 frames in 5.0 seconds = 59.828 FPS
300 frames in 5.0 seconds = 59.978 FPS
301 frames in 5.0 seconds = 60.002 FPS
```


----------



## Phishfry (May 24, 2019)

`glxinfo |grep render`

```
direct rendering: Yes
OpenGL renderer string: GeForce GT 730/PCIe/SSE2
    GL_ARB_conditional_render_inverted, GL_ARB_conservative_depth,
    GL_KTX_buffer_region, GL_NVX_conditional_render, GL_NVX_gpu_memory_info,
    GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_pixel_data_range, GL_NV_point_sprite,
    GL_ARB_compute_variable_group_size, GL_ARB_conditional_render_inverted,
    GL_KTX_buffer_region, GL_NVX_conditional_render, GL_NVX_gpu_memory_info,
    GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_pixel_data_range, GL_NV_point_sprite,
    GL_EXT_multisample_compatibility, GL_EXT_multisampled_render_to_texture,
    GL_EXT_multisampled_render_to_texture2, GL_EXT_occlusion_query_boolean,
    GL_EXT_render_snorm, GL_EXT_robustness, GL_EXT_sRGB,
    GL_NV_blend_equation_advanced, GL_NV_conditional_render,
    GL_NV_packed_float_linear, GL_NV_path_rendering,
    GL_OES_fbo_render_mipmap, GL_OES_geometry_point_size,
```


----------



## Phishfry (May 24, 2019)

Just for real world experience I installed OpenArena. It behaves quite regularly. No issues at all.


----------



## Phishfry (May 24, 2019)

I have went as far as I can. With a 7 slot motherboard I can only get 2 maybe 3 Xorg VM desktops running from it.
If money was no object I found a USB3 controller with dual channels. Probably uses 2 Bus ID's.
With that I could possibly serve 4 independant Xorg desktops. Each with a monitor(multiple possible too) and USB.
I must say I had little success passing through a subset of onboard USB PCI address.
eHCI, xHCI- hands off - USB Enabled, Auto, Smart Auto. So many BIOS settings and I ran them all.
No way could I split the two USB PCI devices. Either all on host or all passed through. The only way I could get onboard USB.

Considered trying Linux VM Xorg but I don't need that. I keep a Devuan VM on the side for building OpenWRT images.

I could see a use case for a single Xorg VM just for isolation and security purpose. Similar to running X in a jail.

I would like to hear others experiences. I never bothered to nail down the lack of audio, but I see the device in guest VM dmesg.
Maybe an Xorg setting or mixer.

Somehow I was hoping this TwitterGuy was using the bhyve framebuffer VNC-EFI and pass that through for console too.
Was thinking why can't you redirect VNC-EFI console video-out to a video card instead of over an IP.


----------



## tingo (May 25, 2019)

This thread would be so much better with pictures...


----------



## Phishfry (May 25, 2019)

shkhln said:


> And after multiple VM reboots?


My dirty little bhyve scripting doesn't do VM reboots I have to take down the whole rig.
I use a shutdown.rc.local script from host to issue SIGTERM events..
I like freewheeling it.
Learning scripting for launching exec, daemon and finallly tmux sessions has been great learning.



Phishfry said:


> I am now launching my two Xorg VM's with tmux. So no issues there.


Actually this was slightly wrong..
With tmux launching I lost my ability to autostart x/autologin via rc.local script I was using.
Manually running via `startxfce4` in the tmux session worked fine. Tried extra sleep time in script with no success.

For hardware virt I look alot at this picture.





						X10DRX | Motherboards | Products | Super Micro Computer, Inc.
					

X10D, X10DR, X10DRX, X10DRX



					www.supermicro.com


----------



## shkhln (May 28, 2019)

Ok, so I finally decided to give bhyve a try:


```
% sudo sh /usr/share/examples/bhyve/vmrun.sh -c 1 -d FreeBSD-12.0-RELEASE-amd64.raw -p 1/0/0 -p 1/0/1 -p 1/0/2 -p 1/0/3 vm0

...

root@freebsd:~ # kldload nvidia-modeset
nvidia0: <Unknown> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
root@freebsd:~ # X -retro

X.Org X Server 1.18.4
Release Date: 2016-07-19
X Protocol Version 11, Revision 0
Build Operating System: FreeBSD 12.0-RELEASE-p5 amd64
Current Operating System: FreeBSD freebsd 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64
Build Date: 16 May 2019  09:00:46AM

Current version of pixman: 0.34.0
        Before reporting problems, check http://wiki.x.org
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Tue May 28 17:11:27 2019
(==) Using config file: "/etc/X11/xorg.conf"
NVRM: GPU 0000:00:04.0: RmInitAdapter failed! (0x26:0x65:1133)
nvidia0: NVRM: rm_init_adapter() failed!
(EE)
Fatal server error:
(EE) no screens found(EE)
(EE)
Please consult the The X.Org Foundation support
         at http://wiki.x.org
for help.
(EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.
```

Am I missing something obvious?


----------



## Phishfry (May 28, 2019)

Did you make a video card config file in VM's /usr/local/etc/X11/xorg.d/ with the BusID of passed-thru video card?


----------



## shkhln (May 28, 2019)

Phishfry said:


> Did you make a video card config file in /usr/local/etc/X11/xorg.d/ with the video card BusID ?



Yes, the relevant section:

```
Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:0:4:0"
EndSection
```


```
root@freebsd:~ # pciconf -lv
...
vgapci0@pci0:0:4:0:     class=0x030000 card=0x218410de chip=0x218410de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'TU116 [GeForce GTX 1660]'
    class      = display
    subclass   = VGA
hdac0@pci0:0:5:0:       class=0x040300 card=0x218410de chip=0x1aeb10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    class      = multimedia
    subclass   = HDA
xhci0@pci0:0:6:0:       class=0x0c0330 card=0x218410de chip=0x1aec10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    class      = serial bus
    subclass   = USB
none0@pci0:0:7:0:       class=0x0c8000 card=0x218410de chip=0x1aed10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    class      = serial bus
```

I also had to set _kern.vty=sc_ in _boot/loader.conf_ because of the very bizarre "xf86OpenConsole: No console driver found" error message from Xorg.


----------



## shkhln (May 28, 2019)

Should I try UEFI mode?


----------



## Phishfry (May 28, 2019)

shkhln said:


> Should I try UEFI mode?


That is what I used as you can see from my above command.


----------



## shkhln (May 28, 2019)

Phishfry said:


> That is what I used as you can see from my above command.



I see. UEFI actually fails with "Boot Failed. EFI Misc Device", which is not something I expected from an official bhyve image.


----------



## Phishfry (May 28, 2019)

I was using FreeBSD 12-STABLE on host and FreeBSD 12-RELEASE on guest VM's.
Hardware is SuperMicro X9SRL and then when messing with onboard USB ports I moved the setup to X10SRI.
PCIe x8 PNY GT710 and GT730 Video cards. The GTX650 I started with was too wide for my slots arrangement.
Started with a pair of 960GB PM983 NVMe then downsized to one to free up PCIe slots..


----------



## Phishfry (May 28, 2019)

How do you have everything in pciconf as NVIDIA?? For example:: NVIDIA Serial Bus - USB

Is this a Nvidia chipset motherboard?? AMD board? LGA2011?

I saw no such errors as you have.

Maybe I need to try this on a Skylake Xeon too. I thought it was easy breezy.
Just like an ordinary Xorg-Nvidia setup.


----------



## shkhln (May 28, 2019)

Phishfry said:


> How do you have everything in pciconf as NVIDIA?? For example:: NVIDIA Serial Bus - USB Is this a Nvidia chipset motherboard?? AMD board? LGA2011?



Let's have a look: besides the GPU itself there is some kind of an audio device, a USB device and some mysterious serial bus. The first one is used for HDMI / Display Port audio, the second is probably for USB-C port present on some Turing GPUs (I don't have one — it's a bit of a premium feature and GTX 1660 is a bottom of the middle range GPU, but whatever) and I have no clue about the third device.

Oh, it's not "everything" — I trimmed uninteresting entries, as usual.



Phishfry said:


> I saw no such errors as you have.



Eh… As far as loading in UEFI mode is concerned, it's just a broken image. Reinstalling everything from ISO works for me. Still the same RmInitAdapter error, though.


----------



## shkhln (May 29, 2019)

A bit more info:

```
root@attempt2:~ # sysctl hw.nvidia.registry.ResmanDebugLevel=0
hw.nvidia.registry.ResmanDebugLevel: 4294967295 -> 0
root@attempt2:~ # echo 123 > /dev/nvidia0
NVRM: GPU 0000:00:04.0: RmInitAdapter
NVRM: GPU 0000:00:04.0: RmSetupRegisters for 0x10de:0x2184
NVRM: GPU 0000:00:04.0: pci config info:
NVRM: GPU 0000:00:04.0:    registers look  like: 0xc1000000 0x1000000NVRM: GPU 0000:00:04.0:    fb        looks like: 0xd000000000 0x10000000NVRM: GPU 0000:00:04.0: Successfully mapped framebuffer and registers
NVRM: GPU 0000:00:04.0: final mappings:
NVRM: GPU 0000:00:04.0:     regs: 0xc1000000 0x1000000 0x0xfffff800c1000000
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dod_method: failed to evaluate _DOD method!
NVRM: GPU 0000:00:04.0: RM reports GPU is not primary VGA
NVRM: GPU 0000:00:04.0: PCI config space reports GPU is not primary VGA
NVRM: GPU 0000:00:04.0:  is not primary VGA
NVRM: nv_acpi_dod_method: failed to evaluate _DOD method!
NVRM: GPU 0000:00:04.0: Tearing down registers
NVRM: GPU 0000:00:04.0: RmInitAdapter failed! (0x26:0x65:1133)
nvidia0: NVRM: rm_init_adapter() failed!
/dev/nvidia0: Input/output error.
```

(Any attempt to access /dev/nvidia* device node is enough to trigger that behavior, so Xorg is not really necessary for testing.)


----------



## Phishfry (May 29, 2019)

Do you have any older Nvidia cards? I had this working with nvidia 340 driver with a legacy card too. GT2xx based card.
How about Radeon. I don't think there is anything special here. I should have tried that.


----------



## shkhln (May 29, 2019)

I gifted my old Kepler card away a few days ago. I'm mostly interested in the newer drivers, hence it wouldn't be very useful even if it does work.


----------



## Phishfry (May 29, 2019)

If there is anything I can do my bhyve VM setup NVMe is on a shelf. Can revist.



shkhln said:


> UEFI actually fails with "Boot Failed. EFI Misc Device", which is not something I expected from an official bhyve image.


Yes this is strange. I have ran the bhyve Tiano UEFI firmware on both Legacy host install as well as EFI host install.
Running package system 'latest' for the firmware. Copy it out to VM directory for convenience.
The Tiano -UEFI-CSM module is broke but I saw DianeB. investigating that in mailing list post.

On my test machines BIOS I had all the PCIe settings jammed to EFI. Every setting. No legacy. EFI only.
With that my newer LSI Controller Cards actually show setup screen in BIOS and you can configure them there.
EFI Payload I assume.

Can you please list your motherboard and CPU. The Skylake Xeon needs bhyve patch for IOMMU to work correctly.


----------



## shkhln (May 30, 2019)

Phishfry said:


> Yes this is strange. I have ran the bhyve Tiano UEFI firmware on both Legacy host install as well as EFI host install.
> Running package system 'latest' for the firmware. Copy it out to VM directory for convenience.
> The Tiano -UEFI-CSM module is broke but I saw DianeB. investigating that in mailing list post.



To be precise, I mean specifically this VM image: https://download.freebsd.org/ftp/re...md64/Latest/FreeBSD-12.0-RELEASE-amd64.raw.xz. I have no complaints about UEFI firmware.


----------



## Phishfry (May 30, 2019)

Ahha. The official raw disk image for VM. I still use a  proper "installed" VM but I am working to just generating custom images.
The raw disk images made me feel hemmed-in by the pre-determined size.


----------



## Phishfry (Jun 10, 2019)

I ended up getting a Startech 4 port USB3 card WITH 4 Channels. Yehawwwww.

```
pcib12@pci0:8:9:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:9:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt0@pci0:10:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
xhci1@pci0:11:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
xhci2@pci0:12:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
```
In ONE slot I have 4 seperate USB devices I can pass.
PLX controller was recognized and it looks groovy.
I did get a different model than the one I talked about above.
I found Startech has 4 separate bus USB3 cards and had to try that. 40 Bucks used.
Looks like it was recognised liked I hoped it would. Each port is on its own BUS-ID.

Still need to verify it all works and re-arrange my PCI stuff for passthru..


----------



## Phishfry (Jun 12, 2019)

OK Short and dirty on this USB3 card.
PEXUSB3S400 STARTECH
https://www.startech.com/support/PEXUSB3S400
It has everything I wanted but does not work for virt.
One VM I connect my physical KVM cable and bingo, it shows up.
Move to another VM and connect my USB KVM cable and I see usbhub errors spewing across screen.
I also tested a memstick in the assigned USB slot and that caused error spew as well and console was unusable.
So great card for ordinary use but not for VM use. Unless you pass all through to one VM.

I kinda knew this was coming as I read EVERYTHING before buying.
Luckily people left virt comments when purchasing. KVM on Linux had the same issue.

These cards do have a PLX chip for control. It shows up as 4 separate PLX devices in pciconf.
So I tried pairing these with a physical USB device. The sibling PLX chips will not passthrough. At all. Ever.
The structure seems to be like this
13:0:0 = Parent PLX address
14:1:0=Sibling PLX address
14:5:0=Sibling PLX address
14:7:0=Sibling PLX address
14:9:0=Sibling PLX address
15:0:0=USB controller
16:0:0=USB controller
17:0:0=USB controller
18:0:0=USB  controller


```
pcib13@pci0:13:0:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib14@pci0:14:1:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib15@pci0:14:5:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib16@pci0:14:7:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib17@pci0:14:9:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
ppt6@pci0:15:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt7@pci0:16:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt8@pci0:17:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt9@pci0:18:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
```


----------



## Camofelix (Jan 12, 2020)

Hi there! Any luck with other OS's with this setup? Thinking of using this to create a steam machine VM for remote play on my network. 

Also, any luck with cuda virtualization? Would be nice to be able to passthrough to a linux VM for some machine learning type applications.


----------



## shkhln (Jan 12, 2020)

Camofelix said:


> Any luck with other OS's with this setup?



Try it and report your results. Everybody is happy to _talk_ about GPU passthrough, but what we really need are tests with different hardware configurations.



> Thinking of using this to create a steam machine VM for remote play on my network.



I actually have a Linux Steam setup script for Linuxulator you might be interested in, see Thread 72140


----------



## forgiven_noob (Jan 27, 2020)

i thought you could onl connect to byhve sessions with VNC
that would not be suitable for gaming it hought?


----------



## kpedersen (Jan 27, 2020)

forgiven_noob said:


> i thought you could onl connect to byhve sessions with VNC
> that would not be suitable for gaming it hought?



Since X11 is network aware you can also forward that to the host. It will be indirect rendering however so quite a bit slower than direct because it is going through the network (by design). In my experience, for retained stuff (i.e vertex data uploaded to the GPU) it is pretty good but for old OpenGL immediate mode usage (i.e the games I want to play!!!), it is not great performance.

If you do try this; don't forward through ssh (even though that is easier). The extra encryption and compression is wasteful. Instead use `xhost` to allow the guest OS X11 programs to access your running X11 server.

Off topic ramble:
Luckily the older games don't do *that* much in terms of rendering so a software rasteriser is feasible (Xvnc + llvmpipe is OK). Unfortunately the platforms that they were designed for (Windows 98) have no real software rendering support and the third party stuff (i.e Google's SwiftShader) are implemented in unportable ways and only support Windows 7+ which my good old Windows 98 games don't support. I just can't seem to win. It is like the world doesn't want me to play my Windows 98 games!


----------



## drozdowsky (Jan 27, 2020)

shkhln said:


> Try it and report your results. Everybody is happy to _talk_ about GPU passthrough, but what we really need are tests with different hardware configurations.


I mean, I would try win 7/10 but anyone got it running? Linux passthrough is possible, windows as far as I know - nope


----------



## shkhln (Jan 27, 2020)

drozdowsky said:


> Linux passthrough is possible



I'm pretty sure it is broken. Some "lucky" hardware might work.


----------



## shkhln (Jan 27, 2020)

forgiven_noob said:


> i thought you could onl connect to byhve sessions with VNC
> that would not be suitable for gaming it hought?



That's completely irrelevant for GPU passthrough. What's relevant is having displays for host and guest GPUs respectively.



kpedersen said:


> Off topic ramble:
> Luckily the older games don't do *that* much in terms of rendering so a software rasteriser is feasible (Xvnc + llvmpipe is OK). Unfortunately the platforms that they were designed for (Windows 98) have no real software rendering support and the third party stuff (i.e Google's SwiftShader) are implemented in unportable ways and only support Windows 7+ which my good old Windows 98 games don't support. I just can't seem to win. It is like the world doesn't want me to play my Windows 98 games!



SwiftShader definitely supported Win XP at some time in the past. Why would you want SwiftShader on Win 98 anyway? It only works with d3d 8 and 9.


----------



## kpedersen (Jan 28, 2020)

shkhln said:


> SwiftShader definitely supported Win XP at some time in the past.



Possibly when it was still managed by the TransGaming guys? I think it could very well be that Google are linking it against a recent C/C++ runtime that prevent its usage on anything below Windows 7. I know I tried compiling it on an older OS but I made very little progress.



shkhln said:


> Why would you want SwiftShader on Win 98 anyway? It only works with d3d 8 and 9.



Yeah, that is true. I suppose I just haven't seen a DirectX 3 software renderer so I can run Interstate '76 haha. Admittedly looking at it now; I might have a bit more luck with some sort of voodoo/glide emulation.


----------



## Phishfry (Jan 28, 2020)

forgiven_noob said:


> i thought you could onl connect to byhve sessions with VNC
> that would not be suitable for gaming it hought?


It is possible to directly passthru some NVidia video cards. I used 7xx series and had 3 cards attached to different VM's with passthru.
I also passed thru USB cards for user inputs. This was using FreeBSD only. I did not test Linux VM's.
GLXgears showed normal performance.


----------



## patpro (Sep 3, 2020)

Just a side note on this very interesting thread. I've just decommissioned my 4 years old multi-head virtualized workstation built on top of ESXi. It used PCI pass-through extensively: 1 Win7 (later Win10) VM with Radeon GPU + full USB controller + digital audio, 1 OSX VM with Radeon GPU + full USB controller, 1 Linux VM with full USB controller. 
Building parts are of paramount importance. I've chosen at the time a Supermicro X10SRA-F motherboard that sports 3 distinct USB controllers, allowing me to use one per VM (as I don't need one for ESXi). Passing through of a complete USB controller allows you to plug absolutely what you want: any keyboard/mouse of course, but also thumdrives, CF card reader, USB hub, yubikey, DVD burner, Nooelec NESDR SMArt, USB-serial adapter, USB Creative Sound card, USB keylogger (yes I did) etc.
I used a USB switch (Aten US424-AT) in order to present a set of devices to a unique VM at the same time (keyboard, mouse, Creative sound card, Yubikey): one press on a button and those devices would be unplugged from USB controller 1 and plugged on USB controller 2, etc. Every GPU was hooked on the same display: one press of a button on the display to switch inputs. Very convenient 
I really hope you'll overcome all obstacles and limitation and get a nice multihead workstation built on top of FreeBSD+bhyve!


----------



## bendany (Sep 11, 2020)

Heads up guys.

Corvin Köhne has created a patch for bhyve, it allows passing through an Intel IGD to VM. Windows guest and Linux guest both work.
You can check following URL 

D26209


----------



## patpro (Sep 11, 2020)

that sounds great  hopefully we won't wait too long before it's available in a FreeBSD RELEASE.


----------



## shkhln (Sep 11, 2020)

Good for them, however this is a totally different use case from passthrough with Nvidia.



bendany said:


> D26209





> Base of Data Stolen Memory


----------



## Phishfry (Sep 13, 2020)

It is a strange use case. Passing through the IGD. I guess you would need to use ssh to control the host.
I am glad someone is working on different options for VM graphics.


----------



## patpro (Sep 14, 2020)

Using FreeBSD as a headless hypervisor can be a very nice idea, giving up its integrated GPU to a guest VM. You got ZFS as storage and that alone is a killing feature.


----------



## ziomario (Aug 23, 2021)

Phishfry said:


> Here is my /etc/rc.local I use to start up my VM. Device 5:0 and 5:1 are my video card.
> 
> ```
> # cat /etc/rc.local
> ...



why you passed thru 5/0/0 and 5/0/1 if your system has :

vgapci0@pci0:0:5:0:    class=0x030000 card=0x129119da chip=0x0fc610de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 [GeForce GTX 650]'
    class      = display
    subclass   = VGA
--------------------------------------------------------------------------------------------------------------------------------------------
hdac0@pci0:0:5:1:    class=0x040300 card=0x129119da chip=0x0e1b10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 HDMI Audio Controller'
    class      = multimedia
    subclass   = HDA

shouldn't you have passed 0/5/0 and 0/5/1 ?


----------



## Phishfry (Aug 24, 2021)

ziomario said:


> shouldn't you have passed 0/5/0 and 0/5/1 ?


That must be a copying error on my part.
On my LGA2011 boards the bus address would be 5:0:0 and 5:0:1 for a PCIe card. Not 0:5:0


----------



## ziomario (Aug 24, 2021)

anyway I did the same. I created a xorg.conf file and I've placed inside the ubuntu VM and then I've tried to boot it in bhyve. On my part it didn't work. The VM does not boot at all. Did u install also the nvidia drivers inside the linux vm before to launch it with bhyve ? (I'm trying with the RTX 2080 ti and the Gtx 1060)


----------



## Phishfry (Aug 25, 2021)

Well I think shkhln made a good point above.
We need to document where this works and where it doesn't.

I am wondering about some things.

Is anybody trying this on Dual CPU boards? Single socket only? E3 Xeon or E5 Xeon?

As stated LGA2011 SuperMicro E5- X9/X10 was my testbed.

My tested platforms were GT710 and GT730. Some 1 Gig some 2 Gig video memory.
All different brands. All x8 slot PCIe cards due to lack of x16 slots on my board.

And like patpro I used a Belkin 16 port KVM connected to my 3 video cards.
One port used for host too..


----------

