# Bhyve GPU pass-through



## ludovitkramar (Nov 30, 2021)

Hello! I've been trying to get my RX570 working on an Asrock H97 Pro 4 motherboard, I've passed the GPU successfully (with vm-bhyve) to an archlinux vm, but it seems like the amdgpu is not working correctly, I can only get stable graphics with acceleration disabled, and everything is extremely slugish.

In dmesg I get the following messages: (partial)


```
[  473.038005] [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
[  475.444739] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=17, emitted seq=20
[  475.444981] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process WebKitWebProces pid 1568 thread WebKitWebP:cs0 pid 1581
[  475.445197] amdgpu 0000:00:06.0: amdgpu: GPU reset begin!
[  476.042614] amdgpu: cp is busy, skip halt cp
[  476.227217] amdgpu: rlc is busy, skip halt rlc
[  476.228234] amdgpu 0000:00:06.0: amdgpu: BACO reset
[  476.845294] amdgpu 0000:00:06.0: amdgpu: GPU reset succeeded, trying to resume
[  476.845807] [drm] PCIE GART of 256M enabled (table at 0x000000F400000000).
[  476.845829] [drm] VRAM is lost due to GPU reset!
[  477.072011] [drm] UVD and UVD ENC initialized successfully.
[  477.174992] [drm] VCE initialized successfully.
[  477.179435] amdgpu 0000:00:06.0: amdgpu: recover vram bo from shadow start
[  477.179451] amdgpu 0000:00:06.0: amdgpu: recover vram bo from shadow done
[  477.179453] [drm] Skip scheduling IBs!
```

it doesn't matter what I do, after a while the driver crashes and resets itself, meanwhile the screen is a glitched color noise mess, it is impossible to make out anything. That is the case on xorg and on wayland.

Is it more likely for this to be a bhyve issue or a driver inside the vm issue?


----------



## SirDice (Nov 30, 2021)

> Note: VGA / GPU pass-through devices are not currently supported.





			bhyve/pci_passthru - FreeBSD Wiki


----------



## ziomario (Nov 30, 2021)

SirDice : is not actually totally true. There are a lot of good and working patches that have been proposed and they worked great with the amd and the nvidia graphic cards. A lot of them haven't been accepted by the freebsd developers,but they work. I'm passing thru my nvidia rtx 2080 ti  with the nvidia patches and it is working great. For sure,it's under my own responsability if something will be wrong with my PC. But I took the risk and everything is going ok. So,if you want to pass thru your amd gpu,you can do that. It works, just right now. But you should know how to apply them.


----------



## ziomario (Nov 30, 2021)

ludovitkramar said:


> Hello! I've been trying to get my RX570 working on an Asrock H97 Pro 4 motherboard, I've passed the GPU successfully (with vm-bhyve) to an archlinux vm, but it seems like the amdgpu is not working correctly, I can only get stable graphics with acceleration disabled, and everything is extremely slugish.
> 
> In dmesg I get the following messages: (partial)
> 
> ...



I suggest to report this behavior to : freebsd-virtualization@freebsd.org


----------



## ludovitkramar (Nov 30, 2021)

Looking for patches I've found this patch that's for review, I've never applied one but I'm sure I could figure it out somehow.
https://reviews.freebsd.org/D27456

Would it take a long time for these patches to get into the binary packages?




SirDice said:


> bhyve/pci_passthru - FreeBSD Wiki


Maybe I haven't said it clearly in the post but the passthrough actually works "fine".


----------



## ziomario (Nov 30, 2021)

ludovitkramar said:


> Would it take a long time for these patches to get into the binary packages?



Who knows ?


----------



## freezr (Nov 30, 2021)

Hi guys, do you know if the GPU-passthrough would work also with an eGPU connected through a Thunderbolt port?

Thanks,
TGL


----------



## SirDice (Nov 30, 2021)

ludovitkramar said:


> Would it take a long time for these patches to get into the binary packages?


Merged into existing releases is unlikely. These things are tested in -CURRENT first. Then they may or may not be MFC'ed to -STABLE. If they are MFC'ed to -STABLE they'll be included in the _next_ minor release of that major branch.


----------



## hsnyder (Dec 1, 2021)

ziomario said:


> SirDice : is not actually totally true. There are a lot of good and working patches that have been proposed and they worked great with the amd and the nvidia graphic cards. A lot of them haven't been accepted by the freebsd developers,but they work. I'm passing thru my nvidia rtx 2080 ti  with the nvidia patches and it is working great. For sure,it's under my own responsability if something will be wrong with my PC. But I took the risk and everything is going ok. So,if you want to pass thru your amd gpu,you can do that. It works, just right now. But you should know how to apply them.


Do you have a link to the patch that you used? I had a look in Phabricator but I only saw the GVT-d patches and the AMD patch that was linked by ludovitkramar...
​


----------



## ziomario (Dec 1, 2021)

I don't have the links for the patches,because its a mess. Some of them are in pre-release stage,some others have been accepted but in the while they have been modified. I can suggest to send an email to : <freebsd-virtualization@freebsd.org>


----------



## GoNeFast_01 (Dec 4, 2021)

ludovitkramar said:


> Looking for patches I've found this patch that's for review, I've never applied one but I'm sure I could figure it out somehow.
> https://reviews.freebsd.org/D27456
> 
> Would it take a long time for these patches to get into the binary packages?


Appreciate this very much.. Thank you



ziomario said:


> There are a lot of good and working patches that have been proposed and they worked great with the amd and the nvidia graphic cards. A lot of them haven't been accepted by the freebsd developers,but they work. I'm passing thru my nvidia rtx 2080 ti with the nvidia patches and it is working great. For sure,it's under my own responsability if something will be wrong with my PC. But I took the risk and everything is going ok. So,if you want to pass thru your amd gpu,you can do that. It works, just right now. But you should know how to apply them.



I am going to do some more testing with these patches I have a couple GPU (Nvidia mostly) laying around I could test emulation .... Were you able to play STABLE games or high performing programs on your nvidia passthru ? ziomario


----------



## ziomario (Dec 7, 2021)

All Intel, AMD and Nvidia GPU passthrough patches are included in these links:

https://github.com/Beckhoff/freebsd-src/commits/phab/corvink/13.0-nvidia

(Nvidia related patches starts at “apply nvidia gpu bar 0 quirk”)

https://github.com/Beckhoff/edk2/commits/phab/corvink/gvt-d


----------



## ziomario (Dec 7, 2021)

Watitsthis said:


> Appreciate this very much.. Thank you
> 
> 
> 
> I am going to do some more testing with these patches I have a couple GPU (Nvidia mostly) laying around I could test emulation .... Were you able to play STABLE games or high performing programs on your nvidia passthru ? ziomario



yes,but only for Linux. Those patches aren't mine. I'm only a tester. They come from the mind of 2 really nice developers,Corvin and Peter.


----------



## hsnyder (Dec 18, 2021)

Hi everyone. I was able to get GPU passthrough working with an nvidia GPU passed through to a Linux guest (ubuntu 20.04). I used the patches posted above, authored by Corvin Köhne of Beckhoff, but I didn't apply them manually. Instead, starting with a relatively clean install of 13.0-RELEASE, I cloned the beckhoff FreeBSD source code mirror at https://github.com/Beckhoff/freebsd-src, specifically Corvin's branch phab/corvink/13.0-nvidia-wip. Note that there's another branch that's just called 13.0-nvidia, without the '-wip', which did _not_ work for me (though it's possible that's my own fault).

Anyways, with that repo, I rebuilt world & kernel, and installed (I literally just followed the steps in the rebuild world part of the freebsd handbook, I've never done it before and had no special knowledge). Then, I followed the instructions on this page https://wiki.freebsd.org/bhyve/pci_passthru to pass the device through to the VM. If you're on an AMD system like me, be sure to check the "Additional notes" at the bottom of the page.

I also had to use Corvin's modified UEFI bootrom. I didn't build it myself, he just sent me the binary, which is too big to attach to this post (~3MB) otherwise I would. If I understand correctly, it can be produced from the code here: https://github.com/Beckhoff/edk2/commits/phab/corvink/gvt-d
Anyways, the modified UEFI filename was BHYVE_BHF_CODE.fd in my case, so I had the following in my `bhyve` command:
`-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd`

Note also that Corvin said that the current patch won't work for a windows guest yet, only FreeBSD or Linux. But if I understand correctly, he's still working on Windows support.


----------



## ziomario (Dec 19, 2021)

Yes,we are working on the passthrough for Windows for the Nvidia gpus (the AMD gpus already work),but we are on the high seas. I think that 13.0-nvidia-wip is the code work in progress for the passthrough for Windows....

----> So it looks like the GPU is limited to PCIe gen 1 speed ?

I don't know,but I'm interested to know what's the answer.


----------

