# bhyve pci passthrough



## rootbert (Feb 28, 2020)

I just wasted some time on trying to get pci-passthrough to work. I tried a lot, and configured the stuff according to the wiki. pciconf -lv shows

```
ppt1@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
```

and trying to run a system with "sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 2048M -p "3/0/0" -d disk0.img -w freebsd" just aborts while booting with:

```
Loading kernel...
/boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 syms=[0x8+0x178bc0+0x8+0x1969d5]
Loading configured modules...
/boot/entropy size=0x1000
bhyve: PCI device at 3/0/0 is not using the ppt(4) driver
device emulation initialization error: Device busy
```

tried the same with vm-bhyve, which dies with "bhyve exited with status 4". System is 12.1-RELEASE, and CPU: Intel(R) Celeron(R) CPU  J1900  @ 1.99GHz. as mentioned in the wiki, all looks great except for the "acpidump -t | grep DMAR" which does not find anything, but neither on my ryzen 3700X where I also tried to passthrough the network card. Any hints?


----------



## rootbert (Feb 28, 2020)

the system I am talking about has 4 '82583V Gigabit Network Connection' Adapters, one using the host. could it be that the driver being used by the host is somehow interfering?


----------



## Phishfry (Feb 28, 2020)

rootbert said:


> CPU: Intel(R) Celeron(R) CPU J1900


I wonder if your problem is the host. I doubt that a J1900 supports IOMMU
IOMMU is required for PCI passthrough.


			bhyve/pci_passthru - FreeBSD Wiki
		


I am using two very similar cards with no issues. You must passthrough 4 PCI addresses for them to work with all 4 ports.
Like this: 
pptdevs="3/0/0 3/0/1 4/0/0 4/0/1"
That will get you 4 ethernet devices to passthrough.


----------



## Phishfry (Feb 28, 2020)

Looking here:








						Product Specifications
					

quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.




					ark.intel.com
				




I see no VT-d support for your CPU.

Maybe the Ryzen has IOMMU support?


----------



## Phishfry (Feb 28, 2020)

Here is one of my 4 port network cards:

```
42857-ppt0@pci0:6:0:0:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
42939-    vendor     = 'Intel Corporation'
42976:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43048:    class      = network
43073:    subclass   = ethernet
43099-ppt1@pci0:6:0:1:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43181-    vendor     = 'Intel Corporation'
43218:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43290:    class      = network
43315:    subclass   = ethernet
43341-ppt2@pci0:7:0:0:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43423-    vendor     = 'Intel Corporation'
43460:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43532:    class      = network
43557:    subclass   = ethernet
43583-ppt3@pci0:7:0:1:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43665-    vendor     = 'Intel Corporation'
43702:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43774:    class      = network
43799:    subclass   = ethernet
```



rootbert said:


> could it be that the driver being used by the host is somehow interfering?


No. You can pass through whatever you want. Each port does use a seperate PCI address.
So you can pass through one or all four ports.


----------



## rootbert (Feb 28, 2020)

thanks for the reply! here is my pciconf of all 4 ports (now again ppt disabled):

```
em0@pci0:1:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em1@pci0:2:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em2@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em3@pci0:4:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
```

well, bad it does not work on that machine. but will try on the ryzen, just out of interest.
I wanted to run the host, and 3 vms each with an ip-address of my homeoffice network ... any suggestion how to accomplish a similar thing? maybe a private network and a 1:1 mapping from one of the ips for the vms to the private ip, or a bridge?


----------



## Phishfry (Feb 29, 2020)

rootbert said:


> I wanted to run the host, and 3 vms each with an ip-address of my homeoffice network ... any suggestion how to accomplish a similar thing?


I use DHCP on my network so it is simple.
On the host assign the Intel Interface DHCP.
Then in the VM assign the passed-thru interface DHCP.
To assign each ppt ethernet interface I use plain old bhyve. No helper programs.
I setup each bhyve instance to use UEFI-Bhyve firmware. I use a /etc/rc.local script to start my VM's.
It looks like a tangled mess but I feel comfortable with the method.
Here is a snippet of my script.

```
#!/bin/sh
sleep 6
/usr/local/bin/tmux new-session -d -s freebsd1 'exec bhyve -S -c 8 -m 8G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 7:0,passthru,6/0/0 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1'
sleep 6
/usr/local/bin/tmux new-session -d -s freebsd2 'exec bhyve -S -c 4 -m 4G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd2.img -s 7:0,passthru,6/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd2'
```
So I use tmux to execute my bhyve instances.
I use four amd64 VM's and one i386 VM. There is no i386 UEFI so that one has a bit more to it.

I really think it is wise to learn raw bhyve with no helpers. It will give you a good foundation to build off of.
There are great examples in the handbook. It is not that hard.








						Chapter 23. Virtualization
					

Virtualization software allows multiple operating systems to run simultaneously on the same computer




					www.freebsd.org
				



So skip the tap and bridge crap. With passthru it is not needed.
Also skip the vmrun.sh method and start bhyve from the command prompt.
Then you can use command line history to build your startup script.

One word of advice. You can pass through 2 or more disks/images to a VM.
The VM will always boot off the first image presented.
So for initial setup use image #1 as the FreeBSD installer image(memstick image) and make image #2 the new empty image for a VM.

I like to make a /vm directory to hold all my VM's and installers.
I also copy the UEFI-bhyve firmware file to this directory for a shorter file path..(BHYVE_UEFI.fd)


----------



## Phishfry (Feb 29, 2020)

I am a GUI queen so I also have Virt-Manager running on one experimental box.
This is harder as it requires XML file for setup of libvirtd and Bhyve.
Here are some quick screenshots.


----------



## Phishfry (Feb 29, 2020)




----------



## rootbert (Mar 3, 2020)

cool, thanks! Did not expect virt-manager works with bhyve ... I have used it for years in all my corporate linux environments. Do you have a suggestion for the machine where passthrough is not possible to have a similar working setup? Is there a kind of "map-interface-on-host-to-virtual-interface-in-vm"? (looking for a more performant and elegant solution than bridging which comes to my mind)


----------



## joeafterdinner (Jul 13, 2020)

I had the same error, with the following in my vmname.conf for vm-bhyve:
...
passthru0="2/0/0=00:08.0"

Result: Exit status 4

Changed to:
passthru0="2/0/0=08.0"

That change, along with having:
pptdevs="2/0/0"
in /boot/loader.conf works for me.

Good luck.


----------



## diego (Jul 14, 2020)

Phishfry said:


> View attachment 7561


Which software are you using to get Bhyve with GUI? It looks pretty cool 
Comparing the terminal ...

```
root@freebsd:~ # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTOSTART  STATE
centos7        default    grub       2    2048M   -    No         Stopped
debian-docker  default    grub       2    1024M   -    No         Stopped
freebsd12      default    bhyveload  1    1024M   -    No         Stopped
ubuntu16       default    grub       2    2048M   -    No         Stopped
```
Cheers


----------



## SirDice (Jul 14, 2020)

Read post #8.


----------



## Phishfry (Jul 14, 2020)

Bhyve GUI with libvirtd requires this:
Install devel/libvirt

Configure libvirt for bhyve by making an XML file.


			libvirt: Bhyve driver
		

Start up bhyve and test your configuration with `virsh`.

Then you can install the GUI facility on your host anywhere you want and log into Virtual Machine Manager.
deskutils/virt-manager
From here you can now create new VM instances and manage them with a GUI.


----------



## rootbert (Jan 26, 2022)

I just came here again via searching the internet for the same error, just to note for future users: if nmdm kernel module is not loaded and you want to use vm-bhyve with an uefi VM with vnc graphics it stops with "bhyve exited with status 4". So just load the kernel module before. Time travelling is great, I will probably stumble upon this in a few months again haha


----------

