# bhyve for driver creation workflow



## Farhan Khan (Mar 2, 2017)

Hi all,

Is it possible to pass on my laptop's PCI wifi card to a bhyve VM running Linux?

I am writing (and mostly failing to write ) a FreeBSD device driver for the Realtek RTL8188EE and am looking to increase my workflow. Currently I am relying on the Linux driver as documentation, but switching between Linux and FreeBSD to examine memory is painfully slow.

I figured I could increase my workflow speed by running Linux in a bhyve VM, passing the PCI wifi card to it, and shutting it down and having FreeBSD reclaim the device on an ad hoc basis without rebooting the system.

Is this possible? Also, is it there a guide to running any Debian-based distro in bhyve? The current handbook guide seems out of date, and/or is not working (I can provide a more detailed explanation if need-be, but I also recall a BSD Now! episode where they said grub-bhyve being a temporary tool that is not needed anymore?)

Thanks!


----------



## SirDice (Mar 2, 2017)

Farhan Khan said:


> Also, is it there a guide to running any Debian-based distro in bhyve?


I highly recommend using sysutils/vm-bhyve and the templates it provides.

https://github.com/churchers/vm-bhyve/wiki/Supported-Guest-Examples


```
root@molly:~ # vm list 
NAME            DATASTORE       LOADER      CPU    MEMORY    VNC                  AUTOSTART    STATE
pptimg01        default         uefi        1      512M      -                    No           Stopped                  
ppttst01        default         uefi        1      512M      0.0.0.0:5900         No           Running (33789)              
ubuntu12        default         grub        2      2048M     -                    No           Stopped                       
wintermute      default         bhyveload   4      2048M     -                    Yes [1]      Running (86515)                
root@molly:~ # vm create -t ubuntu ubttst01                                                                                    
root@molly:~ # vm list                                                                                                           
NAME            DATASTORE       LOADER      CPU    MEMORY    VNC                  AUTOSTART    STATE                               
pptimg01        default         uefi        1      512M      -                    No           Stopped
ppttst01        default         uefi        1      512M      0.0.0.0:5900         No           Running (33789)
ubttst01        default         grub        1      512M      -                    No           Stopped
ubuntu12        default         grub        2      2048M     -                    No           Stopped
wintermute      default         bhyveload   4      2048M     -                    Yes [1]      Running (86515)
root@molly:~ # vm destroy ubttst01
Are you sure you want to completely remove this virtual machine (y/n)? y
root@molly:~ # vm create -t debian debtst01                                      
root@molly:~ # vm list                                                                              
NAME            DATASTORE       LOADER      CPU    MEMORY    VNC                  AUTOSTART    STATE    
debtst01        default         grub        1      512M      -                    No           Stopped         
pptimg01        default         uefi        1      512M      -                    No           Stopped           
ppttst01        default         uefi        1      512M      0.0.0.0:5900         No           Running (33789)    
ubuntu12        default         grub        2      2048M     -                    No           Stopped             
wintermute      default         bhyveload   4      2048M     -                    Yes [1]      Running (86515)
```

Pass-through should be working but it's still a fairly new feature so it may not work correctly.


----------



## Farhan Khan (Mar 2, 2017)

Before I jump into vm-bhyve, I wanted to see if I could do the pass-through raw, but I am getting this error:

```
bhyve: PCI device at 2/0/0 is not using the ppt(4) driver
```
I set /etc/loader.conf to pptdevs="2/0/0" and rebooted.
My relevant pciconf -lv is still:

```
none3@pci0:2:0:0:       class=0x028000 card=0x197d103c chip=0x817910ec rev=0x01 hdr=0x00
   vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8188EE Wireless Network Adapter'
    class      = network
```
I rebooted, but the none3 is not replaced with ppt0 or the like. So when I start, I get this:

```
# grub-bhyve -S -m device.map -r hd0,msdos1 -M 1024M ubuntuguest
# bhyve -S -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./ubuntu.img -s 7,passthru,2/0/0  -l com1,stdio -c 2 -m 1024M ubuntuguest
```
Again, this error:

```
bhyve: PCI device at 2/0/0 is not using the ppt(4) driver
```
pciconf is still showing the device as none3, not ppt0. Not sure why.
I set virtualization in my BIOS, but here's my dmesg(8) in case I am missing something critical: https://dpaste.de/7WwD


----------



## shuxuef (Mar 3, 2017)

Farhan Khan said:


> Before I jump into vm-bhyve, I wanted to see if I could do the pass-through raw, but I am getting this error:
> 
> ```
> bhyve: PCI device at 2/0/0 is not using the ppt(4) driver
> ...


Did you really mean /etc/loader.conf or was it just a typo? It should've been /boot/loader.conf. Did you also check whether your CPU supports the IOMMU feature?

As far as I know, it is not possible for the host machine to reclaim the pass-through PCI devices without a reboot. But I am not following CURRENT, so I might be wrong. But I guess one work-around is to also run a FreeBSD virtual machine, so that you can pass through the PCI device to it after releasing it from the Linux virtual machine.


----------



## aragats (Mar 3, 2017)

shuxuef said:


> you can pass through the PCI device to it after releasing it from the Linux virtual machine


Some time ago I've opened a bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205549 which has never been assigned...
It didn't work in particular with a Linux guest (but worked with MS Windows). Maybe it's changed now though.


----------



## grehan@ (Mar 4, 2017)

The `pptdevs` variable in `/boot/loader.conf` only works if the bhyve kernel module is also loaded at boot time using `vmm_load="YES"` in the said file.

However, in FreeBSD CURRENT and 11-stable, the moving of a device to/from ppt status can be done at run-time using `devctl`. See John Baldwin's post on this for details: http://bsdjhb.blogspot.com/2016/10/using-pci-pass-through-with-bhyve.html


----------



## aragats (Mar 5, 2017)

There is another issue which has been reported several times (e.g. here and here):
	
	



```
passthru device 0/29/0 BAR 0: base 0xdf823000 or size 0x400 not page aligned
```
It happened to me many times with USB controllers, which is a big drawback. It exists in FreeBSD 11, haven't tried CURRENT yet.
E.g. I have a good 12-core Xeon desktop with 3 USB controllers and need to path just a software license protection USB dongle to my Windows 7 bhyve guest, and I cannot because of that! (Eventually I've found a USB-to-Ethernet adapter just for this purpose.)


----------



## Farhan Khan (Mar 8, 2017)

My device is PCIe.


----------



## aragats (Mar 8, 2017)

Farhan Khan said:


> My device is PCIe


Sure! The USB controllers are also PCI(e) devices. Currently there is no USB pass-trough functionality in bhyve(8).


----------



## abishai (Mar 8, 2017)

aragats said:


> I've found a USB-to-Ethernet adapter just for this purpose


Never heard about such device class. How it works? I need to passthrough a software protection dingle as well, byt my system lacks the ability to passthrough PCI devices at all.  Do you need to passthrough Ethernet controller to guest or only network connectivity and drivers for guest are needed?
Sorry for thread hijacking =/


----------



## aragats (Mar 8, 2017)

You need just a network connection and the corresponding driver provided by the vendor (for MS Windows guest).
Here is one of such USB-to-Ethernet hubs, the same model I have, but they are quite expensive:
http://www.bb-elec.com/Products/USB-Connectivity/USB-Over-Ethernet/USB-Over-Ethernet-Hub/UE204.aspx
Of course, I wouldn't pay $199 for such purpose only, just found an abandoned one at my workplace, here is a MS Windows guest screenshot with a dongle attached:
http://files.ara-ler.com/freebsd/usb-eth.png


----------



## aragats (Mar 8, 2017)

abishai , it looks that there are commercial software solutions as well, and they have trial versions:
http://www.kernelpro.com/usb-over-ethernet.html
http://www.eltima.com/products/usb-over-ethernet/
Also there exist significantly less expensive devices, although dedicated for printers:
https://www.newegg.com/Product/Product.aspx?Item=9SIA50M1NT6554


----------

