# Bhyve with Windows 10



## rdunkle (Jul 13, 2016)

Recent changes to Bhyve have allowed for a VNC console to complete
guest installs.  This example will install Windows 10 as a guest.

For this example FreeBSD 12-CURRENT was installed:
FreeBSD 12.0-CURRENT #0 r302526

Edit:  /boot/loader.conf
Add the following lines:

```
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
```
Edit: /etc/rc.conf
Add the following lines
**Note -- change em0 to whatever your real network interface is

```
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm em0 addm tap0"
```

For simplicity -- this example -- all files are copied to a single directory
The MS Windows 10 install iso:  win10.iso

The latest virtio iso file:
`fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.118-2/virtio-win-0.1.118.iso`

Then install the latest Bhyve UEFI:
You can read more here:
https://wiki.freebsd.org/bhyve/UEFI

The basic steps:
Create a temporary directory to store the branch.
Change into that directory.
Execute these commands:

```
# svnlite co http://svn.freebsd.org/base/projects/bhyve_graphics
# cd bhyve_graphics
# make BHYVE_SYSDIR=/usr/src/ -m /usr/src/share/mk
# cd /usr/ports/sysutils/uefi-edk2-bhyve
# make install
```

Copy the new uefi file into our Bhye directory with the rest of our files:
`cp  /usr/local/share/uefi-firmware/BHYVE_UEFI.fd  the_directory_of_our Bhyve files`

Create the empty Bhyve Win10 image:
`truncate -s 70GB win_10.img`

This will be the first boot to start the install:
`bhyve -c 2 -s 0,hostbridge -s 3,ahci-cd,./win10.iso -s 4,ahci-hd,./win_10.img -s 10,virtio-net,tap0 -s 11,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait -s 20,xhci,tablet -s 31,lpc -l bootrom,./BHYVE_UEFI_20160526.fd -m 2G -H -w win10`

The install waits for the vnc connection.
Connect with vnc client (I used tsclient) to the Windows install console:
`ip_address_of_bhyve_host:5900`

Press Enter (on the vnc screen) to get the install started.
Let the installer copy all the files and do the first reboot

After the reboot.  Bhyve exits normally.
Now create an empty install DVD.  We no longer need the install iso, but
Bhyve tries to boot from a DVD.
`touch null_dvd.iso`

Boot Win10 a second time and let it continue the installation.
Notice how we use the null_dvd.iso.

Again, connect to the Windows console using VNC

`bhyve -c 2 -s 0,hostbridge -s 3,ahci-cd,./null_dvd.iso -s 4,ahci-hd,./win_10.img -s 10,virtio-net,tap0 -s 11,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait -s 20,xhci,tablet -s 31,lpc -l bootrom,./BHYVE_UEFI_20160526.fd -m 2G -H -w win10`

The install will continue and then reboot again.

The third boot of Windows use the same command line to continue the installation:
`bhyve -c 2 -s 0,hostbridge -s 3,ahci-cd,./null_dvd.iso -s 4,ahci-hd,./win_10.img -s 10,virtio-net,tap0 -s 11,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait -s 20,xhci,tablet -s 31,lpc -l bootrom,./BHYVE_UEFI_20160526.fd -m 2G -H -w win10`

After you get the Windows desktop there will be no network.
Need to install the virtio network driver.
Shutdown Windows 10.

Boot  this time with the virtio net drivers from the CD:
`bhyve -c 2 -s 0,hostbridge -s 3,ahci-cd,./virtio-win-0.1.118.iso -s 4,ahci-hd,./win_10.img -s 10,virtio-net,tap0 -s 11,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait -s 20,xhci,tablet -s 31,lpc -l bootrom,./BHYVE_UEFI_20160526.fd -m 2G -H -w win10`

Login to Windows.
Look for the CD drive. Open this folder to install the network driver:
NetKVM\w10\amd64
Right-click "netkvm" and select install
Networking should start to install and configure.
That should be all needed for Windows guest install.

You can configure RDP or use VNC to connect.
Use this command to start the Windows guest:
`bhyve -c 2 -s 0,hostbridge -s 3,ahci-cd,./virtio-win-0.1.118.iso -s 4,ahci-hd,./win_10.img -s 10,virtio-net,tap0 -s 11,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait -s 20,xhci,tablet -s 31,lpc -l bootrom,./BHYVE_UEFI_20160526.fd -m 2G -H -w win10`


----------



## usdmatt (Jul 13, 2016)

If you're running 12-CURRENT, you don't need to build the bhyve_graphics project branch - it's already been merged into head. In fact it's also in the 11 beta so both of these can run UEFI with a VNC console out of the box. There was a push to get this code into head before the 11-stable fork.

The uefi port is also a bit of a pain to build as it depends on gcc48. I'd recommend using the package from the latest pkg branch if you can, or downloading it from @grehan's FreeBSD page.

It's also made a lot easier using one of the available bhyve tools such as (sorry, shameless plug) sysutils/vm-bhyve, or sysutils/iohyve. Michael Dexter's vmrc has also been heavily updated in the last few days although I haven't had a chance to see what that's like yet.


----------



## fidaj (Jul 13, 2016)

See also sysutils/cbsd


----------



## priyadarshan (Sep 25, 2016)

And also sysutils/chyves


----------



## ajschot (Dec 13, 2016)

Does this work with an AMD A8 processor too??


----------



## usdmatt (Dec 13, 2016)

As far as I'm aware everything on this list for Win8 Hyper-V should be supported as it specifies the AMD-V and RVI features which are equivalent to VT-x & EPT on Intel. 
http://support.amd.com/en-us/kb-articles/Pages/GPU120AMDRVICPUsHyperVWin8.aspx

The mailing list announcement suggests that "pretty much anything since 2010 will have the features required"
https://lists.freebsd.org/pipermail/freebsd-virtualization/2014-October/002905.html


----------



## usdmatt (Dec 13, 2016)

Oh wait, one thing I'm not sure about is which AMD processors have unrestricted guest support which would be needed for UEFI (or whether it's even available/needed on AMD)...


----------



## ajschot (Dec 13, 2016)

I have no idea only i tried with FreeBSD 10 (on FreeNAS) and bhyve was not supporting UEFI. I only can find that my processor should be working but all information that i found is about FreeBSD 11 so maybe i have to just try it out.... and install FreeBSD 11. My AMD processor is the latest generation, so it should be working... I only can find that it must support POPCNT and NRIPS and it has both features...


----------



## usdmatt (Dec 13, 2016)

I'm fairly sure UEFI only works in 11


----------



## ajschot (Dec 13, 2016)

usdmatt said:


> I'm fairly sure UEFI only works in 11


Ok i will try tomorrow and see if it works i searched a lot and i can not see why it would not work


----------



## roddierod (Dec 13, 2016)

I have an AMD FX 6100 and bhyve works with Windows 10 on FreeBSD 11.  It did not work on FreeBSD 10.


----------



## ajschot (Dec 13, 2016)

tthanks


roddierod said:


> I have an AMD FX 6100 and bhyve works with Windows 10 on FreeBSD 11.  It did not work on FreeBSD 10.


It is just what I already thought, I also read it somewhere but I can not find it anymore. Thanks for your confirmation!


----------



## ajschot (Dec 13, 2016)

dmesg.boot:

```
CPU: AMD A8-7600 Radeon R7, 10 Compute Cores 4C+6G (3094.43-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x630f01 Family=0x15 Model=0x30 Stepping=1
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36
,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x3e98320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSX
SAVE,AVX,F16C>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0xfebbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT
,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC,<b25>,DBE,PTSC>
Structured Extended Features=0x9<FSGSBASE,BMI1>
XSAVE Features=0x1<XSAVEOPT>
SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=65536
TSC: P-state invariant, performance statistics
real memory = 34846277632 (33232 MB)
avail memory = 33202843648 (31664 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 16
cpu1 (AP): APIC ID: 17
cpu2 (AP): APIC ID: 18
cpu3 (AP): APIC ID: 19
```

so it should be working


----------



## grehan@ (Dec 13, 2016)

>which AMD processors have unrestricted guest support

 It's available on all AMD CPUs that bhyve supports - on AMD, it was never a separate feature.


----------



## grehan@ (Dec 13, 2016)

UEFI works on 10-stable, just not the graphical part. For Windows, that means an unattended install.


----------



## usdmatt (Dec 14, 2016)

Ah, good to know that all the AMD processors support it. I got my hands on a Xeon E5504 machine the other day thinking it would make a good test system and it doesn't have UG


----------



## ajschot (Dec 14, 2016)

help needed! new to FreeBSD coding... i'm running FreeBSD 12 and want to install Windows in bhyve (just to test and make clear it is working on my AMD). 
First thing... fetch the virtio iso does not work?? 


```
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.126-2/virtio-win-0.1.126.iso

Certificate verification failed for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA

34374374744:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/usr/src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:1264:

fetch: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.126-2/virtio-win-0.1.126.iso: Authentication error
```


Next i have the windows iso on a usb flash drive how to get it into FreeBSD?
`mount /dev/[usb drive]` ?


----------



## akil (Dec 15, 2016)

Hi,

Just tried this and everything seems to work besides networking. Installation pass like a charm, however, during first install I have had to wait for timeout and to get an console from EFI, than I was able to chose installation media. 

I've used pure FreeBSD11 + p5, I've setuped tap1 -> bridge0 <- wlan0 from my atheros, and also I've tried another physical device, unfortunately, when W10 boots up (I've installed VirtIO from fedora KVM project) there was no flow on tap1 interface, neither on bridge. I'm thinking that virtio-net in FreeBSD11 + Bhyve probably does impact on that behavior (I've tried few previous drivers but they didn't solve networking issues). 

Can any one confirm my assumption that bhyve in FreeBSD11 is still out of date enough to be not capable to bring networking in Windows10 ?


----------



## grehan@ (Dec 16, 2016)

Which version of the virtio windows drivers did you install ? The 1.126 version (at least the first one) was known to not pass packets.

1.118 is know to be solid:
https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.118-2/


----------



## usdmatt (Dec 16, 2016)

I'm not sure what other interface you tried, but lots of people report problems using wlan interfaces for virtual guests. I haven't bothered looking into the details but I don't think wireless access points like seeing traffic for multiple MAC addresses from one client. Usually with wireless you have to resort to using NAT.


----------



## akil (Dec 16, 2016)

It works like a charm with 0.1.118_2. It seems that driver impact on the wrong behavior. I was able to get it work on win7 and win10 with FreeBSD11+p5.

usdmatt: Everything work. Here is my setup: em0 (internet + nat), ath0+wlan0 for home purpose, tap1-bridge0-wlan0 intentionally I've used wlan0 because I'm to lazy to change my PF setup xD. So I've used working one and according to above the only issue was with the virtio driver for windows.


----------



## ajschot (Dec 16, 2016)

it is an certificate eerror but i downloaded it via firefox in XFCE.
Have it up and running only need some tweeking, also i can not use another VNC viewer than tightVNC which is not the nicest... that is very strange... also got no internet connection i think i did something wrong with the bridge setup...


----------



## ajschot (Dec 17, 2016)

I run into problems... on my machine i tried first on Intel machine with FreeBSD 12 it worked with 2 cpu's and 2Gb ram now i tried on my AMD and it freezes the installation of Windows 10 with all the same settings. When i made the VM 4G Ram it freezes later and with 8Gb of ram it freezes again but again later. What am i doing wrong??

EDIT: Think a memory error... ?


----------



## ajschot (Dec 18, 2016)

it only works on AMD A8 with FreeBSD 12 with 1 cpu core

EDIT: It could be a broken memory problem.


----------



## kaN5300 (Jan 16, 2017)

grehan@ said:


> Which version of the virtio windows drivers did you install ? The 1.126 version (at least the first one) was known to not pass packets.
> 
> 1.118 is know to be solid:
> https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.118-2/



Confirm. Even 0.1.126-2 (changelog) does not pass packets. Tried 1.126 - works like a charm in 11.0-RELEASE-p2 GENERIC on Windows 10 LTSB x64.


----------



## kaN5300 (Mar 19, 2018)

UPD:

Today i was baking Win10 LTSB as a bhyve guest on another 11.1-RELEASE system and could not launch 0.1.126 redhat virtio driver i mentioned the post above. Problem solved updating virtio driver to 0.1.141 https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso


----------



## hrenznaet (Apr 5, 2018)

For newbs like me - please explain how 'bhyve with windows 10' will look like. Will it have windows GUI?
Or it will only run in text mode?


----------



## usdmatt (Apr 6, 2018)

> For newbs like me - please explain how 'bhyve with windows 10' will look like. Will it have windows GUI?
> Or it will only run in text mode?



You can access the Windows console using VNC. It's not great for extended use or anything that would require more advanced graphics, but in the server environment which bhyve is mainly aimed at, it's enough to be able to configure the system if you don't have RDP access.


----------



## rdog (Aug 17, 2022)

Hey guys! First post here, an attempt at a 2022 differential update for this great tutorial, on FreeBSD 13.1-RELEASE. Also sharing some learnings.

The official handbook chapter about virtualization, together this forum post, is pretty good already! Some of the surprizes I saw along the way:

Virtualization:

Trying to run bhyve, I got an error about SMV not available. Turns out I needed to enable VT-X in the BIOS.

Virtio disk driver:

The need to remaster Windows ISO to include virtio storage driver mentioned in wiki: not really needed, during the install one can add drivers. So I sticked `-s 5:0,ahci-cd,virtio-win-0.1.118.iso` to expose virtio drivers as an additional CD drive, and let Windows find the driver from there.
But, I found I don't need that either: as the Klara article points, one can use nvme driver using -`s 3:0,nvme,./w0.img`, which is natively found by Windows.
At one point, I encountered `vtblk: write to GUEST_FEATURES: bad size 2` - but in retrospect not sure if this was a genuine error, or something printed on each VM shutdown. Since I moved to nvme since then, didn't check.
Snapshotting:

Initially I just used a file as disk image, but the handbook points one can use a zfs dev volume - probably more effective, but also support snapshotting.
Moving disk image to zfs dev (after turning off VM - always turn off VM before snapshotting, to take a consistent image):
`sudo zfs create -V50G -o volmode=dev zroot/w10disk0`
`sudo dd if=w0.img of=/dev/zvol/zroot/w10disk0 bs=8M`
`sudo zfs snapshot zroot/w10disk0@after_install`


Virtio network driver:

Similarly, Windows found the network driver from the attached CD.
(What I want to test yet: the VNC connection even through localhost is a bit sluggish. See if using RDP would be better, or other ways to speed up graphics. Not sure if it is the protocol or graphics driver here)

UEFI firmware:

`sudo pkg install edk2-bhyve` installs it.
Memory:

Initially I started up with 2048 MBs. After the VM quit, I wanted to bump that to 4096 MB, but got an error `Unable to setup memory (22)`. Took some googling to know that bhyve maintains some reservation per active VM (still not sure how this works, didn't found reference in man pages), that one can reclaim with `bhyvectl --vm=<thevm> --destroy`. Or maybe the first running establishes the settings, and the second invocation with larger memory request doesn't fit into that invocation? Hm, should try `bhyvectl --vm=<thevm> --set-mem ...` the next time I run into this.
Sources:

(this thread)
https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-bhyve
https://klarasystems.com/articles/from-0-to-bhyve-on-freebsd-13-1/
https://wiki.freebsd.org/bhyve/Windows
https://github.com/TritonDataCenter/smartos-live/issues/843#issuecomment-537657378 (about memory reclaiming)


----------



## Lamia (Aug 17, 2022)

CBSD, based on bhyve, runs Win 10 successfully. Several people have got bhyve runs Win too without necessarily using a bhyve-manager like cbsd. You want to try a manager like CBSD, vm-bhyve(which I also use)? 



rdog said:


> the VNC connection even through localhost is a bit sluggish. See if using RDP would be better, or other ways to speed up graphics. Not sure if it is the protocol or graphics driver here


CBSD over vncviewer has been fine so far. I can now play my steam games - EU4, etc.


----------

