# FreeBSD 10+ on Xen(Server). Status update



## vadimk (Dec 12, 2014)

Hello all,

      Have been experimenting recently with XenServer 6.2 and FreeBSD 10.1 amd64 trying to get it work in different modes (particularly PV, PVHVM, HVM) and here what I got:

Default installation on UFS went without errors, but performance is very poor. `ioping` shows low numbers ~ 1k IOPS. Can't get UFS fast though some articles say UFS should be faster under virtualization environment than ZFS.
The same installation on ZFS shows great number of IOPS and server is fast indeed. I got 100k IOPS for the same setup. The only problem during installation: kernel panic due to the lack of swap space. Have this error on 2G_B_ RAM and don't have on 6G_B_.
Latest FreeBSD releases already contain Xen PVHVM support at kernel, but in fact you can't use extra functions until you install Xen guest tools. Particularly - can't shutdown, restart, suspend VM from the host. The only option - hard reset, killing domId.
After installing guest tools got control over VM through `xe vm-*` commands. VM status shows it is still running at HVM mode.
Searching internet for different "recipes" I have found following information. If you know more about it, please edit/confirm:

PV mode is only available for i386 kernels. All examples for PV settings are done for Xen hypervisor only and for older releases 6.0, 7.0 -- 9.0. There is no PV support for XenServer guest. It seems PV support for newer kernels is not under development anymore. Is that right?
Classically it states that PV mode should give more performance than HVM. Some articles claim that PVHVM under FreeBSD is faster than PV. So I guess BSD development goes for this specific PVHVM implementation. Other OS support either PV or HVM, but PVHVM seems to be something "unnatural" and invented by FreeBSD team. Is this right?
Still missing text console from the host. It seems that PVHVM mode really performs well (from the point of IO ) but is not recognized by hypervisor and does not give functionality that PV guests suppose to have. Is that right?
What are the plans for development in this area? Will it be PV for amd64 in the nearest future? Or something better?
Thank you for replies,

Vadim.


----------



## lme@ (Dec 19, 2014)

You can try this patch from Roger Pau Monne which enables unmapped IO. But beware:



> unmapped IO would certainly help, I have a patch to enable it but the kernel will crash if you send it unaligned buffers from userspace (i.e: like newfs does): https://bz-attachments.freebsd.org/attachment.cgi?id=144403



With that patch IO is a lot faster.


----------



## vadimk (Dec 20, 2014)

Will this code be merged into production? What is FreeBSD way to deal with virtualized environments? I have tried VmWare and Xen and it seems FreeBSD is working faster under Xen. Still it does not have PV naturally supported.


----------



## lme@ (Dec 20, 2014)

I don't know if it will get committed. Maybe and maybe in a different version. I used it as a stop-gap.


----------



## vadimk (Dec 21, 2014)

Thank you for tip, Lme, I will definately try it.
The bigger picture is still vague. What are the trends for getting FreeBSD into virtualized environments? The world becomes more and more virtual an there are very few scenarios I can imagine where FreeBSD runs on real hardware. 
I can't estimate what does it mean to get support for PV at Xen or working VmWare tools for ESXi from the developer perspective. May be there is a third way ?


----------



## ondra_knezour (Dec 21, 2014)

You may find following articles useful
http://wiki.xen.org/wiki/Virtualization_Spectrum
http://wiki.xen.org/wiki/Understanding_the_Virtualization_Spectrum


----------



## lme@ (Dec 21, 2014)

And there's also FreeBSD's own Hypervisor http://bhyve.org/


----------



## vadimk (Dec 21, 2014)

Thank you Ondra and Lme, I feel regret not to read these articles before. Have now better understanding of what is going on. Hope PVH will happen for FreeBSD some time, unless then will use PVHVM.


----------



## ondra_knezour (Dec 21, 2014)

The PVH mode for FreeBSD already happened, see http://wiki.xen.org/wiki/FreeBSD_PVH

However I didn't have time to experiment with it, so I have no advice about setup, stability, etc.

There is also XenServer Creedence release candidate - what would be the next XenServer version, which is available for download http://www.xenserver.org/component/content/article/11-product/142-download-pre-release.html - revision mentioned on the wiki page may be already incorporated in it, so any testing may be only matter of installation and VM setup.


----------



## vadimk (Dec 21, 2014)

I've seen similar installation type for Xen hypervisor when was looking for PV guest support. It seems I should install latest Xen hypervisor on Linux and then configure DomU manually. The option to use pre-release for the next XenServer product seems to be more attractive as I am going to use it together with CloudStack.  Will test it when get spare hardware.


----------



## vadimk (Jan 30, 2015)

ondra_knezour said:


> The PVH mode for FreeBSD already happened, see http://wiki.xen.org/wiki/FreeBSD_PVH
> 
> However I didn't have time to experiment with it, so I have no advice about setup, stability, etc.
> 
> There is also XenServer Creedence release candidate - what would be the next XenServer version, which is available for download http://www.xenserver.org/component/content/article/11-product/142-download-pre-release.html - revision mentioned on the wiki page may be already incorporated in it, so any testing may be only matter of installation and VM setup.



XenServer 6.5 has been released with Xen 4.4 on board and I have tried to follow the article at Wiki to create PVH guest.  However some things are confusing me much:

XAPI/XE toolstack is a default one to work with VM-s. XL is also present, but XL seems not be able to work with virtual devices created by XE. I was not able to create VM using pure XL.
Is it possible somehow to exchange (convert) domain configuration between XE and XL ?  Domains are visible in both toolstacks, but they work with them differently.  
Is it possible to create PVH guest using pure XAPI/XE?


----------

