# How to include Xen as Host kernel (on FreeBSD) in my custom kernel



## andrian (Apr 29, 2019)

Hi people, please help me.
How to include XenAsHost kernel in my custom kernel?
I install XenAsHost (ver. 47) in FreeBSD (11.2-RELEASE) from ports and want add option in to kernel, how to?
I have my previous custom kernel (name is "Core"). In the "Core" i build some are needed options (pf, IPv6-disable and ...). So, how to options from "xen-kernel" integrate to my "Core" or as modify "xen-kernel" so as add options.
Thank you.


----------



## SirDice (Apr 29, 2019)

Handbook: 21.8. FreeBSD as a Xen™-Host


			FreeBSD/Xen - FreeBSD Wiki
		


Handbook: Chapter 8. Configuring the FreeBSD Kernel

Note that Xen on FreeBSD is still highly experimental and unstable. It also has a whole bunch of caveats.
Also note that Xen 4.7 is deprecated:

```
DEPRECATED: This port is about to be removed, please update to a newer Xen version
```


----------



## Phishfry (Apr 29, 2019)

I had Xen DOM0 up before moving to bhyve. I really didin't know what I was doing which made it harder.
The one tip I can give you is get `libvirtd` up and running before xen and then make a xen.xml for it.
There is a command console called virsh that is essential to getting all this running.
Once you do get it setup then you can use the gui VirtManager to make VM's.
It worked well once I figured it out. I was seriously disappointing by rudimentary I/O speeds. Net and Disk.
I never tried passthrough only virtio drivers.

I won't say it was a waste of time but it was.


----------



## zirias@ (Apr 29, 2019)

Indeed, did you consider bhyve instead? It's quite simple now using sysutils/vm-bhyve to manage your vms. If there isn't a specific requirement that only XEN can fullfill, I'd strongly recommend bhyve.

I had a server running XEN with Linux and LVM-based storage for the vms and completely replaced it with FreeBSD, jails and, where needed, bhyve with virtual disks in zvols.

There was just one little problem, windows guests didn't work with virtio-blk disks, but a small patch fixes that. If someone needs it, I can post it here when I'm at home.


----------



## Phishfry (Apr 29, 2019)

Looking back through the instructions SirDice posted I can see I went a different route.
I went into Virtualization wanting a GUI and libvirt is required for Virt-Manager.
So I went that tortured route. Nothing wrong with it as I liked the colored status bars for my VM's.

I did not know virsh came from KVM.




__





						Virsh Commands - Xen
					






					wiki.xen.org


----------



## Phishfry (Apr 29, 2019)

The manual is slightly off too as the newest xen kernel is now named this: emulators/xen-kernel/
Xen tools for that kernel now renamed sysutils/xen-tools/


----------



## Phishfry (Apr 29, 2019)

Just for clarity, there is no need to build a custom kernel for Xen. Download the ports and configure.

You do have to run in Legacy BIOS mode not UEFI.


----------



## andrian (Apr 30, 2019)

SirDice said:


> Handbook: 21.8. FreeBSD as a Xen™-Host
> 
> 
> FreeBSD/Xen - FreeBSD Wiki
> ...


Thank you. 
Yes, i read these but i am not about deploying XenAsHost.
 I ask about how to include in xen-kernel other option:

```
#options        INET6                   # IPv6 communications protocols
```
Because after restart for apply settings for Xen will be:

```
# sysrc -f /boot/loader.conf xen_kernel="/boot/xen"
```
This xen-kernel have support "INET6", how it disable.

You understand me? Please sorry me for my bad English (i study).


----------



## andrian (Apr 30, 2019)

Zirias said:


> Indeed, did you consider bhyve instead? It's quite simple now using sysutils/vm-bhyve to manage your vms. If there isn't a specific requirement that only XEN can fullfill, I'd strongly recommend bhyve.
> 
> I had a server running XEN with Linux and LVM-based storage for the vms and completely replaced it with FreeBSD, jails and, where needed, bhyve with virtual disks in zvols.
> 
> There was just one little problem, windows guests didn't work with virtio-blk disks, but a small patch fixes that. If someone needs it, I can post it here when I'm at home.


Thank you. "Bhyve" is nice idea. I read about sysutils/vm-bhyve  on FreshPorts and i see good words "BSD/Linux/Windows guest support".
But support guest for managment or support run this guest Operation System?


----------



## Phishfry (Apr 30, 2019)

andrian said:


> This xen-kernel have support "INET6", how it disable.


I see your dilemma now. The first place I looked was 'configuration options' for the xen-kernel port. None.
Have you dug thru the ports source? Compiling a port puts all the source on your computer.


----------



## andrian (Apr 30, 2019)

Phishfry said:


> I see your dilemma now. The first place I looked was 'configuration options' for the xen-kernel port. None.
> Have you dug thru the ports source? Compiling a port puts all the source on your computer.


Thank you. You very good gave me a hint. So, where is this file for add or remove kernel options?


----------



## Phishfry (Apr 30, 2019)

Well you need a ports tree.
Then `cd /usr/ports/emulators/xen-kernel`
Then you compile the port (make). It will download files from the internet and build the port.








						About FreeBSD Ports
					

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.




					www.freebsd.org
				



The /Files directory will contain the raw files. It could be a 'pre-compiled' kernel. I dunno. You need to look.

A different path to chose might be a sysctl. When I looked around everything was Linux based.
It seems xenbr0 has a sysctl to limit IPv6 on Linux.
So maybe dig around a see what sysctl's exist for FreeBSD Xen if any.


----------



## Phishfry (May 1, 2019)

This blog is where I found my instructions for FreeBSD dom0. As you can see it uses libvirt.




__





						libvirt/libxl on FreeBSD
					

Few months ago FreeBSD Xen dom0 support was announced. There's even a guide available how to run it: http://wiki.xen.org/wiki/FreeBSD_Dom0 ....




					empt1e.blogspot.com
				



You can control/disable INET6 with libvirt.
I saw instructions for that.


----------



## SirDice (May 1, 2019)

andrian said:


> You understand me?


Oh, right. You're asking about your own custom kernel to run _on_ Xen (as a Xen guest). 

You will need to build a custom kernel: 8.2. Why Build a Custom Kernel?



andrian said:


> So, how to options from "xen-kernel" integrate to my "Core" or as modify "xen-kernel" so as add options.


You need these options:

```
# Xen HVM Guest Optimizations
# NOTE: XENHVM depends on xenpci.  They must be added or removed together.
options         XENHVM                  # Xen HVM kernel infrastructure
device          xenpci                  # Xen HVM Hypervisor services driver
```

There's probably some more but I don't have access to Xen. Just start with a GENERIC kernel, and have a look at what's being detected. The GENERIC kernel should have everything included already.


----------



## Phishfry (May 1, 2019)

I did notice the difference between xen-kernel 4.12 (Built for FreeBSD 12) and xen-kernel 4.11(Built for FreeBSD 11)


----------



## zirias@ (May 1, 2019)

andrian said:


> Thank you. "Bhyve" is nice idea. I read about sysutils/vm-bhyve  on FreshPorts and i see good words "BSD/Linux/Windows guest support".
> But support guest for managment or support run this guest Operation System?


In case I understood this question correctly, here's the clarification you need:

`bhyve` supports all kinds of guest OS', definitely BSDs, Linux and Windows. As mentioned before, I had a problem with a Windows guest using `virtio-blk`, but this can be fixed with a simple patch.
sysutils/vm-bhyve provides management-tools for bhyve VMs, these tools are console/commandline tools, so they will run only on the machine actually hosting the bhyve VMs.


----------



## andrian (May 2, 2019)

hello everybody again. Thank all. I really liked it "bhyve" especially "supports all kinds of guest OS "!
I used to read about it before 1-2 years ago, but there was only support for Unix, so I forgot about Bhyve.
Very very good and simple. I have not fully understood it completely, but already fell in love with him.
Xen - bullshit.
Thank you forums.freebsd.org


----------



## andrian (May 2, 2019)

Zirias said:


> Indeed, did you consider bhyve instead? It's quite simple now using sysutils/vm-bhyve to manage your vms. If there isn't a specific requirement that only XEN can fullfill, I'd strongly recommend bhyve.
> 
> I had a server running XEN with Linux and LVM-based storage for the vms and completely replaced it with FreeBSD, jails and, where needed, bhyve with virtual disks in zvols.
> 
> There was just one little problem, windows guests didn't work with virtio-blk disks, but a small patch fixes that. If someone needs it, I can post it here when I'm at home.


I have stopped on "bhyve" (vmm load module) for studying so far. How to configure when to restart a virtual machine (windows) did not generate a process completion signal "0" and bhyve do not shutdown.

```
man bhyve
...
EXIT STATUS
     Exit status indicates how the VM was terminated:

     0       rebooted
     1       powered off
     2       halted
     3       triple fault
...
```
That is, I need to write a script to process these signals or is there another way I have not read it and did not know?


----------



## zirias@ (May 2, 2019)

I never used `bhyve` directly ... sysutils/vm-bhyve handles everything, shutdown and reboot just works.


----------



## zirias@ (May 2, 2019)

Regarding Windows guests, here's the patch that enables Windows to use `virtio-blk` disks:

```
Index: usr.sbin/bhyve/block_if.h
===================================================================
--- usr.sbin/bhyve/block_if.h    (revision 340722)
+++ usr.sbin/bhyve/block_if.h    (working copy)
@@ -41,7 +41,7 @@
#include <sys/uio.h>
#include <sys/unistd.h>

-#define BLOCKIF_IOV_MAX        33    /* not practical to be IOV_MAX */
+#define BLOCKIF_IOV_MAX        128    /* not practical to be IOV_MAX */

struct blockif_req {
     struct iovec    br_iov[BLOCKIF_IOV_MAX];
```

Without this, you must use `ahci-hd` drive emulation for Windows guests, which is considerably slower. So, better apply this patch (and build and install world from source).

Then it's even possible to install Windows directly on a `virtio-blk` disc. You have to additionally attach the .iso with the Windows virtio drivers during installation. You can download the drivers here: https://docs.fedoraproject.org/en-U...tual-machines-using-virtio-drivers/index.html


----------



## andrian (May 3, 2019)

Zirias said:


> Regarding Windows guests, here's the patch that enables Windows to use `virtio-blk` disks:
> 
> ```
> Index: usr.sbin/bhyve/block_if.h
> ...


Thank you.  My problem from Xen is solved - i will use sysutils/vm-bhyve. Really powerful!
And what about support with migration from others hypervisors (kvm or hyper-v) in to "vm-bhyve"?
And migration or live-migration in "vm-bhyve" two and more hosts?


----------



## zirias@ (May 3, 2019)

andrian said:


> And what about support with migration from others hypervisors (kvm or hyper-v) in to "vm-bhyve"?
> And migration or live-migration in "vm-bhyve" two and more hosts?


These are both features I don't need, so I can't tell you too much about them.

The first should be possible manually, it's mostly a matter of being able to convert the virtual disk and create a matching bhyve configuration.

As for the second, AFAIK live migration is a feature being worked on, but not yet available. You can always migrate a VM that's powered off, that's pretty comfortable when using a zvol for the virtual disk (zfs send/receive). If you want to minimize downtime, you could zfs send a snapshot from the running VM, then power it off and only send the diff using a second snapshot. But again, if I didn't miss anything, _live_ migration is not there yet, at least officially.


----------



## andrian (May 8, 2019)

Hi again. I need your help. I use vm-bhyve, it works perfectly. I have one options in config file which i do not understand. How his remove or disable?
My log run and work fine guest mashine:

```
May 08 14:15:49: initialising
May 08 14:15:49:  [loader: uefi]
May 08 14:15:50:  [cpu: 1,sockets=1,cores=1]
May 08 14:15:50:  [memory: 2G]
May 08 14:15:50:  [hostbridge: amd]
May 08 14:15:50:  [com ports: com1]
May 08 14:15:50:  [uuid: 3f4c7262-7165-11e9-8590-f4f26d026783]
May 08 14:15:50:  [utctime: no]
May 08 14:15:50:  [debug mode: no]
May 08 14:15:50:  [primary disk: win10.img]
May 08 14:15:50:  [primary disk dev: file]
May 08 14:15:50: initialising network device tap0
May 08 14:15:50: adding tap0 -> vm-ForVMS (ForVMS addm)
May 08 14:15:50: bring up tap0 -> vm-ForVMS (ForVMS addm)
May 08 14:15:50: booting
May 08 14:15:50:  [bhyve options: -c 1,sockets=1,cores=1 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -p 1:1 -U 3f4c7262-7165-11e9-8590-f4f26d026783]
May 08 14:15:50:  [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,ahci,hd:/virtmachins/win10/win10.img,nocache,cd:/virtmachins/win10/Windows10x64ukr-1803.iso -s 5:0,virtio-net,tap0,mac=58:9c:fc:01:bf:03 -s 6:0,fbuf,tcp=10.144.40.6:5999,w=800,h=600,vga=io -s 7:0,xhci,tablet]
May 08 14:15:50:  [bhyve console: -l com1,/dev/nmdm-win10.1A]
May 08 14:15:50:  [bhyve iso device: -s 3:0,ahci-cd,/virtmachins/.config/null.iso]
May 08 14:15:50: starting bhyve (run 1)
```

I want remove from guest mashine, how to?




My config file for this guest (do not have line about "...device -s 3:0...")

```
loader="uefi"
cpu="1"
cpu_sockets="1"
cpu_cores="1"
cpu_threads="1"
memory="2G"
wired_memory="no"
hostbridge="amd"
ignore_bad_msr="yes"
bhyve_options="-p 1:1"
utctime="no"
uuid="3f4c7262-7165-11e9-8590-f4f26d026783"
ahci_device_limit="8"
disk0_type="ahci-hd"
disk0_dev="file"
disk0_name="win10.img"
disk0_opts="nocache"
disk1_type="ahci-cd"
disk1_name="Windows10x64ukr-1803.iso"
network0_type="virtio-net"
network0_switch="ForVMS"
network0_device=""
network0_mac="58:9c:fc:01:bf:03"
network0_span="no"
passthru0=""
start_slot="4"
# install_slot
# The slot to use for an installation ISO. By default this is 3,
# which is the first available slot with the original UEFI firmware.
# Using this makes sure the ISO is the first device, and leaves
# 4-6 available for hd devices. Being able to change this may
# be useful for non-UEFI guests, especially if a passthru device
# requires this slot.
#
install_slot="3"
virt_random=""
graphics="yes"
graphics_port="5999"
graphics_listen="10.144.40.6"
graphics_res="800x600"
graphics_wait="auto"
graphics_vga="io"
xhci_mouse="yes"
```


----------



## SirDice (May 8, 2019)

Remove these two:

```
disk1_type="ahci-cd"
disk1_name="Windows10x64ukr-1803.iso"
```

You don't need to add a CD device for the install, it's automatically added when you use `vm install ...` and then automatically removed after the fist boot.









						Running Windows · churchers/vm-bhyve Wiki
					

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.




					github.com


----------



## andrian (May 8, 2019)

How to remove from system (May 08 14:15:50:  [bhyve iso device: -s 3:0,ahci-cd,/virtmachins/.config/null.iso])
ahci-cd disk drive - This let me have it so.


----------



## SirDice (May 8, 2019)

That's really a Windows support question. You can remove the device from the device manager. Right click on it and select 'uninstall device'.






						How to Uninstall Drivers in Windows
					

If your computer is acting up, hardware drivers may be to blame. Learn how to roll back or even uninstall driver software with this handy guide.




					www.pcworld.com


----------



## andrian (May 8, 2019)

SirDice said:


> Remove these two:
> 
> ```
> disk1_type="ahci-cd"
> ...











						Running Windows · churchers/vm-bhyve Wiki
					

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.




					github.com
				



There it is written:

If the machine has Internet access via the e1000 device, you can download and install the virtio drivers directly in the guest. Once installed, shutdown the guest, change the device in guest configuration and restart.
The guest can be booted in install mode, but specifying the VirtIO ISO file.
# vm install winguest virtio-installer.iso




A CD device can be added to the guest and pointed at the ISO file
disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/full/path/to/virtio-installer.iso"


----------



## andrian (May 8, 2019)

This is the drive that I want to delete is present in the system because there is a configuration file:

```
May 08 14:15:50: booting
May 08 14:15:50:  [bhyve options: -c 1,sockets=1,cores=1 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -p 1:1 -U 3f4c7262-7165-11e9-8590-f4f26d026783]
May 08 14:15:50:  [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,ahci,hd:/virtmachins/win10/win10.img,nocache,cd:/virtmachins/win10/Windows10x64ukr-1803.iso -s 5:0,virtio-net,tap0,mac=58:9c:fc:01:bf:03 -s 6:0,fbuf,tcp=10.144.40.6:5999,w=800,h=600,vga=io -s 7:0,xhci,tablet]
May 08 14:15:50:  [bhyve console: -l com1,/dev/nmdm-win10.1A]
*****
--->>> May 08 14:15:50:  [bhyve iso device: -s 3:0,ahci-cd,/virtmachins/.config/null.iso]
*****
May 08 14:15:50: starting bhyve (run 1)
```

but in config file have only line about "-s 4:0,ahci,hd:/virtmachins/win10/win10.img,nocache,cd:/virtmachins/win10/Windows10x64ukr-1803.iso"
...
ahci_device_limit="8"
disk0_type="ahci-hd"         <<<<<< ( -s 4:0,ahci,hd: .....
disk0_dev="file"
disk0_name="win10.img"
disk0_opts="nocache"
disk1_type="ahci-cd"         <<<<<< ( .... cd:
disk1_name="Windows10x64ukr-1803.iso"
network0_type="virtio-net"

...

Where is going from this "bhyve iso device: -s 3:0,ahci-cd,/virtmachins/.config/null.iso" i do not understand !!!!


----------



## andrian (May 8, 2019)

Zirias said:


> These are both features I don't need, so I can't tell you too much about them.


Please help me.


----------



## andrian (May 8, 2019)

SirDice said:


> Remove these two:
> 
> ```
> disk1_type="ahci-cd"
> ...


I made your recommendations, but they were so doomed.


----------



## SirDice (May 8, 2019)

Make sure you stop the VM; `vm stop ...`, then modify the configuration file. Then start the VM. If I recall correctly you need to stop, then start the VM for the configuration changes to take hold. Restarting the VM does not re-read the config.


----------



## andrian (May 8, 2019)

SirDice said:


> Make sure you stop the VM; `vm stop ...`, then modify the configuration file. Then start the VM. If I recall correctly you need to stop, then start the VM for the configuration changes to take hold. Restarting the VM does not re-read the config.



yes, i stop vm -> modify the configuration file and -> start vm again.

```
May 08 15:15:49: stopped
May 08 15:16:44: initialising
May 08 15:16:44:  [loader: uefi]
May 08 15:16:44:  [cpu: 1,sockets=1,cores=1]
May 08 15:16:44:  [memory: 2G]
May 08 15:16:44:  [hostbridge: amd]
May 08 15:16:44:  [com ports: com1]
May 08 15:16:44:  [uuid: 3f4c7262-7165-11e9-8590-f4f26d026783]
May 08 15:16:44:  [utctime: no]
May 08 15:16:44:  [debug mode: no]
May 08 15:16:44:  [primary disk: win10.img]
May 08 15:16:44:  [primary disk dev: file]
May 08 15:16:44: initialising network device tap0
May 08 15:16:44: adding tap0 -> vm-ForVMS (ForVMS addm)
May 08 15:16:44: bring up tap0 -> vm-ForVMS (ForVMS addm)
May 08 15:16:44: booting
May 08 15:16:44:  [bhyve options: -c 1,sockets=1,cores=1 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 3f4c7262-7165-11e9-8590-f4f26d026783]
****(see changed - not cd)
--->May 08 15:16:44:  [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,ahci,hd:/virtmachins/win10/win10.img,nocache -s 5:0,virtio-net,tap0,mac=58:9c:fc:01:bf:03 -s 6:0,fbuf,tcp=10.144.40.6:5999,w=800,h=600,vga=io -s 7:0,xhci,tablet]
****
May 08 15:16:44:  [bhyve console: -l com1,/dev/nmdm-win10.1A]
May 08 15:16:44:  [bhyve iso device: -s 3:0,ahci-cd,/virtmachins/.config/null.iso]
May 08 15:16:44: starting bhyve (run 1)
```

Removed only not that drive, but you have to give out another, as in the photo!


----------



## SirDice (May 8, 2019)

Did you double click an ISO file in Windows? Windows 8 and higher will create a virtual CD drive and load the ISO automatically. Just right click on the drive and select "eject", the virtual CD drive will disappear if it's empty.


----------



## zirias@ (May 10, 2019)

Clean up your config, there's a lot of cruft. "start_slot" and "install_slot" look suspicious, never needed this. My whole config for a windows server looks like this:

```
uefi="yes"
cpu=4
memory=8G
graphics="yes"
xhci_mouse="yes"
network0_type="virtio-net"
network0_switch="internal"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"

uuid="[.....]"
network0_mac="[.....]"
```
with the last two lines added by vm-bhyve on first run.

If you start in install mode, a dvd drive is added automatically, as told by sirdice.

Don't use virtio-blk for a windows guest unless you patched your bhyve as I already wrote here. ahci-hd is safe but slower.


----------

