# Xorg on FreeBSD based VM on bhyve



## Lars Wittebrood (Mar 18, 2019)

Hi,

I have a requirement (end goal) to run a FreeBSD based VM (as a guest) as a Xorg 'workstation' on the bhyve hypervisor. This 'workstation' will be accessed using a X server running on a Windows 8 or 10 laptop/pc. I use vm-bhyve as bhyve management tool. The bhyve hosts runs latest FreeBSD 11.2 version and vm-bhyve version 1.2.3 is used.

My understanding is that 

I have to boot my FreeBSD based VM using UEFI
I have to use the SCFB driver within my FreeBSD VM

I got this from

https://forums.freebsd.org/threads/xorg-under-bhyve-freebsd-11.58226/
https://wiki.freebsd.org/GraphicsOld/SCFB

I have 2 questions:

Is the above correct? Are these the requirements for my end goal? Is more and/or other things needed?
How do I run/boot a FreeBSD VM with UEFI using vm-bhyve? The vm-bhye wiki provides a Windows example, but not for FreeBSD as a guest. I've tried to use uefi="yes" in the VM config file, but then the VM does not boot

Thank you in advance for any pointers!

With kind regards,
Lars


----------



## SirDice (Mar 18, 2019)

Possible but probably not required. Just install a desktop and run any one of the VNC servers. You don't need to have a working "local" Xorg for VNC server to work. For remote X access ( through SSH X forwarding) you don't even need to have a complete desktop installed (the X server runs on your Windows desktop). Just start the application from the SSH command line and the application's GUI will magically appear on your Windows desktop.


----------



## abishai (Mar 18, 2019)

Actually, I've tried similar setup at home on weekend just to see how it works for Electron app.  (with CentOS@bhyve). I used net/xrdp. Probably, it's the best path if your workstations are Windows. For *NIX I'd choose SirDice method. Not sure if xrdp supports multiple sessions and properly handles disconnected one.


----------



## Lars Wittebrood (Mar 18, 2019)

Hi,
thank you both for the quick reply. Really appreciated.
I installed XOrg and the i3wm window manager on this VM. When I issue the `startx` command then I get the following error:


```
user@vmname:user $ startx
xauth:  file /home/user/.serverauth.647 does not exist

X.Org X Server 1.18.4
Release Date: 2016-07-19
X Protocol Version 11, Revision 0
Build Operating System: FreeBSD 11.2-RELEASE-p9 amd64
Current Operating System: FreeBSD wrkstn.intra.socruel.nu 11.2-RELEASE-p9 FreeBSD 11.2-RELEASE-p9 #0: Tue Feb  5 15:30:36 UTC 2019     [email]root@amd64-builder.daemonology.net[/email]:/usr/obj/usr/src/sys/GENERIC amd64
Build Date: 07 February 2019  09:44:50AM

Current version of pixman: 0.34.0
        Before reporting problems, check [URL]http://wiki.x.org[/URL]
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Mon Mar 18 20:15:45 2019
(==) Using config directory: "/usr/local/etc/X11/xorg.conf.d"
(EE)
Fatal server error:
(EE) xf86OpenConsole: No console driver found
        Supported drivers: pccons (with X support), syscons, pcvt
        Check your kernel's console driver configuration and /dev entries(EE)
(EE)
Please consult the The X.Org Foundation support
         at [URL]http://wiki.x.org[/URL]
 for help.
(EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.
xinit: giving up
xinit: unable to connect to X server: Connection refused
xinit: server error
```

When I search on the error message I do not get a definite answer. Most write that you have to use  x11-servers/xorg-vfbserver?


----------



## Ancient (Mar 18, 2019)

Which other options are apart of SSH? Just curiosity.


----------



## abishai (Mar 18, 2019)

Lars Wittebrood said:


> Hi,
> thank you both for the quick reply. Really appreciated.
> I installed XOrg and the i3wm window manager on this VM. When I issue the 'startx' command then I get the following error:


Xorg doesn't autodetect scfb driver, you need to provide a hint. Try to create /usr/local/etc/X11/xorg.conf.d/video.conf and put the following into it

```
Section "Device"
    Identifier "Card0"
    Driver     "scfb"
EndSection
```


----------



## Lars Wittebrood (Mar 19, 2019)

Hi,
thank you for your responses both.
Regarding sshd_config, see the added config options to the FreeBSD default sshd_config:

```
ListenAddress 10.26.16.199
AllowGroups sshlogin
Protocol 2
X11Forwarding yes
X11UseLocalhost no
IgnoreRhosts yes
PermitEmptyPasswords no
PermitRootLogin no
ChallengeResponseAuthentication no
# From Lynis
#AllowAgentForwarding no
UseDNS no
MaxSessions 2
MaxAuthTries 2
LogLevel VERBOSE
Compression no
ClientAliveCountMax 2
#AllowTcpForwarding no
```

I have a config for the scfb driver:

```
user@vmname:user $ less /usr/local/etc/X11/xorg.conf.d/driver-scfb.conf
Section "Device"
     Identifier    "Card0"
     Driver        "scfb"
EndSection
```

I've also tried to do:
`sudo X -configure`
And then I get the following output:

```
X.Org X Server 1.18.4
Release Date: 2016-07-19
X Protocol Version 11, Revision 0
Build Operating System: FreeBSD 11.2-RELEASE-p9 amd64
Current Operating System: FreeBSD wrkstn.intra.socruel.nu 11.2-RELEASE-p9 FreeBSD 11.2-RELEASE-p9 #0: Tue Feb  5 15:30:36 UTC 2019     [email]root@amd64-builder.daemonology.net[/email]:/usr/obj/usr/src/sys/GENERIC amd64
Build Date: 07 February 2019  09:44:50AM

Current version of pixman: 0.34.0
        Before reporting problems, check [URL]http://wiki.x.org[/URL]
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Tue Mar 19 07:03:56 2019
List of video drivers:
        vesa
        modesetting
        scfb
scfb trace: probe start
No devices to configure.  Configuration failed.
(EE) Server terminated with error (2). Closing log file.
```


----------



## Phishfry (Mar 19, 2019)

There is no video device available from a bhyve vm.


			bhyve/pci_passthru - FreeBSD Wiki
		

See the note there>>> *Note:* VGA / GPU pass-through devices are not currently supported. <<<

You can load the scfb driver all day long. There is no video device for it to attach to.
A video device would need to be passed-thru to the VM and that is not supported.

Now for X11Forwarding you need no video driver at all. Just xorg-minimal.
If you must install a video driver for some reason there is x11-drivers/xf86-video-dummy for this purpose.
Remember that X11-Forwarding only forwards applications, not the whole desktop.


----------



## SirDice (Mar 19, 2019)

Phishfry said:


> You can load the scfb driver all day long. There is no video device for it to attach to.


I beg to differ. 


```
------------------------
Virtual Machine: lady3jane
------------------------
  state: stopped
  datastore: stor10k
  loader: bhyveload
  uuid: f31dc44e-f1c9-11e8-8acb-002590f15838
  uefi: yes
  cpu: 4
  memory: 4096M
  memory-resident: 0 (0.000B)

  console-ports
    com1: /dev/nmdm-lady3jane.1B
    vnc: 0.0.0.0:5900
```


```
root@hosaka:~ # cat /storage/vm/lady3jane/lady3jane.conf
uefi="yes"
graphics="yes"
graphics_port="5900"
graphics_res="1920x1200"
graphics_wait="yes"
#xhci_mouse="yes"
utctime="yes"
loader="bhyveload"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
#zfs_dataset_opts="compress=off"
zfs_zvol_opts="volblocksize=4k compress=off"
uuid="f31dc44e-f1c9-11e8-8acb-002590f15838"
network0_mac="58:9c:fc:00:c5:50"
```

Mouse works weird with TigerVNC client on Windows. If I recall correctly it should work better with the RealVNC client.


----------



## Phishfry (Mar 19, 2019)

Well your video device in this case is a VNC server.
Your not going to get a desktop with stdio.


----------



## SirDice (Mar 19, 2019)

No, from the VM's viewpoint the video device is a UEFI framebuffer device. There is no VNC server running on the VM. The VM doesn't even know it's not a "real" graphics card. It's bhyve that 'translates' the EUFI-GOP to VNC. 









						UEFI Graphics (VNC) · churchers/vm-bhyve Wiki
					

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.




					github.com
				





			bhyve/UEFI - FreeBSD Wiki
		









						UEFI - Wikipedia
					






					en.wikipedia.org


----------



## abishai (Mar 19, 2019)

Lars Wittebrood said:


> I have a config for the scfb driver:
> 
> ```
> user@vmname:user $ less /usr/local/etc/X11/xorg.conf.d/driver-scfb.conf
> ...




Are you sure you use UEFI loader ?
Start X from vnc session.

Please provide xorg logs with this config.


----------



## abishai (Mar 19, 2019)

Phishfry said:


> Well your video device in this case is a VNC server.
> Your not going to get a desktop with stdio.


SirDice is right.
Here is Xorg startup from bhyve vnc console, specially for you 

```
[   413.218] (II) LoadModule: "scfb"
[   413.218] (II) Loading /usr/local/lib/xorg/modules/drivers/scfb_drv.so
[   413.218] (II) Module scfb: vendor="X.Org Foundation"
[   413.218]    compiled for 1.18.4, module version = 0.0.4
[   413.218]    ABI class: X.Org Video Driver, version 20.0
[   413.218] (II) scfb: driver for wsdisplay framebuffer: scfb
[   413.218] (--) Using syscons driver with X support (version 2.0)
[   413.218] (--) using VT number 9

[   413.219] (WW) Falling back to old probe method for scfb
[   413.219] scfb trace: probe start
[   413.219] (II) scfb(0): using default device
[   413.219] scfb trace: probe done
[   413.219] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support
[   413.219] scfb: PreInit 0
[   413.219] (II) scfb(0): Using: depth (32),   width (1024),    height (768)
[   413.219] (II) scfb(0): Creating default Display subsection in Screen section
        "Default Screen Section" for depth/fbbpp 24/32
[   413.219] (==) scfb(0): Depth 24, (==) framebuffer bpp 32
[   413.219] (==) scfb(0): RGB weight 888
[   413.219] (==) scfb(0): Default visual is TrueColor
[   413.219] (==) scfb(0): Using gamma correction (1.0, 1.0, 1.0)
[   413.219] (II) scfb(0): Vidmem: 3072k
[   413.219] (==) scfb(0): DPI set to (96, 96)
[   413.219] (**) scfb(0): Using "Shadow Framebuffer"
[   413.219] (II) Loading sub module "shadow"
[   413.219] (II) LoadModule: "shadow"
[   413.219] (II) Loading /usr/local/lib/xorg/modules/libshadow.so
[   413.219] (II) Module shadow: vendor="X.Org Foundation"
[   413.219]    compiled for 1.18.4, module version = 1.1.0
[   413.219]    ABI class: X.Org ANSI C Emulation, version 0.4
[   413.220] (II) Loading sub module "fb"
[   413.220] (II) LoadModule: "fb"
[   413.220] (II) Loading /usr/local/lib/xorg/modules/libfb.so
[   413.220] (II) Module fb: vendor="X.Org Foundation"
[   413.220]    compiled for 1.18.4, module version = 1.0.0
[   413.220]    ABI class: X.Org ANSI C Emulation, version 0.4
[   413.220] scfb: PreInit done
[   413.220] (--) Depth 24 pixmap format is 32 bpp
[   413.220] scfb: ScfbScreenInit 0
[   413.220]    bitsPerPixel=32, depth=24, defaultVisual=TrueColor
        mask: ff0000,ff00,ff, offset: 16,8,0
[   413.220] mmap returns: addr 0x806a00000 len 0x300000, fd 6, off 0
[   413.220] scfb: ScfbSave 0
[   413.220] scfb: ScfbSave done
[   413.221] (==) scfb(0): Backing store enabled
[   413.221] scfb: ScfbScreenInit done
```


----------



## Lars Wittebrood (Mar 20, 2019)

Hi,
first thank you for all the pointers above!
I had a couple of wrong assumptions as it seems. I assumed that you could display a whole desktop remotely using X. I now understand why that is not possible. Starting X applications like 'xclock' just works in my existing configuration.
I still want to display a whole desktop remotely (I want to test the i3wm window manager). So I tested what SirDice showed above. Thanks for that, I assumed that you could not use 'loader="bhyveload" and uefi="yes"' together in a vm-bhyve vm config!). So I created a new VM with UEFI on my bhyve box. Bhyve related packages installed on my bhyve box:


```
user@bhyvebox:user $ pkg info | grep bhyve
bhyve-firmware-1.0_1           Collection of Firmware for bhyve
grub2-bhyve-0.40_6             Grub-emu loader for bhyve
uefi-edk2-bhyve-0.2_1,1        UEFI-EDK2 firmware for bhyve
uefi-edk2-bhyve-csm-0.2_1,1    UEFI-EDK2 firmware for bhyve with CSM
vm-bhyve-1.2.3                 Management system for bhyve virtual machines
```

The vm-bhyve commands to create my new vm:

```
# vm create -s 128G -t freebsd-uefi test
# vm install test FreeBSD-11.2-RELEASE-amd64-dvd1.iso
# tmux attach -t test
```

It just shows:

```
[B]__[/B]               [B]__   ___[/B] [B]_[/B]
 |  [B]__|             |  _ \ / __[/B]|  __ \
 | |[B]_ _ __ _[/B]  [B]_ | |[I]) | ([/I][/B] | |  | |
 |  [B]_| '[/B]/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  [B]/  [B]/| |_) |[/B][/B]) | |__| |
 | |   | | |    |    ||     |      |      |
 |[I]|   |[/I]|  \[B][I]|\[/I][/B]||[B][B]/|_[/B][/B]/|[B]_[/B]/    ``[ICODE]                        [/ICODE]
                                             s[ICODE] [/ICODE].....---.......--.``[ICODE]   -/
 ┌────────────Welcome to FreeBSD───────────┐ +o   .--[/ICODE]         /y:[ICODE]      +.
 │                                         │  yo[/ICODE]:.            :o      [ICODE]+-
 │  1. Boot Multi User [Enter]             │   y/               -/[/ICODE]   -o/
 │  2. Boot Single User                    │  .-                  ::/sy+:.
 │  3. Escape to loader prompt             │  /                     [ICODE]--  /
 │  4. Reboot                              │ [/ICODE]:                          :[ICODE]
 │                                         │ [/ICODE]:                          :[ICODE]
 │  Options:                               │  /                          /
 │  5. Kernel: default/kernel (1 of 2)     │  .-                        -.
 │  6. Configure Boot Options...           │   --                      -.
 │                                         │    [/ICODE]:[ICODE]                  [/ICODE]:[ICODE]
 │                                         │      .--             [/ICODE]--.
 │                                         │         .---.....----.
 └─────────────────────────────────────────┘

-
/boot/kernel/kernel text=0x1547b08 data=0x143f30+0x4bc418 syms=[0x8+0x16ad00+0x8
+0x183cac]
Booting...
Start @ 0xffffffff80310000 ...
-
```

The logfile of the VM shows:


```
root@bhyvebox:/virtuals/vms/test # less vm-bhyve.log
Mar 20 07:07:32: initialising
Mar 20 07:07:32:  [loader: bhyveload]
Mar 20 07:07:32:  [uefi: yes]
Mar 20 07:07:32:  [cpu: 2]
Mar 20 07:07:32:  [memory: 8G]
Mar 20 07:07:32:  [hostbridge: standard]
Mar 20 07:07:32:  [com ports: com1]
Mar 20 07:07:32:  [uuid: 6d230680-4ad6-11e9-b402-6805ca4d6b32]
Mar 20 07:07:32:  [utctime: yes]
Mar 20 07:07:32:  [debug mode: no]
Mar 20 07:07:32:  [primary disk: disk0]
Mar 20 07:07:32:  [primary disk dev: sparse-zvol]
Mar 20 07:07:32: generated static mac 58:9c:fc:0e:8b:b6 (based on 'test:0:1553062052:0')
Mar 20 07:07:32: initialising network device tap12
Mar 20 07:07:32: adding tap12 -> vm-intranet (intranet addm)
Mar 20 07:07:32: bring up tap12 -> vm-intranet (intranet addm)
Mar 20 07:07:32: booting
Mar 20 07:07:32: bhyveload -m 8G -e autoboot_delay=3 -d /virtuals/vms/.iso/FreeBSD-11.2-RELEASE-amd64-dvd1.iso test
Mar 20 07:07:35:  [bhyve options: -c 2 -m 8G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 6d230680-4ad6-11e9-b402-6805ca4d6b32 -u]
Mar 20 07:07:35:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/dev/zvol/virtuals/vms/test/disk0 -s 5:0,virtio-net,tap12,mac=58:9c:fc:0e:8b:b6]
Mar 20 07:07:35:  [bhyve console: -l com1,stdio]
Mar 20 07:07:35:  [bhyve iso device: -s 3:0,ahci-cd,/virtuals/vms/.iso/FreeBSD-11.2-RELEASE-amd64-dvd1.iso]
Mar 20 07:07:35: starting bhyve (run 1)
```


My freebsd-uefi.conf vm-bhyve template file looks like:

```
guest="freebsd"
loader="bhyveload"
uefi="yes"
#graphics="yes"
#graphics_port="5900"
#graphics_res="1024x768"
#graphics_wait="yes"
cpu=2
memory=8G
network0_type="virtio-net"
network0_switch="intranet"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
```

Using this will NOT boot this VM. I've read that the FreeBSD installer ISOs are UEFI aware (?). So I am doing something wrong here. But for now I don't know what. But then again this whole UEFI stuff is new to me.

Another option I can try is using XRDP on top of Xorg on my existing VM. 

Cheers,
Lars


----------



## abishai (Mar 20, 2019)

You indeed doing it wrong. Remove loader= line entirely.
Start it with
vm install test FreeBSD-11.2-RELEASE-amd64-dvd1.iso
Check for opened 5900 port on bhyve host and connect with VNC client.


----------



## Lars Wittebrood (Mar 22, 2019)

Hi,
all (seems) ok now. I can start a FreeBSD VM installation using a VNC connection to the VM. Had no time yet to finish this. But sure it will be ok and have a UEFI based VM after finishing the install.
Thanks all for helping me and providing the opportunity to learn about this all.
Cheers,
Lars


----------

