# [Bhyve Virtualization] CentOS guest Display problem



## SilentSovereign (Nov 26, 2019)

I'm running FreeBSD 12.1-RELEASE-p1 and trying to host a CentOS VM (v8) with Bhyve.
When trying to do this with the MBR guide on the handbook bhyve page, I could not even get the VM working no matter what, I read that centos machines need UEFI booting with bhyve.
So I followed this guide UEFI bhyve  to do so.
Now my problem is that I managed to get the VM shell when running it through a VNC. But my goal is to be able to access it using nmdm consoles(cf 21.7.7. Virtual Machine Consoles).
After installing the VM through VNC with the UEFI bhyve guide, I changed the booting code to this :

```
bhyve -c 2 -m 2G -w -H \
        -s 0,hostbridge \
        -s 4,ahci-hd,centos8.img \
        -s 5,virtio-net,tap0 \
        -s 31,lpc -l com1,/dev/nmdm0A \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        vm1
```
Then I booted the vm and tried to connect to it from a terminal using this command `cu -l /dev/nmdm0B` but nothing is displayed.
Note : I managed to make this work(accessing with nmdm devices) for debian and ubuntu (while not using UEFI firmware pkg) but I can't seem to make it work for CentOS (using UEFI pkg because it didn't work otherwise).


----------



## SirDice (Nov 26, 2019)

I suspect you need to start the getty service on ttyS0 on the CentOS side; `systemctl start getty@ttyS0`. I suspect it's off by default and needs to be explicitly enabled.


----------



## SilentSovereign (Nov 27, 2019)

You're right that does it thx!, is there anyway to enable this in the VM booting step with bhyve ? If I can avoid using UEFI-GOP (Graphical UEFI Framebuffer) and VNC altogether that would be great because I want to run a freebsd host machine without a windows display manager in the end to run this guest VM.


----------



## toorski (Nov 27, 2019)

I have not  tried Centos, yet. But, Debian and FreeBSD install fine, as per FreeBSD handbook - MBR or  UEFI.

But now, I tried "vm-bhyve" to install OpenBSD-6.6, just for test.





						FreshPorts -- sysutils/vm-bhyve: Management system for bhyve virtual machines
					

A frontend for bhyve which provides the 'vm' command Create/start/stop virtual machines easily Bridged/NAT networking BSD/Linux/Windows guest support




					www.freshports.org
				




I was using following instructions:








						GitHub - churchers/vm-bhyve: Shell based, minimal dependency bhyve manager
					

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.




					github.com
				




It worked OK too.




Few things to note when using "vm-bhyve" utility after using basic "bhyve"

Firstly, you may need to comment out (as in my case) any references in /etc/rc.conf to a bridge and/or switch that were created for use with bhyve.

```
# cloned_interfaces="bridge0 tap0"
# ifconfig_bridge0="addm em0 addm tap0"
```

Follow vm-bhyve instructions on howto create switch, as in my case:



Secondly, look inside the template configuration files (in .templates dir) to make sure that settings in those files match your guest's OS version and specs.

Thirdly, you can download your fav. guest OS disrto (iso or img) to any location and then copy or move it to the .iso or .img  dirs of you vm for installation and set-up of the OS with vm-bhyve.

So, try vm-bhyve. It may work better for Centos.


----------



## SilentSovereign (Nov 27, 2019)

Tried it already for centos, can't seem to make it work, tried severals cfgs and always get stuck at installation somewhere


----------



## SirDice (Nov 27, 2019)

SilentSovereign said:


> is there anyway to enable this in the VM booting step with bhyve ?


No, I don't think so. You're probably going to need to create a custom install image that has the serial console enabled by default.


----------



## toorski (Nov 27, 2019)

SilentSovereign said:


> If I can avoid using UEFI-GOP (Graphical UEFI Framebuffer) and VNC altogether that would be great because I want to run a freebsd host machine without a windows display manager in the end to run this guest VM.



Try the "*boot.iso" image from there. I think that's non-GUI installer.


			Index of /centos/8.0.1905/isos/x86_64
		


ATM, I'm playing with plain "bhyve" and CentOS GUI installer as per:

"Support for the UEFI-GOP framebuffer may also be enabled with the -s 29,fbuf,tcp=_0.0.0.0:5900_ flags. The framebuffer resolution may be configured with w=_800_ and h=_600_, and bhyve can be instructed to wait for a VNC connection before booting the guest by adding wait. The framebuffer may be accessed from the host or over the network via the VNC protocol.
The resulting bhyve command would look like this:"


```
bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc -s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img -s 4:0,ahci-cd,./install.iso -c 4 -m 1024M -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd guest
```

Did you try this option with CentOS GUI installer?


----------



## toorski (Nov 28, 2019)

Got to this point with Centos in bhyve. After installation and reboot, I'm stuck in grub because I cannot find the boot image on the virtual hd 



These are my choices in grub, but I don't know where to go from here

```
grub> ls
(hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) (cd0) (cd0,msdos2) (host) (lvm/cl-root) (lvm/cl-home) (lvm/cl-swap)

grub> ls (hd0,gpt1) 
        Partition hd0,gpt1: Filesystem type fat, UUID BF26-D5E5 - Partition start at 2048 - Total size 1228800
sectors
```


----------



## SilentSovereign (Nov 29, 2019)

toorski said:


> Try the "*boot.iso" image from there. I think that's non-GUI installer.
> 
> 
> Index of /centos/8.0.1905/isos/x86_64
> ...


I've already managed to run it with VNC but my end goal is to to be able to run the vm installation and booting on a machine without needing to install xorg+xfce or smthg else which I won't need. That is why I'm trying to find a way to do the whole installation in serial console so in a "text" mode.


----------



## SilentSovereign (Nov 29, 2019)

toorski said:


> Got to this point with Centos in bhyve. After installation and reboot, I'm stuck in grub because I cannot find the boot image on the virtual hd
> 
> View attachment 7251
> 
> ...


If my memory serves me well I managed to boot from (hd0,gpt2) where the files vmlinuz-XXX and initramfs-XXX were available, just ls through the different partitions and u'll you'll find it I'm sure if ur your installation went on smoothly.


----------



## diego (Jul 19, 2020)

toorski said:


> Got to this point with Centos in bhyve. After installation and reboot, I'm stuck in grub because I cannot find the boot image on the virtual hd



Update July 2020

I have been playing for a while with bhyve for centos7 and centos8 with and without RNC connection. Finally I could run the latest version of with CentOS-8.2.2004 using just the console (not uefi) . The issue is how to select properly the grub partition for the linux kernel and root. Using vm-bhyve, this the template created for centos8:

./template/centos8.2.conf

```
loader="grub"
cpu=2
memory=2048M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

# Kernel from ISO --> centos-8.2.2004
grub_install0="linux /isolinux/vmlinuz"
grub_install1="initrd /isolinux/initrd.img"

#Kernel from disk --> centos-8.2.2004
grub_run0="linux /vmlinuz-4.18.0-193.el8.x86_64 root=/dev/mapper/cl-root"
grub_run1="initrd /initramfs-4.18.0-193.el8.x86_64.img"
```
If you install different version of centos8.x, you have to match the version of the kernel with your grub file (always) !!

```
[root@centos8 ~]# cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)
[root@centos8 ~]#
[root@centos8 ~]# neofetch
                 ..                    root@centos8
               .PLTJ.                  ------------
              <><><><>                 OS: CentOS Linux 8 (Core) x86_64
     KKSSV' 4KKK LJ KKKL.'VSSKK        Host: BHYVE 1.0
     KKV' 4KKKKK LJ KKKKAL 'VKK        Kernel: 4.18.0-193.el8.x86_64
     V' ' 'VKKKK LJ KKKKV' ' 'V        Uptime: 1 min
     .4MA.' 'VKK LJ KKV' '.4Mb.        Packages: 577 (rpm)
   . KKKKKA.' 'V LJ V' '.4KKKKK .      Shell: bash 4.4.19
.4D KKKKKKKA.'' LJ ''.4KKKKKKK FA.    Terminal: /dev/ttyS0
<QDD ++++++++++++  ++++++++++++ GFD>   CPU: Intel Pentium N3700 (2) @ 1.332GHz
'VD KKKKKKKK'.. LJ ..'KKKKKKKK FV     Memory: 135MiB / 1987MiB
   ' VKKKKK'. .4 LJ K. .'KKKKKV '
      'VK'. .4KK LJ KKA. .'KV'                                
     A. . .4KKKK LJ KKKKA. . .4                              
     KKA. 'KKKKK LJ KKKKK' .4KK
     KKSSA. VKKK LJ KKKV .4SSKK
              <><><><>
               'MKKM'
                 ''

[root@centos8 ~]#
```


----------



## Eric A. Borisch (Jun 4, 2021)

For anyone else running into pain trying to get text-only CentOS/RHEL/RockyLinux 8 to work (as the GRUB-based install doesn't like the resulting XFS partitions):

Use UEFI and modify the installer boot line to use the com port.

Works great with vm-bhyve, the key steps (from the above link, here for convenience) during booting of the install CD:

Go to the line Install CentOS Linux 8
Press e to edit the command line
Modify the vmlinuz line to replace quiet by console=ttyS0,115200n8
Hit CTRL-X to boot.
This also gets away from having to deal with the GRUB boot loader which is a little less clean inside Bhyve than just letting UEFI take care of it. You also don't have to do a two-step of use an old installer to get up-and-running.


----------

