# Bhyve failing with VMX detection



## FrostKiwi (Jun 17, 2021)

Hey there :]
Similar to this thread "grub-bhyve -m device.map -r cd0 -M 2048M linuxguest" reports back

```
Could not create VM linuxguest
Error in initializing VM
```
With the kernel complaining:

```
Jun 17 20:49:32 stripes kernel: vmx_modinit: processor does not support desired primary processor-based controls
Jun 17 20:49:32 stripes kernel: module_register_init: MOD_LOAD (vmm, 0xffffffff82604030, 0) error 22
```
My laptop (Intel Core2Duo P8800) has the Coreboot firmware flashed, so a bios is not technically present. In the Laptops's firmware VMX is enabled and detected by FreeBSD and VT-X is listed in DMESG. 
Often concerning Virtualization is how the I32 Lock bit is set. Reading the corresponding MSR with cpuctl returns:


> MSR 0x3a: 0x00000000 0x0000000f


HEX F is 1111 in binary, so bit's 0, 1, and 2 are set, Virtualization should work if I understand correctly.

So in contrast to that thread, the firmware has virtualization enabled. What am I missing? Why is bhyve failing?

Full DMESG for completness sake: 


```
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-RELEASE #2 n244733-ea31abc261f-dirty: Thu Jun 17 10:30:10 CEST 2021
    root@stripes:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
VT(vga): resolution 640x480
CPU: Intel(R) Core(TM)2 Duo CPU     P8800  @ 2.66GHz (2666.83-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xc08e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,OSXSAVE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  VT-x: HLT,PAUSE
  TSC: P-state invariant, performance statistics
real memory  = 10401873920 (9920 MB)
avail memory = 7883137024 (7517 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <COREv4 COREBOOT>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 1
Timecounter "TSC-low" frequency 1333413135 Hz quality 1000
KTLS: Initialized 2 threads
random: entropy device external interface
000.000019 [4354] netmap_init               netmap: loaded module
[ath_hal] loaded
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0.
kbd1 at kbdmux0
mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020)
nexus0
vtvga0: <VT VGA driver>
cryptosoft0: <software crypto>
aesni0: No AES or SHA support.
acpi0: <COREv4 COREBOOT>
acpi0: Power Button (fixed)
ACPI Error: No handler for Region [ERAM] (0xfffff800038d7780) [EmbeddedControl] (20201113/evregion-290)
ACPI Error: Region EmbeddedControl (ID=3) has no handler (20201113/exfldio-428)
ACPI Error: Aborting method \134_SB.PCI0.LPCB.EC.BAT0._STA due to previous error (AE_NOT_EXIST) (20201113/psparse-689)
ACPI Error: No handler for Region [ERAM] (0xfffff800038d7780) [EmbeddedControl] (20201113/evregion-290)
ACPI Error: Region EmbeddedControl (ID=3) has no handler (20201113/exfldio-428)
ACPI Error: Aborting method \134_SB.PCI0.LPCB.EC.BAT1._STA due to previous error (AE_NOT_EXIST) (20201113/psparse-689)
ACPI Error: No handler for Region [ERAM] (0xfffff800038d7780) [EmbeddedControl] (20201113/evregion-290)
ACPI Error: Region EmbeddedControl (ID=3) has no handler (20201113/exfldio-428)
ACPI Error: Aborting method \134_SB.DOCK._STA due to previous error (AE_NOT_EXIST) (20201113/psparse-689)
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
acpi_ec0: <Embedded Controller: GPE 0x11> port 0x62,0x66 on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x1100-0x1107 mem 0x95000000-0x953fffff,0xa0000000-0xafffffff irq 16 at device 2.0 on pci0
agp0: <Intel GM45 SVGA controller> on vgapci0
WARNING: Device "agp" is Giant locked and may be deleted before FreeBSD 14.0.
agp0: aperture size is 256M, detected 360444k stolen memory
vgapci0: Boot video device
vgapci1: <VGA-compatible display> mem 0x95400000-0x954fffff at device 2.1 on pci0
em0: <Intel(R) PRO/1000 Network Connection> port 0x1000-0x101f mem 0x94000000-0x9401ffff,0x95604000-0x95604fff irq 16 at device 25.0 on pci0
em0: Setup of Shared code failed, error -6
em0: IFDI_ATTACH_PRE failed 6
device_attach: em0 attach returned 6
uhci0: <Intel 82801I (ICH9) USB controller> port 0x1020-0x103f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x3f00
uhci_interrupt: host controller process error
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801I (ICH9) USB controller> port 0x1040-0x105f irq 17 at device 26.1 on pci0
uhci_interrupt: host controller process error
uhci1: LegSup = 0x3f00
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801I (ICH9) USB controller> port 0x1060-0x107f irq 18 at device 26.2 on pci0
uhci_interrupt: host controller process error
uhci2: LegSup = 0x3f00
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x95606000-0x956063ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
hdac0: <Intel 82801I HDA Controller> mem 0x95600000-0x95603fff irq 16 at device 27.0 on pci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
iwn0: <Intel Ultimate N WiFi Link 5300> mem 0x95500000-0x95501fff irq 17 at device 0.0 on pci2
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pcib4: failed to allocate initial prefetch window: 0x94000000-0x947fffff
pci4: <ACPI PCI bus> on pcib4
uhci3: <Intel 82801I (ICH9) USB controller> port 0x1080-0x109f irq 16 at device 29.0 on pci0
uhci_interrupt: host controller process error
uhci3: LegSup = 0x3f00
usbus4 on uhci3
usbus4: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801I (ICH9) USB controller> port 0x10a0-0x10bf irq 17 at device 29.1 on pci0
uhci_interrupt: host controller process error
uhci4: LegSup = 0x3f00
usbus5 on uhci4
usbus5: 12Mbps Full Speed USB v1.0
uhci5: <Intel 82801I (ICH9) USB controller> port 0x10c0-0x10df irq 18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6 on uhci5
usbus6: 12Mbps Full Speed USB v1.0
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x95607000-0x956073ff irq 16 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
usbus7: 480Mbps High Speed USB v2.0
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH9M AHCI SATA controller> port 0x1108-0x110f,0x1118-0x111b,0x1110-0x1117,0x111c-0x111f,0x10e0-0x10ff mem 0x95605000-0x956057ff irq 17 at device 31.2 on pci0
ahci0: AHCI v1.20 with 4 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
acpi_acad0: <AC Adapter> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IBM/Lenovo TrackPoint, device ID 14
acpi_acad1: <AC Adapter> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
battery1: <ACPI Control Method Battery> on acpi0
acpi_button0: <Sleep Button> on acpi0
acpi_lid0: <Control Method Lid Switch> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
Timecounters tick every 1.000 msec
hdacc0: <Conexant CX20561 (Hermosa) HDA CODEC> at cad 0 on hdac0
hdaa0: <Conexant CX20561 (Hermosa) Audio Function Group> at nid 1 on hdacc0
ugen7.1: <Intel EHCI root HUB> at usbus7
ugen3.1: <Intel EHCI root HUB> at usbus3
ugen6.1: <Intel UHCI root HUB> at usbus6
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen5.1: <Intel UHCI root HUB> at usbus5
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen0.1: <Intel UHCI root HUB> at usbus0
uhub0 on usbus7
uhub1 on usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub2 on usbus3
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub3 on usbus6
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub4 on usbus1
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub5 on usbus2
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub6 on usbus4
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub7 on usbus5
uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
pcm0: <Conexant CX20561 (Hermosa) (Analog 2.0+HP/2.0)> at nid 26,22 and 24 on hdaa0
pcm1: <Conexant CX20561 (Hermosa) (Internal Analog Mic)> at nid 29 on hdaa0
unknown: <Conexant CX20561 (Hermosa) HDA CODEC Modem Function Group> at nid 2 on hdacc0 (no driver attached)
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Samsung SSD 860 EVO 500GB RVT03B6Q> ACS-4 ATA SATA 3.x device
ada0: Serial Number S4XBNF0M917888K
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors)
em0: <Intel(R) PRO/1000 Network Connection> port 0x1000-0x101f mem 0x94000000-0x9401ffff,0x95604000-0x95604fff irq 16 at device 25.0 on pci0
GEOM_ELI: Device ada0.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:26:2d:fd:71:0c
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
em0: netmap queues/slots: TX 1/1024, RX 1/1024
Trying to mount root from ufs:ada0.eli []...
random: unblocking device.
ichsmb0: <Intel 82801I (ICH9) SMBus controller> port 0x400-0x41f mem 0x95608000-0x956080ff irq 18 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
drmn0: <drmn> on vgapci0
vgapci0: child drmn0 requested pci_enable_io
vgapci0: child drmn0 requested pci_enable_io
[drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19).
Successfully added WC MTRR for [0xa0000000-0xafffffff]: 0;
[drm] Got stolen memory base 0x7e000000, size 0x16000000
[drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[drm] Driver supports precise vblank timestamp query.
[drm] Connector LVDS-1: get mode from tunables:
[drm]   - kern.vt.fb.modes.LVDS-1
[drm]   - kern.vt.fb.default_mode
[drm] Connector VGA-1: get mode from tunables:
[drm]   - kern.vt.fb.modes.VGA-1
[drm]   - kern.vt.fb.default_mode
[drm] Connector HDMI-A-1: get mode from tunables:
[drm]   - kern.vt.fb.modes.HDMI-A-1
[drm]   - kern.vt.fb.default_mode
[drm] Connector DP-1: get mode from tunables:
[drm]   - kern.vt.fb.modes.DP-1
[drm]   - kern.vt.fb.default_mode
[drm] Connector HDMI-A-2: get mode from tunables:
[drm]   - kern.vt.fb.modes.HDMI-A-2
[drm]   - kern.vt.fb.default_mode
[drm] Connector DP-2: get mode from tunables:
[drm]   - kern.vt.fb.modes.DP-2
[drm]   - kern.vt.fb.default_mode
[drm] Connector DP-3: get mode from tunables:
[drm]   - kern.vt.fb.modes.DP-3
[drm]   - kern.vt.fb.default_mode
[drm] RC6 disabled, disabling runtime PM support
sysctl_warn_reuse: can't re-use a leaf (hw.dri.debug)!
[drm] Initialized i915 1.6.0 20190822 for drmn0 on minor 0
uhub0: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
WARNING: Device "fb" is Giant locked and may be deleted before FreeBSD 14.0.
VT: Replacing driver "vga" with new "fb".
start FB_INFO:
type=11 height=800 width=1280 depth=32
cmsize=16 size=4096000
pbase=0xa000e000 vbase=0xfffff800a000e000
name=drmn0 flags=0x0 stride=5120 bpp=32
cmap[0]=0 cmap[1]=7f0000 cmap[2]=7f00 cmap[3]=c4a000
end FB_INFO
drmn0: fb0: i915drmfb frame buffer device
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
wlan0: Ethernet address: 00:21:6a:a9:45:de
lo0: link state changed to UP
ugen1.2: <Lenovo Computer Corp ThinkPad Bluetooth with Enhanced Data Rate II> at usbus1
ubt0 on uhub4
ubt0: <Lenovo Computer Corp ThinkPad Bluetooth with Enhanced Data Rate II, class 224/1, rev 2.00/3.99, addr 2> on usbus1
WARNING: attempt to domain_add(bluetooth) after domainfinalize()
WARNING: attempt to domain_add(netgraph) after domainfinalize()
acpi_ibm0: <ThinkPad ACPI Extras> on acpi0
acpi_ibm0: Firmware version is 0x100
uhub1: detached
ugen0.1: <Intel UHCI root HUB> at usbus0 (disconnected)
unknown: at usbus0, port 1, addr 1 (disconnected)
usbus0: detached
uhci0: detached
pci0: <serial bus, USB> at device 26.0 (no driver attached)
ugen1.2: <Lenovo Computer Corp ThinkPad Bluetooth with Enhanced Data Rate II> at usbus1 (disconnected)
ubt0: at uhub4, port 2, addr 2 (disconnected)
ubt0: detached
uhub4: detached
ugen1.1: <Intel UHCI root HUB> at usbus1 (disconnected)
unknown: at usbus1, port 1, addr 1 (disconnected)
usbus1: detached
uhci1: detached
pci0: <serial bus, USB> at device 26.1 (no driver attached)
uhub5: detached
ugen2.1: <Intel UHCI root HUB> at usbus2 (disconnected)
unknown: at usbus2, port 1, addr 1 (disconnected)
usbus2: detached
uhci2: detached
pci0: <serial bus, USB> at device 26.2 (no driver attached)
uhub6: detached
ugen4.1: <Intel UHCI root HUB> at usbus4 (disconnected)
unknown: at usbus4, port 1, addr 1 (disconnected)
usbus4: detached
uhci3: detached
pci0: <serial bus, USB> at device 29.0 (no driver attached)
uhub7: detached
ugen5.1: <Intel UHCI root HUB> at usbus5 (disconnected)
unknown: at usbus5, port 1, addr 1 (disconnected)
usbus5: detached
uhci4: detached
pci0: <serial bus, USB> at device 29.1 (no driver attached)
uhub3: detached
ugen6.1: <Intel UHCI root HUB> at usbus6 (disconnected)
unknown: at usbus6, port 1, addr 1 (disconnected)
usbus6: detached
uhci5: detached
pci0: <serial bus, USB> at device 29.2 (no driver attached)
iwn0: iwn_read_firmware: ucode rev=0x08530501
wlan0: link state changed to UP
vmx_modinit: processor does not support desired primary processor-based controls
module_register_init: MOD_LOAD (vmm, 0xffffffff82604030, 0) error 22
```


----------



## Alain De Vos (Jun 17, 2021)

About early core processors,


			bhyve - FreeBSD Wiki


----------



## FrostKiwi (Jun 17, 2021)

Alain De Vos said:


> About early core processors,
> 
> 
> bhyve - FreeBSD Wiki


Thanks for the reference :]
So, my Processor does not support EPT I guess, as sugested by this section, though the error messages are not the same. So I presume the VMM kernel module is a no go.
However, this gives me hope:


> early "Core" processors may only support one virtual machine


That sounds fine. I only need one. Virtual machines without VT-x are always possible, though slow.
On Linux KVM works as far as I recall. Is it possible to still to get a VM on FreeBSD accelerated by VT-x?


----------



## jgrafton (Jun 18, 2021)

I've got a FreeBSD 13 system from roughly that time frame (2009):
`Intel(R) Xeon(R) CPU E5506  @ 2.13GHz (2128.05-MHz K8-class CPU`

It will boot a FreeBSD VM with just a single CPU.  Just have to modify command from the wiki:
What is the easiest way to try bhyve

This works on my system:
`sudo sh /usr/share/examples/bhyve/vmrun.sh -c 1 -d FreeBSD-12.0-RELEASE-amd64-memstick.img vm0`


----------



## SirDice (Jun 18, 2021)

FrostKiwi said:


> ```
> CPU: Intel(R) Core(TM)2 Duo CPU P8800 @ 2.66GHz (2666.83-MHz K8-class CPU) Origin="GenuineIntel" Id=0x1067a Family=0x6 Model=0x17 Stepping=10 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> Features2=0xc08e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,OSXSAVE>
> AMD Features=0x20100800<SYSCALL,NX,LM>
> ...


POPCNT seems to be missing.


----------



## SirDice (Jun 18, 2021)

FrostKiwi said:


> Is it possible to still to get a VM on FreeBSD accelerated by VT-x?


Not with bhyve(8) as your CPU seems to be lacking a required feature (POPCNT). VirtualBox might work though. 



jgrafton said:


> I've got a FreeBSD 13 system from roughly that time frame (2009):
> `Intel(R) Xeon(R) CPU E5506 @ 2.13GHz (2128.05-MHz K8-class CPU`


Mine's slightly newer:

```
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.05-MHz K8-class CPU)
```



jgrafton said:


> It will boot a FreeBSD VM with just a single CPU.


I'm pretty sure you can use more vCPUs.



jgrafton said:


> This works on my system:


That's a rather moot point if vmm(4) can't be loaded on the OP's system.


----------



## jgrafton (Jun 18, 2021)

SirDice said:


> That's a rather moot point if vmm(4) can't be loaded on the OP's system.


Haha, good point!

The POPCNT requirement is interesting since bhyve itself doesn't seem to actually *use* the op code as far as I can tell.  It was introduced with the Intel Nehalem micro architecture along with the VT-x and EPT extensions which are absolutely required.

This link has some interesting info I never knew about the POPCNT op code:  You Won’t Believe This One Weird CPU Instruction!


----------



## PMc (Jun 18, 2021)

SirDice said:


> I'm pretty sure you can use more vCPUs




```
VT-x: UG
```
If that is not present, there can be only one.


----------



## jgrafton (Jun 19, 2021)

PMc said:


> ```
> VT-x: UG
> ```
> If that is not present, there can be only one.


Yup, no unrestricted guest support on my processor.


----------



## FrostKiwi (Jun 19, 2021)

So to sum the knowledge in this thread:

to use FreeBSD's bhyve, loading the *vmm* kernel module is a hard requirement
Proccesors not supporting the EPT VT-x extensions (e.g. Intel Proccessors before Nehalem) cannot load VMM and thus are relegated to other hypervisors
Other hypervisors like VirtualBox or QEMU + KVM can use VT-x to accelerate virtual machines, but if "VT-x UG" is not present, only with *one* vCPU.
The option of doing pure Software emulation without VT-x is always an option and although slow allows you to have more vCPUs.
Did I get things right?


----------



## jgrafton (Jun 20, 2021)

I *believe* the missing UG only allowing 1 vCPU primarily affects bhyve.  I remember creating multi vCPU VMs on this box in Linux/KVM long ago.  Haven't had a need for VMs since migrating to jails.


----------



## SirDice (Jun 21, 2021)

jgrafton said:


> The POPCNT requirement is interesting since bhyve itself doesn't seem to actually *use* the op code as far as I can tell. It was introduced with the Intel Nehalem micro architecture along with the VT-x and EPT extensions which are absolutely required.


Probably easier to check (not that it's actually used). If it has the POPCNT feature it'll have the required features for bhyve(8) too.


----------



## PMc (Jun 21, 2021)

jgrafton said:


> I *believe* the missing UG only allowing 1 vCPU primarily affects bhyve.  I remember creating multi vCPU VMs on this box in Linux/KVM long ago.  Haven't had a need for VMs since migrating to jails.


It certainly affects bhyve. We do not know what else.
UG allows the CPU to hardware-emulate some 16/32bit iAPX (8086/80286) CPU as vCPU. A physical CPU can still run in these modes, but a vCPU can do it only with the UG feature. Hypervisors may use these operational modes at some place during the bootstrap, e.g bhyve uses it while enumerating the CPUs (or something like that). It is certainly possible to software-emulate these modes, but bhyve does not do elaborate software-emulation. For other hypervisors, it depends...


----------



## grehan@ (Jun 30, 2021)

Qemu includes an entire s/w x86 emulation so the missing real mode/32-bit flat mode instructions can be handled in s/w. Bhyve doesn't have this, so an o/s can't be booted from the real-mode reset vector when UG isn't present.

FreeBSD 64-bit guests can still work because bhyveload sets up the kernel to be booted directly in 64-bit mode. However, spinning up secondary CPUs requires real mode, and that will no longer work.

Linux won't work, even with grub-bhyve, because it requires 32-bit "flat mode" (paging disabled) to work, and that requires UG.


----------

