# Linux guests panic on boot in bhyve vm



## arf (Sep 7, 2019)

Linux guests crash during startup with a general protection fault.

FreeBSD 12.0-p10
vm-bhyve-1.3.0
AMD Ryzen 1700 & 2700

Console output booting debian-10.0.0-amd64-netinst.iso:


```
[    0.127064] Spectre V2 : Spectre mitigation: LFENCE not serializing, switching to generic retpoline
[    0.638197] ACPI Error: Could not enable RealTimeClock event (20180810/evxfevnt-184)
[    0.683838] general protection fault: 0000 [#1] SMP NOPTI
[    0.684414] CPU: 0 PID: 1 Comm: init Tainted: G        W         4.19.0-5-amd64 #1 Debian 4.19.37-5
[    0.685349] Hardware name:  BHYVE, BIOS 1.00 03/14/2014
[    0.685871] RIP: 0010:switch_mm_irqs_off+0x3ec/0x4f0
[    0.686365] Code: 7d 08 49 83 c5 18 4c 89 fa 31 f6 e8 0e 69 99 00 49 8b 45 00 48 85 c0 75 e5 e9 73 ff ff ff b9 49 00 00 00 b8 01 00 00 00 31 d2 <0f> 30 e9 86 fc ff ff e9 81 00 00 00 48 c7 c2 60 15 02 00 31 c0 65
[    0.688210] RSP: 0018:ffffa776400d3d08 EFLAGS: 00010046
[    0.688740] RAX: 0000000000000001 RBX: ffff92c01e2fa640 RCX: 0000000000000049
[    0.689459] RDX: 0000000000000000 RSI: ffff92c01e2fa640 RDI: ffff92c01e256740
[    0.690168] RBP: ffff92c01e2f9540 R08: ffff92c01f222660 R09: 0000000000000000
[    0.690876] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[    0.691562] R13: ffff92c01e256740 R14: ffff92c01e2f9540 R15: ffff92c01e2fa640
[    0.692243] FS:  00007f95506fa5c0(0000) GS:ffff92c01f200000(0000) knlGS:0000000000000000
[    0.693009] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.693565] CR2: 0000561277a90298 CR3: 000000001e33a000 CR4: 00000000000406f0
[    0.694246] Call Trace:
[    0.694498]  __schedule+0x263/0x870
[    0.694840]  schedule+0x28/0x80
[    0.695150]  do_wait+0x1c7/0x230
[    0.695470]  kernel_wait4+0xa6/0x140
[    0.695820]  ? task_stopped_code+0x50/0x50
[    0.696218]  __do_sys_wait4+0x83/0x90
[    0.696577]  ? handle_mm_fault+0xda/0x200
[    0.696967]  ? __do_page_fault+0x26c/0x4f0
[    0.697368]  do_syscall_64+0x53/0x110
[    0.697728]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[    0.698217] RIP: 0033:0x7f95505fe437
[    0.698565] Code: 5f 4a 0f 00 f7 d8 64 89 02 b8 ff ff ff ff eb bf 0f 1f 00 48 8d 05 c9 a2 0f 00 8b 00 85 c0 75 13 45 31 d2 b8 3d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 41 54 41 89 d4 55 48 89 f5 53 89 fb 48
[    0.700364] RSP: 002b:00007ffd09a4aa18 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
[    0.701134] RAX: ffffffffffffffda RBX: 0000561277a90260 RCX: 00007f95505fe437
[    0.701891] RDX: 0000000000000000 RSI: 00007ffd09a4aa2c RDI: 00000000ffffffff
[    0.702639] RBP: 0000561277a90260 R08: 0000000000000000 R09: 00007f95506fa5c0
[    0.703386] R10: 0000000000000000 R11: 0000000000000246 R12: 0000561277a91f70
[    0.704142] R13: 00007ffd09a4aa2c R14: 000056127794527a R15: 0000000000000000
[    0.705012] Modules linked in:
[    0.705326] ---[ end trace d1cfad8d94370a9c ]---
[    0.705780] RIP: 0010:switch_mm_irqs_off+0x3ec/0x4f0
[    0.706268] Code: 7d 08 49 83 c5 18 4c 89 fa 31 f6 e8 0e 69 99 00 49 8b 45 00 48 85 c0 75 e5 e9 73 ff ff ff b9 49 00 00 00 b8 01 00 00 00 31 d2 <0f> 30 e9 86 fc ff ff e9 81 00 00 00 48 c7 c2 60 15 02 00 31 c0 65
[    0.708069] RSP: 0018:ffffa776400d3d08 EFLAGS: 00010046
[    0.708579] RAX: 0000000000000001 RBX: ffff92c01e2fa640 RCX: 0000000000000049
[    0.709264] RDX: 0000000000000000 RSI: ffff92c01e2fa640 RDI: ffff92c01e256740
[    0.709948] RBP: ffff92c01e2f9540 R08: ffff92c01f222660 R09: 0000000000000000
[    0.710635] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[    0.711317] R13: ffff92c01e256740 R14: ffff92c01e2f9540 R15: ffff92c01e2fa640
[    0.711999] FS:  00007f95506fa5c0(0000) GS:ffff92c01f200000(0000) knlGS:0000000000000000
[    0.712771] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.713323] CR2: 0000561277a90298 CR3: 000000001e33a000 CR4: 00000000000406f0
```

I get similar crashes with other recent Linux guests that I've tried: CentOS, Alpine, Coreos, and Ubuntu. Linux guests with older kernels (4.4.0-135 or earlier) boot and run just fine as well as FreeBSD guests.

I've tried two CPUs (AMD Ryzen 1700 and 2700) on the same motherboard (ASUS PRIME B350 PLUS, latest BIOS) with the same results.

Is there a BIOS setting I can tweak to fix this? Are Linux guests working on Ryzen bhyve hosts?

Thanks!


----------



## arf (Sep 24, 2019)

Do Linux guests work on bhyve hosts running on Ryzen CPUs? I'm still trying to find a way to get this working without replacing my whole system.


----------



## PacketMan (Sep 27, 2019)

I have a Ryze machine running 11.3-RELEASE, that is rock solid, but not doing bhyve yet.  If I get time next week I will try for you.  

`$ dmesg | grep CPU`

```
CPU: AMD Ryzen 5 1600 Six-Core Processor             (3194.09-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
```


----------



## SirDice (Sep 27, 2019)

arf said:


> Linux guests with older kernels (4.4.0-135 or earlier) boot and run just fine as well as FreeBSD guests.


Just shooting in the dark here (I have no idea what the problem could be) but is it possible they all have this in common?

```
[    0.127064] Spectre V2 : Spectre mitigation: LFENCE not serializing, switching to generic retpoline
```
I can imagine a VM trying to update the (virtual) CPU's microcode and failing, causing it to crash and burn.


----------



## Zachary Sandberg (Feb 19, 2020)

Has there been any discovered fix or workaround for this?  I just upgraded to 12.1 from 11.3 and I am experiencing the same thing with my bhyve Linux guests that boot using grub2-bhyve.  I have two FreeBSD 11.3 guests that run just fine as well.


----------

