# unexpected soft update inconsistency



## dr. bob (Oct 11, 2022)

Dear helpful forum people,

My (virtual) server won't boot, and sadly I don't know much about system administration. Any guidance, anyone?

When it tries to boot:


```
[...]
Timecounter "TSC-low" frequency 1100004067 Hz quality 800
Event timer "HyperV" frequency 10000000 Hz quality 1000
kernel trap 9 with interrupts disabled

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer    = 0x20:0xffffffff80dedb63
stack pointer            = 0x28:0xffffffff819c8a60
frame pointer            = 0x28:0xffffffff819c8a90
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = resume, IOPL = 0
current process        = 0 (swapper)
trap number        = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8098e3e0 at kdb_backtrace+0x60
#1 0xffffffff809510b6 at vpanic+0x126
#2 0xffffffff80950f83 at panic+0x43
#3 0xffffffff80d55f8b at trap_fatal+0x36b
#4 0xffffffff80d55c0d at trap+0x77d
#5 0xffffffff80d3b8d2 at calltrap+0x8
#6 0xffffffff8099d725 at smp_rendezvous_cpus+0xd5
#7 0xffffffff80deed0f at vmbus_bus_init+0x2ef
#8 0xffffffff808f7ac8 at mi_startup+0x108
#9 0xffffffff802e266c at btext+0x2c
Uptime: 1s
```

I ran `fsck`, which fixed:



> free blk count(s) wrong in superblk
> summary information bad
> blk(s) missing in bit maps



but not:



> unexpected soft update inconsistency



Maybe my mistake was not using the journal? When I tried it using it, I got:


```
su+j recovering
    journal timestamp does not match fs mount time
```

maybe because I'd mounted it to take a look after its initial crash? Any ideas about what to try next? Thanks!

Bob


----------



## W.hâ/t (Oct 11, 2022)

Did you buidl a custom kernel?


----------



## mer (Oct 11, 2022)

It would be a bit helpful if you told us what version of everything here.


----------



## dr. bob (Oct 11, 2022)

Thanks for the quick responses. No, I didn't build a custom kernel.
The beginning of the boot process, which I left out before:


```
/boot/kernel/kernel text=0xfe2de8 data=0x129430+0x207fa0 syms=[0x8+0x146f88+0x8+0x1613ae]
/boot/kernel/accf_data.ko size 0x1598 at 0x19bd000
/boot/kernel/accf_http.ko size 0x26a8 at 0x19bf000
Booting...
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Hyper-V Version: 6.1.7100 [SP0]
  Features: 0x222<TMREFCNT,HYERCALL,REFTSC>
Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
CPU: Intel Xeon E312xx (Sandy Bridge, IBRS update) (2200.01-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a1  Family=0x6  Model=0x2a  Stepping=1
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x9fb82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1073741824 (1024 MB)
avail memory = 1009868800 (963 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BOCHS  BXPC    >
random: <Software, Yarrow> initialized
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <BOCHS BXPC> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc140-0xc14f at device 1.1 on pci0
ata0: <Hyper-V ATA storage disengage driver> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xc0c0-0xc0df irq 11 at device 1.2 on pci0
usbus0 on uhci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xfc000000-0xfdffffff,0xfebf0000-0xfebf0fff at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xc0e0-0xc0ff mem 0xfebf1000-0xfebf1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:16:3e:01:33:14
virtio_pci1: <VirtIO PCI Console adapter> port 0xc080-0xc0bf mem 0xfebf2000-0xfebf2fff,0xfe004000-0xfe007fff irq 11 at device 4.0 on pci0
virtio_pci2: <VirtIO PCI Block adapter> port 0xc000-0xc07f mem 0xfebf3000-0xfebf3fff,0xfe008000-0xfe00bfff irq 10 at device 5.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci2
vtblk0: 40960MB (83886081 512 byte sectors)
virtio_pci3: <VirtIO PCI Balloon adapter> port 0xc100-0xc11f mem 0xfe00c000-0xfe00ffff irq 10 at device 6.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci3
virtio_pci4: <VirtIO PCI Entropy adapter> port 0xc120-0xc13f mem 0xfe010000-0xfe013fff irq 11 at device 7.0 on pci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xe9000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 10.000 msec
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
cd0 at ata1 bus 0 scbus0 target 1 lun 0
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00004
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: 3193MB (1634993 2048 byte sectors)
random: unblocking device.
```

Does that help?


----------



## _martin (Oct 11, 2022)

You are running unsupported version of FreeBSD, you can read about it here on forums.

The error you showed doesn't add up to the whole boot message you posted above. . Actually the code where system crashes comes from hyperv moduel (expected) and it can be late in the boot. You could try to disable loading of hv_vmbus module (maybe it's loaded directly from /boot/loader.conf or indirectly by some service defined in /etc/rc.conf. If you can show us content of those files for us to match.

FS is most likely in inconsistent state because of the crash before.

We are missing some history to get an idea here. Was the VM working prior to this crash? Did you recently update Windows that would cause this crash? If it was supported version of FreeBSD you could open a PR.


----------



## dr. bob (Oct 11, 2022)

Martin, thanks for your reply. Unsupported because it's not current? What's a PR?

It keeps rebooting and crashing and rebooting and crashing. The hosting company attached a cd-rom so I can boot from that and try to fix things.

The VM was working fine prior to this crash. I haven't updated anything in forever, but maybe the hosting company did some maintenance.

The whole boot message:


```
+============Welcome to FreeBSD===========+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +=========================================+
                                          

/boot/kernel/kernel text=0xfe2de8 data=0x129430+0x207fa0 syms=[0x8+0x146f88+0x8+0x1613ae]
/boot/kernel/accf_data.ko size 0x1598 at 0x19bd000
/boot/kernel/accf_http.ko size 0x26a8 at 0x19bf000
Booting...
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Hyper-V Version: 6.1.7100 [SP0]
  Features: 0x222<TMREFCNT,HYERCALL,REFTSC>
Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
CPU: Intel Xeon E312xx (Sandy Bridge, IBRS update) (2200.01-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a1  Family=0x6  Model=0x2a  Stepping=1
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x9fb82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1073741824 (1024 MB)
avail memory = 1009868800 (963 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BOCHS  BXPC    >
random: <Software, Yarrow> initialized
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <BOCHS BXPC> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc140-0xc14f at device 1.1 on pci0
ata0: <Hyper-V ATA storage disengage driver> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xc0c0-0xc0df irq 11 at device 1.2 on pci0
usbus0 on uhci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xfc000000-0xfdffffff,0xfebf0000-0xfebf0fff at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xc0e0-0xc0ff mem 0xfebf1000-0xfebf1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:16:3e:01:33:14
virtio_pci1: <VirtIO PCI Console adapter> port 0xc080-0xc0bf mem 0xfebf2000-0xfebf2fff,0xfe004000-0xfe007fff irq 11 at device 4.0 on pci0
virtio_pci2: <VirtIO PCI Block adapter> port 0xc000-0xc07f mem 0xfebf3000-0xfebf3fff,0xfe008000-0xfe00bfff irq 10 at device 5.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci2
vtblk0: 40960MB (83886081 512 byte sectors)
virtio_pci3: <VirtIO PCI Balloon adapter> port 0xc100-0xc11f mem 0xfe00c000-0xfe00ffff irq 10 at device 6.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci3
virtio_pci4: <VirtIO PCI Entropy adapter> port 0xc120-0xc13f mem 0xfe010000-0xfe013fff irq 11 at device 7.0 on pci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xe9000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 10.000 msec
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
cd0 at ata1 bus 0 scbus0 target 1 lun 0
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00004
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: 3193MB (1634993 2048 byte sectors)
random: unblocking device.
Timecounter "TSC-low" frequency 1100004067 Hz quality 800
Event timer "HyperV" frequency 10000000 Hz quality 1000
kernel trap 9 with interrupts disabled


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer    = 0x20:0xffffffff80dedb63
stack pointer            = 0x28:0xffffffff819c8a60
frame pointer            = 0x28:0xffffffff819c8a90
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = resume, IOPL = 0
current process        = 0 (swapper)
trap number        = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8098e3e0 at kdb_backtrace+0x60
#1 0xffffffff809510b6 at vpanic+0x126
#2 0xffffffff80950f83 at panic+0x43
#3 0xffffffff80d55f8b at trap_fatal+0x36b
#4 0xffffffff80d55c0d at trap+0x77d
#5 0xffffffff80d3b8d2 at calltrap+0x8
#6 0xffffffff8099d725 at smp_rendezvous_cpus+0xd5
#7 0xffffffff80deed0f at vmbus_bus_init+0x2ef
#8 0xffffffff808f7ac8 at mi_startup+0x108
#9 0xffffffff802e266c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
```

Hope that makes things more clear.


----------



## _martin (Oct 11, 2022)

Unsupported version of FreeBSD is any version that is not listed as supported on main page. You are running FreeBSD 10, so nobody from FreeBSD team is going to look at any issue there (PR means problem report, way to contact FreeBSD people about technical problems). There are some rules here about that on forums, you can read about them in the link I pasted.

I did edit my post before you posted this, that code where the system crashes is stemming from hyperv. From what you said so far it seems this VM is hosted somewhere, it's not running at home at your PC. So most likely some update was done to Windows. One of the possibilities is that code on hypervisor (windows) changed too much for kernel module (FreeBSD) to handle.

So you should update to a supported version first.

As you said you don't know much about system administration it may get complicated very fast to troubleshoot this. Some administration is required to boot the rescue CD, mount the FS and look around. If you can't do that it would be better to have somebody who you know to help you. If we can see the list of the modules loaded (/boot/loader.conf) and services started (/etc/rc.conf) we may be able to blacklist (comment out) that part that loads the kernel module hv_vmbus (assuming it's not built in kernel) and maybe be able to boot your system.
These types of issues you see here are for dev people, people who do develop the system. Crash dump would help too but again, this is unsupported version of FreeBSD.

edit: it might help us to toggle the verbose boot. Verbose mode can be toggled from that menu 6 "Configure boot" that you see before system boots.


----------



## dr. bob (Oct 11, 2022)

Based on something I read online, I ran `fsck` a number of times. Because of the timestamp issue the journal wasn't used. Each time, it told me file system is clean and file system was modified. Does that mean it modified the file system each time? Does fixing problems sometimes create or uncover more problems? Unsure whether I should keep doing that.

Martin, thanks for supporting me even through my version isn't supported. Yes, VM hosted somewhere and not a PC I have at home. I'll ask the hosting company if they're aware of any hypervisor / FreeBSD issues.

I'm able to boot from a CD, that's how I can run `fsck`. I can mount the FS. /boot/loader.conf from the hard disk:






and /etc/rc.conf:





I'm surprised how short / basic they look.


----------



## ralphbsz (Oct 11, 2022)

dr. bob said:


> Based on something I read online, I ran `fsck` a number of times.


Running fsck extra times won't help. The file system is either getting cleaned, or it isn't.



> I'll ask the hosting company if they're aware of any hypervisor / FreeBSD issues.


Please don't do that. If you want, you can ask them whether they're aware of any issues with THE CURRENT hypervisor and AN 8 YEAR OLD VERSION OF FreeBSD. But the answer to that question is self-evident, so don't bother.



> I'm able to boot from a CD, ...


Time to upgrade FreeBSD. About 7 years overdue.


----------



## dr. bob (Oct 11, 2022)

wait, that's not the right rc.conf give me a minute


----------



## _martin (Oct 11, 2022)

dr. bob said:


> Does fixing problems sometimes create or uncover more problems?


Yes, but that's true in general in real life.
I don't know what you did with the fsck so I can't comment. But if the FS was dirty you do need to check it before mounting back.

Did you share the whole loader.conf? That rc.conf is rather empty indeed. Are you sure you're not showing us the rc.conf from CD boot? 

Me personally I'm curious to see what happened if that system was running. Most likely an update on hypervisor broke the old FreeBSD. It's a hunch.
You should update. But maybe we can confirm that. Also share the verbose boot of the machine when it crashes.


----------



## dr. bob (Oct 11, 2022)

/etc/rc.conf from the hard disk:

[removed]


----------



## _martin (Oct 12, 2022)

As you are running this old version on internet remove the picture from this post for security reasons (you're showing public IPs here). There's nothing in that rc.conf that would start that module. That code is then probably part of the GENERIC kernel.

Please paste that verbose boot from the system when it crashes.


----------



## dr. bob (Oct 12, 2022)

Ralph, thanks. I'm afraid you're right, it's overdue. But since I don't know much about this, my philosophy has been, if it ain't broke don't fix it. But now it's broke!

Martin, thanks, I'll ask them for a verbose boot. All I have is this VNC interface that doesn't let me capture or even scroll back.


----------



## Terry_Kennedy (Oct 12, 2022)

dr. bob said:


> Unsupported because it's not current? What's a PR?


Each FreeBSD release has a stated End-of-Life date. These can be found either here or here.

Having said that, the kernel isn't supposed to just panic in most cases. Other than hardware failures, ideally problems should be detected and reported with a concise error message, and if the kernel decides it is not safe to continue, then panic / crash dump / reboot. Unfortunately, checking "everything" isn't possible (if the developers don't know something can happen, they can't check for it happening) and checking "lots of stuff" adds quite a bit of overhead which slows the system down (development versions of FreeBSD have those checks turned on, but -RELEASE and -STABLE have them turned off).

If you can mount CD/DVD images without needing to physically put a disc in the (remote) drive, I'd suggest downloading a 13.1 image and booting from it. One of several things will happen:

It boots and lets you fix the problem - Once the filesystem is fixed, go back to your 10.3 system and make plans to upgrade to a supported release on an expedited basis. NOTE: Do not do anthing that will upgrade your filesystem(s) to something that your 10.3 system can't understand - so, nothing like `# zfs upgrade`.
It boots and panics - Open a PR (start here) so a developer will see the issue and hopefully fix it.
Something else - report back and we'll see where you go from there.
Note that even if 13.1 panics and a developer fixes it, it is unlikely that the fix is going to end up in an unsupported release. A few developers add their fixes to older releases, but that takes a lot of time that doesn't benefit the vast majority of users.

Regardless of how you get out of the immediate problem, you're going to need to use a newer FreeBSD version (preferably sooner than later). I'd categorize trying to do an in-place upgrade of 10.3 to any version that is still supported as "you can't get there from here (or at least you _REALLY_ don't want to try)". Better to do a clean install in another VM, merge your changes to the various files (/etc/rc.conf, etc.) as well as things like usernames / passwords. Then restore your data from backup(s)* to the new VM, test, and then shut down the old VM and deploy the new VM.

* My sometimes-snarky comment about backups is "If you don't have a tested backup and restore system, it must not be wortth writing the data to disk in the first place".


----------



## _martin (Oct 12, 2022)

dr. bob said:


> I'll ask them for a verbose boot.


Thanks. That verbose boot might not help you much but it could shed some interesting information. I like these type of issues. . Especially if it was working and then suddenly it stopped (which is what I said above, most likely triggered by the hypervisor upgrade).

From technical point of view: you are panicing in the code around here: vmbus_bus_init, fairly deep into the function (just an estimate as I don't have a crash). As it called the smp_rendezvous_cpus() where the trap was caught I'd say the init was on the way to be done (that verbose boot could confirm that).

These providers (Azure,etc.) allow users to move disks around. Terry's suggestion to do a clean setup and sync data is the best one but it does require you to get your hands dirty (or call somebody to do it if you're not familiar with it).
But as you are on internet with that VM it does make sense to be on up-to-date OS.


----------



## ralphbsz (Oct 12, 2022)

dr. bob said:


> Ralph, thanks. I'm afraid you're right, it's overdue. But since I don't know much about this, my philosophy has been, if it ain't broke don't fix it. But now it's broke!



With some things I follow the same philosophy. Matter-of-fact, for several years I ran a version of FreeBSD that was unsupported. Fortunately, I got away with it, but part of the reason was that my environment is much more controlled (no hardware changes). The problem is that you build up technical debt; what happened to me was that an upgrade was unfeasible (it would have taken a half dozen steps), so I had to reinstall from scratch.

One of the beauties of FreeBSD is that upgrades are easy, efficient, and nearly always painless. Run freebsd-update {fetch,install} every week or two to get patches. Watch the web site for version upgrades, and run freebsd-update with the correct command-line flag to go to the next version. Run {pkg update,upgrade} at the same schedule to upgrade all packages. With typically 10 minutes of work every week, you will stay current.

In your situation, you first need to pay off your technical debt.


----------



## dr. bob (Oct 12, 2022)

ralphbsz said:


> Running fsck extra times won't help. The file system is either getting cleaned, or it isn't.


Ralph, it was here that I read that it could take hours of running `fsck` to get a system to come back up. Which is why I wondered if one "clean" could create or uncover more "dirt".


----------



## dr. bob (Oct 12, 2022)

ralphbsz said:


> what happened to me was that an upgrade was unfeasible (it would have taken a half dozen steps), so I had to reinstall from scratch.
> 
> One of the beauties of FreeBSD is that upgrades are easy, efficient, and nearly always painless. Run freebsd-update {fetch,install} every week or two to get patches. Watch the web site for version upgrades, and run freebsd-update with the correct command-line flag to go to the next version. Run {pkg update,upgrade} at the same schedule to upgrade all packages. With typically 10 minutes of work every week, you will stay current.
> 
> In your situation, you first need to pay off your technical debt.


Ralph, I like that way of looking at it. And thanks, I wasn't aware that staying current was so easy. Now back to paying off my technical debt!


----------



## dr. bob (Oct 12, 2022)

Terry_Kennedy said:


> If you can mount CD/DVD images without needing to physically put a disc in the (remote) drive, I'd suggest downloading a 13.1 image and booting from it. One of several things will happen:
> 
> It boots and lets you fix the problem - Once the filesystem is fixed, go back to your 10.3 system and make plans to upgrade to a supported release on an expedited basis.
> Better to do a clean install in another VM, merge your changes to the various files (/etc/rc.conf, etc.) as well as things like usernames / passwords. Then restore your data from backup(s)* to the new VM, test, and then shut down the old VM and deploy the new VM.


Terry, thanks. My hosting company attached a CD (I assume a CD image, but possibly a physical CD?) with 11.1 and I can boot from that. Should I ask for 13.1 instead? Would that make me eligible for official support?

Booting from the 11.1 CD, I haven't been able to fix the problem. Well, I fixed it enough to mount, but not enough to boot from.

When I mount it, I can see my files. Looking ahead to another VM, might there be a way to transfer the data from my current setup to it? I don't think my current setup in its current state is online.


----------



## dr. bob (Oct 12, 2022)

_martin said:


> Thanks. That verbose boot might not help you much but it could shed some interesting information.


Martin, thanks, I'm still waiting, but in the meantime, I managed to grab a screenshot in verbose mode just before the trap:





Not sure if that's enough to shed any light.


----------



## Alain De Vos (Oct 12, 2022)

I don't know if this helps. A google search ?


			vmbus kernel trap freebsd - Google Search


----------



## Terry_Kennedy (Oct 12, 2022)

dr. bob said:


> Ralph, it was here that I read that it could take hours of running `fsck` to get a system to come back up. Which is why I wondered if one "clean" could create or uncover more "dirt".


"Way back when" (4BSD) multiple `# fsck` runs could sometimes be required. That was generally because when fsck(8) fixed things in one pass, it might trigger detection of other errors in a prior pass, requiring multiple runs to have the filesystem marked "clean". 2BSD actually requires you to physically halt the CPU (via an actual front panel switch!) if fsck(8) modified the root filesystem to fix errors, because doing a regular reboot would cause the kernel to write out cached info and re-corrupt the filesystem.

But that is all ancient history - fsck(8) has had a huge amount of work done on it since then, and if ir can't fix a problem after being run twice in a row, it either is hitting something it doesn't know how to fix or the "fix" is actually making things worse.

I've never seen a multi-hour fsck(8) run. The longest I've seen was 15+ years ago on a 2TB filesystem made out of a bunch of 200GB drives.  These days I have multiple 128TB filesystems (using ZFS) that scrub (the ZFS equivalent of fsck(8)) the whole 128TB in under 3 hours.


----------



## Terry_Kennedy (Oct 12, 2022)

dr. bob said:


> Terry, thanks. My hosting company attached a CD (I assume a CD image, but possibly a physical CD?) with 11.1 and I can boot from that. Should I ask for 13.1 instead? Would that make me eligible for official support?
> 
> Booting from the 11.1 CD, I haven't been able to fix the problem. Well, I fixed it enough to mount, but not enough to boot from.
> 
> When I mount it, I can see my files. Looking ahead to another VM, might there be a way to transfer the data from my current setup to it? I don't think my current setup in its current state is online.


Probably a CD image. I believe 11.1 has been EoL for 4+ years at this point. Unless there is a specific reason not to, I suggest trying with the latest RELEASE version (13.1 as of today, the tentative schedule for 13.2 is late March 2023).

If you can mount the filesystem from a newer version of FreeBSD (11.1, 13.1, whatever), you can start networking (even in single-user mode) and use whatever method you'd like (rdump(8), NFS export, etc.) to get your data from that system to a new VM. _HOWEVER_ just because a particular FreeBSD version can mount a corrupted filesystem doesn't mean that it won't panic when trying to access files on that filesystem. Mounting doesn't do the full set of consistency checks that fsck(8) does - it just checks to make sure the filesystem is of a known type and that the superblock is properly formed.

I don't believe you stated whether this system was something personal, a business-critical system, or something in between. If it is on the business-critical end, you might want to take a look here to see if there's a consultant with FreeBSD experience that can assist you. That will likely get things done a lot faster and with less risk of "shooting yourself in the foot" than trying to work through things on your own, as your original post said you have limited system administration experience.


----------



## _martin (Oct 12, 2022)

dr. bob said:


> Not sure if that's enough to shed any light.


Yes, it does point to a location I assumed before seeing the output.
I downloaded the vagrant box with FreeBSD 10.3p11 that matches your running kernel. Got a disk with Windows 10, used it in HyperV and .. nothing. Booted up just fine. I wanted to see if I can trigger the bug myself. I'm fighting with the virtio - I'm not sure how one can do that on Windows 10 (is it only server additions maybe?).

Again, upgrade is the way to go. But for those who'd like to do a follow up on (unsupported) version this is the crash:

```
gdb) x/3i 0xffffffff80dedb63
0xffffffff80dedb63 <hv_vmbus_synic_init+51>:    rdmsr
0xffffffff80dedb65 <hv_vmbus_synic_init+53>:    lea    eax,[r15+r15]
0xffffffff80dedb69 <hv_vmbus_synic_init+57>:    cdqe
```
Where rdmsr generates exception when CPL is not 0 (which it is) or when ECX points to not implemented address. If we had a dump we could investigate further. But looking at the code most likely location of crash is at hv_vmbus_synic_init().

Still it would be interesting to know what it fails now.


----------



## dr. bob (Oct 12, 2022)

Terry_Kennedy said:


> If you can mount the filesystem from a newer version of FreeBSD (11.1, 13.1, whatever), you can start networking (even in single-user mode) and use whatever method you'd like (rdump(8), NFS export, etc.) to get your data from that system to a new VM. _HOWEVER_ just because a particular FreeBSD version can mount a corrupted filesystem doesn't mean that it won't panic when trying to access files on that filesystem. Mounting doesn't do the full set of consistency checks that fsck(8) does - it just checks to make sure the filesystem is of a known type and that the superblock is properly formed.
> 
> you might want to take a look here to see if there's a consultant with FreeBSD experience that can assist you.


Terry, thanks. I like the challenge of DIY, but it's definitely slow going. Do the helpful people here ever take work as consultants?

I'm not sure I actually can start networking. I tried `ping` and had 0% success.

As far as `fsck`, now I'm wondering if "unexpected soft update inconsistency" is really "corruption". Would the journal (whatever that is) being out of sync with the filesystem (maybe because I told `fsck` once not to use the journal) be considered a soft update inconsistency? Could the journal be out of sync and the filesystem still be ok? Can I just delete the journal? Would it restart from scratch? Then would there no longer be any inconsistency?

Also, does "file system was modified" necessarily mean `fsck` found problems and fixed them? I see here and elsewhere that updating the badblock and / or free inode counts, even if unchanged, can trigger that message.

I guess my wishful thinking is that the file system is (now) ok and the reason it can't boot is a new Hyper-V / old FreeBSD incompatibility.


----------



## dr. bob (Oct 12, 2022)

_martin said:


> Still it would be interesting to know what it fails now.


Martin, here you go (part 1, bumping into the character limit here):


```
+============Welcome to FreeBSD===========+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +=========================================+
                                          

/boot/kernel/kernel text=0xfe2de8 data=0x129430+0x207fa0 syms=[0x8+0x146f88+0x8+0x1613ae]
/boot/kernel/accf_data.ko size 0x1598 at 0x19bd000
/boot/kernel/accf_http.ko size 0x26a8 at 0x19bf000
Booting...
Table 'FACP' at 0x3ffe16a3
Table 'APIC' at 0x3ffe1717
APIC: Found table at 0x3ffe1717
APIC: Using the MADT enumerator.
MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
SMP: Added CPU 0 (AP)
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Hyper-V Version: 6.1.7100 [SP0]
  Features: 0x222<TMREFCNT,HYERCALL,REFTSC>
  Recommends: 00000000 ffffffff
  Limits: Vcpu:-1 Lcpu:64 Int:0
Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff819c2000.
Preloaded elf obj module "/boot/kernel/accf_data.ko" at 0xffffffff819c2c70.
Preloaded elf obj module "/boot/kernel/accf_http.ko" at 0xffffffff819c3160.
Calibrating TSC clock ... TSC clock: 2200008884 Hz
CPU: Intel Xeon E312xx (Sandy Bridge, IBRS update) (2200.01-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a1  Family=0x6  Model=0x2a  Stepping=1
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x9fb82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
3rd-level cache: 16MByte, 16-way set associative, 64 byte line size
2nd-level cache: 2-MB, 8-way set associative, 64-byte line size
1st-level instruction cache: 32 KB, 8-way set associative, 64 byte line size
1st-level data cache: 32 KB, 8-way set associative, 64 byte line size
L2 cache: 512 kbytes, 16-way associative, 64 bytes/line
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1073741824 (1024 MB)
Physical memory chunk(s):
0x0000000000001000 - 0x000000000009bfff, 634880 bytes (155 pages)
0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
0x0000000001a04000 - 0x000000003e573fff, 1018626048 bytes (248688 pages)
avail memory = 1009868800 (963 MB)
Event timer "LAPIC" quality 600
LAPIC: ipi_wait() us multiplier 2 (r 786515145 tsc 2200008884)
ACPI APIC Table: <BOCHS  BXPC    >
APIC: CPU 0 has ACPI ID 0
x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
x86bios: SSEG 0x001000-0x001fff at 0xfffffe003d12b000
x86bios: EBDA 0x09f000-0x09ffff at 0xfffff8000009f000
x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
XEN: CPU 0 has VCPU ID 0
Hardware, Intel Secure Key RNG: RDRAND is not present
Hardware, VIA Nehemiah Padlock RNG: VIA Padlock RNG not present
Falling back to <Software, Yarrow> random adaptor
random: <Software, Yarrow> initialized
ULE: setup cpu 0
ACPI: RSDP 0x00000000000F5870 000014 (v00 BOCHS )
ACPI: RSDT 0x000000003FFE17B7 000030 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: FACP 0x000000003FFE16A3 000074 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: DSDT 0x000000003FFE0040 001663 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: FACS 0x000000003FFE0000 000040
ACPI: APIC 0x000000003FFE1717 000078 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: WAET 0x000000003FFE178F 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
MADT: Found IO APIC ID 0, Interrupt 0 at 0xfec00000
ioapic0: Routing external 8259A's -> intpin 0
MADT: Interrupt override: source 0, irq 2
ioapic0: Routing IRQ 0 -> intpin 2
MADT: Interrupt override: source 5, irq 5
ioapic0: intpin 5 trigger: level
MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
MADT: Interrupt override: source 10, irq 10
ioapic0: intpin 10 trigger: level
MADT: Interrupt override: source 11, irq 11
ioapic0: intpin 11 trigger: level
lapic: Routing NMI -> LINT1
lapic: LINT1 trigger: edge
lapic: LINT1 polarity: high
ioapic0 <Version 1.1> irqs 0-23 on motherboard
cpu0 BSP:
     ID: 0x00000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
wlan: <802.11 Link Layer>
snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
null: <null device, zero device>
nfslock: pseudo-device
VESA: INT 0x10 vector 0xc000:0x5262
VESA: information block
0000   56 45 53 41 00 03 d9 54 00 c0 00 00 00 00 22 00
0010   00 02 40 00 00 00 ed 54 00 c0 00 55 00 c0 14 55
0020   00 c0 01 01 11 01 10 01 12 01 03 01 14 01 13 01
0030   05 01 17 01 16 01 15 01 18 01 07 01 19 01 1a 01
0040   00 00 01 00 02 00 03 00 04 00 05 00 06 00 07 00
0050   0d 00 0e 00 0f 00 10 00 11 00 12 00 13 00 6a 00
0060   ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0070   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0080   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0090   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0100   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0110   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0120   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0130   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0140   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0150   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0160   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0170   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0180   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0190   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
VESA: 31 mode(s) found
VESA: v3.0, 4096k memory, flags:0x0, mode table:0xfffffe003d15e022 (2000022)
VESA: SeaBIOS VBE(C) 2011
VESA: SeaBIOS Developers SeaBIOS VBE Adapter Rev. 1
io: <I/O>
VMBUS: load
kbd: new array size 4
kbd1 at kbdmux0
mem: <memory>
hptnr: R750/DC7280 controller driver v1.1.4
hpt27xx: RocketRAID 27xx controller driver v1.2.7
hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2
acpi0: <BOCHS BXPC> on motherboard
ACPI: All ACPI Tables successfully acquired
ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu0: switching to generic Cx mode
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock (resolution 1000000us, adjustment 0.500000000s)
ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 49
Event timer "RTC" frequency 32768 Hz quality 0
ACPI timer: 1/53 1/58 1/51 1/50 1/60 1/28 1/27 1/30 1/29 1/35 -> 10
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pci_link0:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   10   N     0  5 10 11
  Validation          0   10   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link1:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   10   N     0  5 10 11
  Validation          0   10   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link2:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   11   N     0  5 10 11
  Validation          0   11   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link3:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   11   N     0  5 10 11
  Validation          0   11   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link4:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0    9   N     0  9
  Validation          0    9   N     0  9
  After Disable       0  255   N     0  9
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: decoding 5 range 0-0xff
pcib0: decoding 4 range 0-0xcf7
pcib0: decoding 4 range 0xd00-0xffff
pcib0: decoding 3 range 0xa0000-0xbffff
pcib0: decoding 3 range 0x40000000-0xfebfffff
pcib0: decoding 3 range 0x100000000-0x17fffffff
ACPI: Found matching pin for 0.1.INTA at func 3: 9
ACPI: Found matching pin for 0.1.INTD at func 2: 11
ACPI: Found matching pin for 0.3.INTA at func 0: 11
ACPI: Found matching pin for 0.4.INTA at func 0: 11
ACPI: Found matching pin for 0.5.INTA at func 0: 10
ACPI: Found matching pin for 0.6.INTA at func 0: 10
ACPI: Found matching pin for 0.7.INTA at func 0: 11
pci0: <ACPI PCI bus> on pcib0
pci0: domain=0, physical bus=0
found->    vendor=0x8086, dev=0x1237, revid=0x02
    domain=0, bus=0, slot=0, func=0
    class=06-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0000, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
found->    vendor=0x8086, dev=0x7000, revid=0x00
    domain=0, bus=0, slot=1, func=0
    class=06-01-00, hdrtype=0x00, mfdev=1
    cmdreg=0x0103, statreg=0x0200, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
found->    vendor=0x8086, dev=0x7010, revid=0x00
    domain=0, bus=0, slot=1, func=1
    class=01-01-80, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0280, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
pcib0: allocated type 4 (0x1f0-0x1f7) for rid 10 of pci0:0:1:1
pcib0: allocated type 4 (0x3f6-0x3f6) for rid 14 of pci0:0:1:1
pcib0: allocated type 4 (0x170-0x177) for rid 18 of pci0:0:1:1
pcib0: allocated type 4 (0x376-0x376) for rid 1c of pci0:0:1:1
    map[20]: type I/O Port, range 32, base 0xc140, size  4, enabled
pcib0: allocated type 4 (0xc140-0xc14f) for rid 20 of pci0:0:1:1
found->    vendor=0x8086, dev=0x7020, revid=0x01
    domain=0, bus=0, slot=1, func=2
    class=0c-03-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0107, statreg=0x0000, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=d, irq=11
    map[20]: type I/O Port, range 32, base 0xc0c0, size  5, enabled
pcib0: allocated type 4 (0xc0c0-0xc0df) for rid 20 of pci0:0:1:2
pcib0: matched entry for 0.1.INTD (src \_SB_.LNKD:0)
pcib0: slot 1 INTD routed to irq 11 via \_SB_.LNKD
found->    vendor=0x8086, dev=0x7113, revid=0x03
    domain=0, bus=0, slot=1, func=3
    class=06-80-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0280, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=9
pcib0: matched entry for 0.1.INTA (src \_SB_.LNKS:0)
pcib0: slot 1 INTA routed to irq 9 via \_SB_.LNKS
found->    vendor=0x1013, dev=0x00b8, revid=0x00
    domain=0, bus=0, slot=2, func=0
    class=03-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0000, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    map[10]: type Prefetchable Memory, range 32, base 0xfc000000, size 25, enabled
pcib0: allocated type 3 (0xfc000000-0xfdffffff) for rid 10 of pci0:0:2:0
    map[14]: type Memory, range 32, base 0xfebf0000, size 12, enabled
pcib0: allocated type 3 (0xfebf0000-0xfebf0fff) for rid 14 of pci0:0:2:0
found->    vendor=0x1af4, dev=0x1000, revid=0x00
    domain=0, bus=0, slot=3, func=0
    class=02-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=11
    MSI-X supports 3 messages in map 0x14
    map[10]: type I/O Port, range 32, base 0xc0e0, size  5, enabled
pcib0: allocated type 4 (0xc0e0-0xc0ff) for rid 10 of pci0:0:3:0
    map[14]: type Memory, range 32, base 0xfebf1000, size 12, enabled
pcib0: allocated type 3 (0xfebf1000-0xfebf1fff) for rid 14 of pci0:0:3:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe000000, size 14, enabled
pcib0: allocated type 3 (0xfe000000-0xfe003fff) for rid 20 of pci0:0:3:0
pcib0: matched entry for 0.3.INTA (src \_SB_.LNKC:0)
pcib0: slot 3 INTA routed to irq 11 via \_SB_.LNKC
found->    vendor=0x1af4, dev=0x1003, revid=0x00
    domain=0, bus=0, slot=4, func=0
    class=07-80-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=11
    MSI-X supports 2 messages in map 0x14
    map[10]: type I/O Port, range 32, base 0xc080, size  6, enabled
pcib0: allocated type 4 (0xc080-0xc0bf) for rid 10 of pci0:0:4:0
    map[14]: type Memory, range 32, base 0xfebf2000, size 12, enabled
pcib0: allocated type 3 (0xfebf2000-0xfebf2fff) for rid 14 of pci0:0:4:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe004000, size 14, enabled
pcib0: allocated type 3 (0xfe004000-0xfe007fff) for rid 20 of pci0:0:4:0
pcib0: matched entry for 0.4.INTA (src \_SB_.LNKD:0)
pcib0: slot 4 INTA routed to irq 11 via \_SB_.LNKD
found->    vendor=0x1af4, dev=0x1001, revid=0x00
    domain=0, bus=0, slot=5, func=0
    class=01-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0107, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=10
    MSI-X supports 2 messages in map 0x14
    map[10]: type I/O Port, range 32, base 0xc000, size  7, enabled
pcib0: allocated type 4 (0xc000-0xc07f) for rid 10 of pci0:0:5:0
    map[14]: type Memory, range 32, base 0xfebf3000, size 12, enabled
pcib0: allocated type 3 (0xfebf3000-0xfebf3fff) for rid 14 of pci0:0:5:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe008000, size 14, enabled
pcib0: allocated type 3 (0xfe008000-0xfe00bfff) for rid 20 of pci0:0:5:0
pcib0: matched entry for 0.5.INTA (src \_SB_.LNKA:0)
pcib0: slot 5 INTA routed to irq 10 via \_SB_.LNKA
found->    vendor=0x1af4, dev=0x1002, revid=0x00
    domain=0, bus=0, slot=6, func=0
    class=00-ff-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=10
    map[10]: type I/O Port, range 32, base 0xc100, size  5, enabled
pcib0: allocated type 4 (0xc100-0xc11f) for rid 10 of pci0:0:6:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe00c000, size 14, enabled
pcib0: allocated type 3 (0xfe00c000-0xfe00ffff) for rid 20 of pci0:0:6:0
pcib0: matched entry for 0.6.INTA (src \_SB_.LNKB:0)
pcib0: slot 6 INTA routed to irq 10 via \_SB_.LNKB
found->    vendor=0x1af4, dev=0x1005, revid=0x00
    domain=0, bus=0, slot=7, func=0
    class=00-ff-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=11
    map[10]: type I/O Port, range 32, base 0xc120, size  5, enabled
pcib0: allocated type 4 (0xc120-0xc13f) for rid 10 of pci0:0:7:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe010000, size 14, enabled
pcib0: allocated type 3 (0xfe010000-0xfe013fff) for rid 20 of pci0:0:7:0
pcib0: matched entry for 0.7.INTA (src \_SB_.LNKC:0)
pcib0: slot 7 INTA routed to irq 11 via \_SB_.LNKC
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc140-0xc14f at device 1.1 on pci0
ata0: <Hyper-V ATA storage disengage driver> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 50
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xc0c0-0xc0df irq 11 at device 1.2 on pci0
ioapic0: routing intpin 11 (ISA IRQ 11) to lapic 0 vector 51
usbus0 on uhci0
uhci0: usbpf: Attached
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xfc000000-0xfdffffff,0xfebf0000-0xfebf0fff at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xc0e0-0xc0ff mem 0xfebf1000-0xfebf1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x79bfffe7 <EventIdx,RingIndirect,0x8000000,NotifyOnEmpty,SetMacAddress,GuestAnnounce,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxUFO,TxTSOECN,TxTSOv6,TxTSOv4,RxUFO,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,MacAddress,0x4,RxChecksum,TxChecksum>
virtio_pci0: negotiated features: 0x308fbbe3 <EventIdx,RingIndirect,SetMacAddress,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxTSOECN,TxTSOv6,TxTSOv4,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,MacAddress,RxChecksum,TxChecksum>
vtnet0: bpf attached
vtnet0: Ethernet address: 00:16:3e:01:33:14
virtio_pci0: attempting to allocate 3 MSI-X vectors (3 supported)
msi: routing MSI-X IRQ 256 to local APIC 0 vector 52
msi: routing MSI-X IRQ 257 to local APIC 0 vector 53
msi: routing MSI-X IRQ 258 to local APIC 0 vector 54
virtio_pci0: using IRQs 256-258 for MSI-X
virtio_pci0: using per VQ MSIX interrupts
virtio_pci1: <VirtIO PCI Console adapter> port 0xc080-0xc0bf mem 0xfebf2000-0xfebf2fff,0xfe004000-0xfe007fff irq 11 at device 4.0 on pci0
virtio_pci2: <VirtIO PCI Block adapter> port 0xc000-0xc07f mem 0xfebf3000-0xfebf3fff,0xfe008000-0xfe00bfff irq 10 at device 5.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci2
virtio_pci2: host features: 0x79006e54 <EventIdx,RingIndirect,0x8000000,NotifyOnEmpty,0x4000,0x2000,ConfigWCE,Topology,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci2: negotiated features: 0x10000e54 <RingIndirect,ConfigWCE,Topology,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci2: attempting to allocate 2 MSI-X vectors (2 supported)
msi: routing MSI-X IRQ 259 to local APIC 0 vector 55
msi: routing MSI-X IRQ 260 to local APIC 0 vector 56
virtio_pci2: using IRQs 259-260 for MSI-X
virtio_pci2: using per VQ MSIX interrupts
vtblk0: 40960MB (83886081 512 byte sectors)
virtio_pci3: <VirtIO PCI Balloon adapter> port 0xc100-0xc11f mem 0xfe00c000-0xfe00ffff irq 10 at device 6.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci3
virtio_pci3: host features: 0x79000002 <EventIdx,RingIndirect,0x8000000,NotifyOnEmpty,StatsVq>
virtio_pci3: negotiated features: 0
ioapic0: routing intpin 10 (ISA IRQ 10) to lapic 0 vector 57
virtio_pci3: using legacy interrupt
virtio_pci4: <VirtIO PCI Entropy adapter> port 0xc120-0xc13f mem 0xfe010000-0xfe013fff irq 11 at device 7.0 on pci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 58
uart0: fast interrupt
uart0: PPS capture mode: DCDinvalid
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
atkbd: the current kbd controller command byte 0061
atkbd: keyboard ID 0x41ab (2)
kbdc: RESET_KBD return code:00fa
kbdc: RESET_KBD status:00aa
kbd0 at atkbd0
kbd0: atkbd0, AT 101/102 (2), config:0x0, flags:0x3d0000
ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 59
atkbd0: [GIANT-LOCKED]
psm0: unable to allocate IRQ
psmcpnp0: <PS/2 mouse port> irq 12 on acpi0
psm0: current command byte:0061
kbdc: TEST_AUX_PORT status:0000
kbdc: RESET_AUX return code:00fa
kbdc: RESET_AUX status:00aa
kbdc: RESET_AUX ID:0000
kbdc: RESET_AUX return code:00fa
kbdc: RESET_AUX status:00aa
kbdc: RESET_AUX ID:0000
psm: status 00 02 64
psm: status 00 00 64
psm: status 00 03 64
psm: status 00 03 64
psm: data 08 00 00
psm: status 00 02 64
psm0: <PS/2 Mouse> irq 12 on atkbdc0
ioapic0: routing intpin 12 (ISA IRQ 12) to lapic 0 vector 60
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4-00, 5 buttons
psm0: config:00000000, flags:00000008, packet size:4
psm0: syncmask:08, syncbits:00
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
ACPI: Enabled 2 GPEs in block 00 to 0F
acpi0: wakeup code va 0xfffffe004e675000 pa 0x4000
ex_isa_identify()
pcib0: allocated type 3 (0xa0000-0xa07ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa0800-0xa0fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa1000-0xa17ff) for rid 0 of orm0
```


----------



## dr. bob (Oct 12, 2022)

Part 2:


```
pcib0: allocated type 3 (0xa1800-0xa1fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa2000-0xa27ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa2800-0xa2fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa3000-0xa37ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa3800-0xa3fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa4000-0xa47ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa4800-0xa4fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa5000-0xa57ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa5800-0xa5fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa6000-0xa67ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa6800-0xa6fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa7000-0xa77ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa7800-0xa7fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa8000-0xa87ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa8800-0xa8fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa9000-0xa97ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa9800-0xa9fff) for rid 0 of orm0
pcib0: allocated type 3 (0xaa000-0xaa7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xaa800-0xaafff) for rid 0 of orm0
pcib0: allocated type 3 (0xab000-0xab7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xab800-0xabfff) for rid 0 of orm0
pcib0: allocated type 3 (0xac000-0xac7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xac800-0xacfff) for rid 0 of orm0
pcib0: allocated type 3 (0xad000-0xad7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xad800-0xadfff) for rid 0 of orm0
pcib0: allocated type 3 (0xae000-0xae7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xae800-0xaefff) for rid 0 of orm0
pcib0: allocated type 3 (0xaf000-0xaf7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xaf800-0xaffff) for rid 0 of orm0
pcib0: allocated type 3 (0xb0000-0xb07ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb0800-0xb0fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb1000-0xb17ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb1800-0xb1fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb2000-0xb27ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb2800-0xb2fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb3000-0xb37ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb3800-0xb3fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb4000-0xb47ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb4800-0xb4fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb5000-0xb57ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb5800-0xb5fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb6000-0xb67ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb6800-0xb6fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb7000-0xb77ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb7800-0xb7fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb8000-0xb87ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb8800-0xb8fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb9000-0xb97ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb9800-0xb9fff) for rid 0 of orm0
pcib0: allocated type 3 (0xba000-0xba7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xba800-0xbafff) for rid 0 of orm0
pcib0: allocated type 3 (0xbb000-0xbb7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbb800-0xbbfff) for rid 0 of orm0
pcib0: allocated type 3 (0xbc000-0xbc7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbc800-0xbcfff) for rid 0 of orm0
pcib0: allocated type 3 (0xbd000-0xbd7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbd800-0xbdfff) for rid 0 of orm0
pcib0: allocated type 3 (0xbe000-0xbe7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbe800-0xbefff) for rid 0 of orm0
pcib0: allocated type 3 (0xbf000-0xbf7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbf800-0xbffff) for rid 0 of orm0
ahc_isa_identify 0: ioport 0xc00 alloc failed
ahc_isa_identify 1: ioport 0x1c00 alloc failed
ahc_isa_identify 2: ioport 0x2c00 alloc failed
ahc_isa_identify 3: ioport 0x3c00 alloc failed
ahc_isa_identify 4: ioport 0x4c00 alloc failed
ahc_isa_identify 5: ioport 0x5c00 alloc failed
ahc_isa_identify 6: ioport 0x6c00 alloc failed
ahc_isa_identify 7: ioport 0x7c00 alloc failed
ahc_isa_identify 8: ioport 0x8c00 alloc failed
ahc_isa_identify 9: ioport 0x9c00 alloc failed
ahc_isa_identify 10: ioport 0xac00 alloc failed
ahc_isa_identify 11: ioport 0xbc00 alloc failed
ahc_isa_identify 12: ioport 0xcc00 alloc failed
ahc_isa_identify 13: ioport 0xdc00 alloc failed
ahc_isa_identify 14: ioport 0xec00 alloc failed
isa_probe_children: disabling PnP devices
atkbdc: atkbdc0 already exists; skipping it
atrtc: atrtc0 already exists; skipping it
sc: sc0 already exists; skipping it
uart: uart0 already exists; skipping it
isa_probe_children: probing non-PnP devices
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xe9000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sc0: fb0, kbd1, terminal emulator: scteken (teken terminal)
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
pcib0: allocated type 4 (0x3c0-0x3df) for rid 0 of vga0
pcib0: allocated type 3 (0xa0000-0xbffff) for rid 0 of vga0
attimer0: <AT timer> at port 0x40 on isa0
pcib0: allocated type 4 (0x40-0x43) for rid 0 of attimer0
Timecounter "i8254" frequency 1193182 Hz quality 0
ioapic0: routing intpin 2 (ISA IRQ 0) to lapic 0 vector 61
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
fdc0 failed to probe at port 0x3f0 irq 6 drq 2 on isa0
ppc0: cannot reserve I/O port range
ppc0 failed to probe at irq 7 on isa0
pcib0: allocated type 4 (0x2f8-0x2ff) for rid 0 of uart1
uart1 failed to probe at port 0x2f8-0x2ff irq 3 on isa0
wbwd0 failed to probe on isa0
isa_probe_children: probing PnP devices
Device configuration finished.
procfs registered
lapic: Divisor 2, Frequency 500021880 Hz
Timecounters tick every 10.000 msec
vlan: initialized, using hash tables with chaining
tcp_init: net.inet.tcp.tcbhashsize auto tuned to 8192
lo0: bpf attached
hptnr: no controller detected.
hpt27xx: no controller detected.
hptrr: no controller detected.
GEOM: new disk vtbd0
usbus0: 12Mbps Full Speed USB v1.0
ata1: reset tp1 mask=03 ostat0=41 ostat1=50
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ata1: stat0=0x50 err=0x01 lsb=0xff msb=0xff
ata1: stat1=0x00 err=0x01 lsb=0x14 msb=0xeb
ata1: reset tp2 stat0=50 stat1=00 devices=0x20000
pass0 at ata1 bus 0 scbus0 target 1 lun 0
pass0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
pass0: Serial Number QM00004
pass0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0 at ata1 bus 0 scbus0 target 1 lun 0
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00004
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: 3193MB (1634993 2048 byte sectors)
GEOM: new disk cd0
random: unblocking device.
TSC timecounter discards lower 1 bit(s)
Timecounter "TSC-low" frequency 1100004442 Hz quality 800
Event timer "HyperV" frequency 10000000 Hz quality 1000
VMBUS: vmbus callback vector 254
VMBUS: Calling smp_rendezvous, smp_started = 0
kernel trap 9 with interrupts disabled

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer    = 0x20:0xffffffff80dedb63
stack pointer            = 0x28:0xffffffff819c8a60
frame pointer            = 0x28:0xffffffff819c8a90
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = resume, IOPL = 0
current process        = 0 (swapper)
trap number        = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8098e3e0 at kdb_backtrace+0x60
#1 0xffffffff809510b6 at vpanic+0x126
#2 0xffffffff80950f83 at panic+0x43
#3 0xffffffff80d55f8b at trap_fatal+0x36b
#4 0xffffffff80d55c0d at trap+0x77d
#5 0xffffffff80d3b8d2 at calltrap+0x8
#6 0xffffffff8099d725 at smp_rendezvous_cpus+0xd5
#7 0xffffffff80deed0f at vmbus_bus_init+0x2ef
#8 0xffffffff808f7ac8 at mi_startup+0x108
#9 0xffffffff802e266c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
```

I don't know if that changes your thinking or not.


----------



## _martin (Oct 12, 2022)

I hope you didn't have too much trouble with getting that log in ascii. As I mentioned earlier your photo was enough. However having this information doesn't hurt.

I'm trying to reproduce the bug. This caught my attention:
	
	



```
Hypervisor: Origin = "Microsoft Hv"
ACPI APIC Table: <BOCHS  BXPC    >
```
It's an interesting combination. I tried HyperV on my Win10, the ACPI tables have different maker (label). I spawned a VM in Azure to see it running on their infrastructure, same issue.

This got me thinking whether this is not a qemu VM on windows with hyperv accelerator (whpx). Funny enough I can't get FreeBSD 13 to boot under this combination. There's known problem that was fixed in PR 213155 long time ago.
This doesn't answer the question why it suddenly stopped working for you but then we don't know the history of the VM nor the hypervisor it was running on.
Also this doesn't change a fact you should upgrade the FreeBSD, even more so as it hosts internet facing services.

I don't want to write long posts explaining if it makes sense or not so I'd say it this way: it's a long shot but worth trying. If you have serial console attached and if you can influence the boot (which I assume you can as you were able to toggle the verbose boot) as soon as you are presented with FreeBSD boot menu press 3 to escape to boot prompt (OK prompt) and type this
	
	



```
set hw.use_xsave=0
boot -v
```
And see if you can boot the machine.


----------



## dr. bob (Oct 12, 2022)

_martin said:


> type this
> 
> 
> 
> ...


Martin, thanks. Waiting for it to reboot so I can try that. I feel like you're a wizard giving me a spell to cast!


----------



## dr. bob (Oct 12, 2022)

alas:


----------



## _martin (Oct 12, 2022)

It was a long shot.
I was not able to reproduce your setup, I wonder what kind of qemu with hyperv enlightenments is that. But unfortunately without being able to trigger the crash it's practically nothing we can do.


----------



## Terry_Kennedy (Oct 13, 2022)

dr. bob said:


> Terry, thanks. I like the challenge of DIY, but it's definitely slow going. Do the helpful people here ever take work as consultants?


I'm sure some do. You don't even need someone on the same continent as long as the remote console is available.


dr. bob said:


> I'm not sure I actually can start networking. I tried `ping` and had 0% success.


If booting from CD, it doesn't have any knowledge of your network configuration. If you can mount the / filesystem, look in /etc/rc.conf for something like this:
	
	



```
defaultrouter="a.b.c.d"
ifconfig_vtnet0="e.f.g.h netmask i.j.k.l maybe-options"
```
where all of the x.x.x.x are IPv4 addresses/masks and maybe-options is any additional stuff for your config (like "mtu 9000"). If there isn't any additional stuff, stop after the "netmask x.x.x.x". You don't type the literal word "maybe-options".
You want to translate those into shell commands so you can issue them interactively, like this:

`# ifconfig vtnet0 up` (ensure that the interface is initialized and up)
`# ifconfig vtnet0 inet e.f.g.h netmask i.j.k.l maybe-options` (set the IP address, mask and any options)
`# route add default a.b.c.d` (tell the system how to get to the outside world)


dr. bob said:


> As far as `fsck`, now I'm wondering if "unexpected soft update inconsistency" is really "corruption". Would the journal (whatever that is) being out of sync with the filesystem (maybe because I told `fsck` once not to use the journal) be considered a soft update inconsistency? Could the journal be out of sync and the filesystem still be ok? Can I just delete the journal? Would it restart from scratch? Then would there no longer be any inconsistency?
> 
> Also, does "file system was modified" necessarily mean `fsck` found problems and fixed them? I see here and elsewhere that updating the badblock and / or free inode counts, even if unchanged, can trigger that message.
> 
> I guess my wishful thinking is that the file system is (now) ok and the reason it can't boot is a new Hyper-V / old FreeBSD incompatibility.


Your system is panicing before it ever tries to mount any filesystems. The following is a _very_ simplified view of the boot process, which has complexities for BIOS vs. UEFI booting, different root partition types and a bunch of other stuff:

The boot loaders have some rudimentary knowledge of filesystems and grab the necessary pieces (kernel, modules, loader config files, etc.) and place them at special memory  locations. Then the boot loaders transfer control to the kernel which continues with initialization, etc., mounts the root filesystem (via a hint it got from the boot loader) and eventually ends up by starting the init(8) process in userspace, which then proceeds to bring the system up to the desired state (normally multiuser).

It looks like your system is failing to get even to the point where it starts any additional CPU cores/threads - refer to the "smp_started = 0" in your console output just before the panic.

I think your VM may only have a single non-hyperthreaded CPU core. Using the method _martin mentioned in comment 29, try escaping to the loader prompt as shown in that comment and typing:

```
set kern.smp.disabled=1
boot -v
```
and see if anything changes.


----------



## dr. bob (Oct 13, 2022)

Terry_Kennedy said:


> I'm sure some do. You don't even need someone on the same continent as long as the remote console is available.


Terry, do you? You seem to know what you're talking about. And then this wouldn't be all volunteer. 

Or, Martin, how about you?



Terry_Kennedy said:


> You want to translate those into shell commands so you can issue them interactively


Super, thanks! `ping` went from 0% success to 0% packet loss.



Terry_Kennedy said:


> Your system is panicing before it ever tries to mount any filesystems.


Right, if I boot from the hard disk. But if I boot from the CD, I can get through to `fsck`, which if I say to use the journal doesn't seem to find any problems, though does report "file system was modified":





Weirdly (if you ask me), `fsck -n`, which (as far as I can tell) just says not to use the journal, instead of it deciding to skip the journal, gives me that "unexpected soft update inconsistency":







Terry_Kennedy said:


> try escaping to the loader prompt as shown in that comment and typing:
> 
> ```
> set kern.smp.disabled=1
> ...


Alas, it hangs at:





I remember Martin said something about virtio, I don't know if this is related.


----------



## dr. bob (Oct 13, 2022)

_martin said:


> I was not able to reproduce your setup, I wonder what kind of qemu with hyperv enlightenments is that. But unfortunately without being able to trigger the crash it's practically nothing we can do.


Martin, would you like me to ask? I'd be happy to. "What kind of qemu with hyperv enlightenments is this?"


----------



## _martin (Oct 13, 2022)

dr. bob said:


> I guess my wishful thinking is that the file system is (now) ok and the reason it can't boot is a new Hyper-V / old FreeBSD incompatibility.


FS is just a victim of the panic you're getting. That panic is linked to the hyperv, failure in vmbus initialization routine. Every panic your FS gets dirty.



dr. bob said:


> Martin, would you like me to ask? I'd be happy to. "What kind of qemu with hyperv enlightenments is this?"


I compiled new qemu from sources on Linux to see if I can get this running I'm right now I'm failing to do so. I would be glad if you ask, I'm really curious how that VM is launched.



dr. bob said:


> I remember Martin said something about virtio, I don't know if this is related.


Virtio is not a problem here. It's the acceleration (optimization between host and guest) and initialization procedure.



dr. bob said:


> Or, Martin, how about you?


You mean install fresh VM ? I don't have problem with that. But you're trusting with your stuff to somebody from internet. That's always a big risk.


----------



## dr. bob (Oct 13, 2022)

_martin said:


> Every panic your FS gets dirty.


Martin, thanks, I didn't know that, but it makes sense.



_martin said:


> I would be glad if you ask, I'm really curious how that VM is launched.


OK, I asked.



_martin said:


> You mean install fresh VM ? I don't have problem with that. But you're trusting with your stuff to somebody from internet. That's always a big risk.


Install fresh VM and also get email, web server, etc, running. I've already been trusting you. I've invoked spells I don't understand myself! You seem knowledgeable and have been responsive. What more could I ask for?


----------



## dr. bob (Oct 13, 2022)

Hi, everyone. I asked for FreeBSD 13.1-RELEASE on a new VM. STABLE would have a later Expected EoL, but would be more experimental, and anyway is n/a?


----------



## _martin (Oct 13, 2022)

You should stick to the release versions and patch those. 13.1 is the newest right now.


----------



## _martin (Oct 13, 2022)

I don't know what I was doing last night but this command under Linux `qemu-system-x86_64 -cpu host,hv_relaxed,hv_time -accel kvm...` is doing what I want. Now it's interesting to see which enlightenments could help here.


----------



## _martin (Oct 13, 2022)

As I was able to reproduce the bug I was able to attach debugger and debug that boot. I think I can give you a fix. Most likely your provider is using qemu with those enlightenments (I'm not aware of any other hypervisor of being able to do so). If so you need to ask them to add these two: `hv_synic,hv-vpindex` into the qemu VM specification. They may have others set.

For example part of my testing vm setup:
	
	



```
qemu-system-x86_64 -cpu host,hv_relaxed,hv_time,hv_synic,hv-vpindex -accel kvm -m 1024 -s \
        -device virtio-blk-pci,id=pci0,drive=hd \
        -drive if=none,id=hd,file=fbsd10.qcow2
```


----------



## dr. bob (Oct 14, 2022)

Martin, wow. I asked them to make that change, let's see if they can do that.

In the meantime, they set me up with a new VM with 12.1-RELEASE and a disk image of the old VM. Does it make sense for me to start by upgrading to 13.1-RELEASE?


----------



## _martin (Oct 14, 2022)

I don't quite understand their reasoning, most likely they don't have available images and there's just not enough demand for them to make them (some sort of local provider I guess). If you can boot CD of your choice (13.1) you can overwrite the 12.1 with the 13.1 fresh installation. I wouldn't bother with rolling upgrades, especially on fresh system.

I'm curious about that fix. I'm not sure if they share how they launch the VM, they may consider that as their know-how. But let's see.


----------



## dr. bob (Oct 14, 2022)

Martin, thanks. Not sure how I'd boot from a CD of my choice, as opposed to a CD they gave me. Download an ISO image, mount it, and reboot from it??

`freebsd-update` only updates to the next version? are there so many versions between 12.1 and 13.1?


----------



## _martin (Oct 14, 2022)

I don't know what options you have with your provider or how you can manage the VM. Yes, idea is to download the iso and mount it. Personally I don't use freebsd-update much ( always used sources), I'd have to google myself to see which versions are available and what upgrade paths are still valid.
But with vanilla image it's easier to install it from rescue mode (new disk layout, extract sources, check fstab, loader.conf, root passwd) and off you go with shiny new 13.1.


----------



## Erichans (Oct 14, 2022)

dr. bob said:


> [...] `freebsd-update` only updates to the next version? are there so many versions between 12.1 and 13.1?


When going with freebsd-update(8) you can easily go from FreeBSD RELEASE-12.1 to a next major version such as RELEASE-13.1; consult the relevant FreeBSD 13.1-RELEASE Release Notes too.


----------



## dr. bob (Oct 16, 2022)

OK, getting ready to try updating! Thought it might be a good idea first to see what if any packages were installed? Ran into a version issue:


```
# pkg info
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.18.4...
Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1203000
- running kernel: 1201000
Ignore the mismatch and continue? [y/N]: N
```

If that `pkg` assumes 12.3, is it ok to install on 12.1? Thanks!


----------



## dr. bob (Oct 16, 2022)

took a chance and installed it. didn't seem to be an issue. now running 13.1!


----------



## dr. bob (Oct 23, 2022)

_martin said:


> I'm curious about that fix. I'm not sure if they share how they launch the VM, they may consider that as their know-how. But let's see.


Martin, you da bomb! They said "we found the right combination of odd tweaks and settings". I imagine you got them thinking there might actually be a fix + pointed them in the right direction.
How do `hv_synic` and `hv-vpindex` make a difference?


----------



## _martin (Oct 23, 2022)

Oh boy, I should have sent my bill to them then! ;-). Great, then my work was not a waste.

I attached the debugger to live FreeBSD boot and was able to see what's happening. Issue is in a very early boot code when CPU is being identified and set. There are only two reasons why GPF can occur in the case you had. While FreeBSD should not touch that part of the code under VM (hence the PR and fix in FreeBSD 11) qemu is capable of providing many features to guest (has to be on Linux as FreeBSD is missing kvm for it). With debugger I was able to tell what was FreeBSD asking for (even though it should not ask for it at all).

In a bigger picture these are speed and performance optimizations which guest could potentionaly benefit from. Here main reason for them to be enabled is to work with the buggy code of FreeBSD 10.


----------



## dr. bob (Oct 24, 2022)

so now i need to upgrade again, but this time it's not a fresh system. if i go straight from 10.3 to 13.1 will email and web server stop working? perl scripts? ssh and sftp?

also, if it's running, can i assume the file system is ok? i think i saw that the root file system is harder to `fsck`. thanks!


----------



## _martin (Oct 24, 2022)

Some may require admin touch (apache2.2 to apache2.4 for example).
Yes, if fsck was able to fix it it's ok to assume so. Also your crash was very on a virtual disk, that helps a bit too.


----------



## ralphbsz (Oct 24, 2022)

dr. bob said:


> so now i need to upgrade again, but this time it's not a fresh system. if i go straight from 10.3 to 13.1 ...


Please ask for advice for how to do such a giant upgrade in place. I've never done it, so I won't give you advice. Just a little story: About 5 years ago, I had fallen behind by two major version (I think from 9.x to 11.x), and based on the advice here, I decided to not do an upgrade, but a full reinstall of the OS (but keeping the data disks in place, those are on separate drives). YMMV, obviously.

One of the ways to prevent upgrades from turning into reinstalls (and for your system, that advice might be too late): When editing config files that are supplied in base (for example/etc/rc.local, /etc/crontab, /etc/*/*.conf), do minimal changes, leave the copyright notice at the top alone, and do not change the formatting. Yes, I understand that we all have our personal ways we like to indent and format things, but if you have changed every line, then the merging tool (which tries to combine the changes you made with the changes coming from the update) will likely fail, and you'll have to update the configurations manually.

The important part to remember is that FreeBSD upgrades works on two things separately. One is upgrading the base OS, which does not include most major applications or servers, such as perl and apache. The second one is upgrading the packages (or ports, same concept different install/upgrade mechanism). The two are quite decoupled.

And if you do lots and lots of package updates, your cache directory for these updates will eventually use lots of disk space. Use "pkg clean" to fix that.



> ... will email and web server stop working? perl scripts? ssh and sftp?


As Martin already said: I would expect most things to simply work. My current system has been upgraded from 11.1 to 12.3, and most version upgrades (about half a dozen), absolutely nothing broke. Most of the time, my entries in the sys admin log file are "upgraded to 12.X and updated packages, boring". Sometimes, upgrading a package causes incompatibility with older configurations; the example of apache is good, because I had that happen. Same with Samba. Sometimes packages no longer exist, and you need to install newer versions that have different names (for example bind91 -> bind911). The other source of upgrade incompatibilities that I experience is Python: You may have hand-installed (perhaps using pip) special python libraries that are specific to version 3.X, and are not installed yet (and perhaps not even available) in version 3.Y. The same can happen with perl. In most of these cases, just re-installing things fixes the problem.



> also, if it's running, can i assume the file system is ok?


If it passes fsck once, it is as OK as can be.


----------

