# gmirror works when booting from mfsBSD but not at boot



## Ben (Jan 5, 2013)

Hi,

I have a server which is running FreeBSD 9.0-RELEASE. It has one provider gm0 which consists of ada0 and ada1.

When I boot to mfsBSD I can load gmirror and gm0 works fine.

When I reboot from harddisk it will prompt with error 19, asking for a mountroot. This error seems to be popular after upgrading from FreeBSD 8.x to 9.x which is not the case here. It was working fine but now...

Any idea what I could do? It seems the geom_mirror.ko module can't be loaded at boot. How can I fix this?

Some output:

```
rescue-bsd# gpart show
=>       63  234441584  mirror/gm0  MBR  (111G)
         63  234436482           1  freebsd  [active]  (111G)
  234436545       5102              - free -  (2.5M)

=>        0  234436482  mirror/gm0s1  BSD  (111G)
          0  213954560             1  freebsd-ufs  (102G)
  213954560    1024000             2  freebsd-ufs  (500M)
  214978560   19457922             4  freebsd-swap  (9.3G)[/CMD]

rescue-bsd# gmirror list
Geom name: gm0
State: COMPLETE
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3106185129
Providers:
1. Name: mirror/gm0
   Mediasize: 120034123264 (111G)
   Sectorsize: 512
   Mode: r0w0e0
Consumers:
1. Name: ada1
   Mediasize: 120034123776 (111G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
   Priority: 1
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 3005574804
2. Name: ada0
   Mediasize: 120034123776 (111G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 3658582627[/CMD]
```

This is when I boot from mfsBSD.

/etc/fstab:

```
/dev/mirror/gm0s1a              /               ufs             rw      1       1
/dev/mirror/gm0s1b              /root           ufs             rw      2       2
/dev/mirror/gm0s1d              none            swap            sw      0       0
```

/boot/loader.conf

```
geom_mirror_enable="YES"
```


----------



## wblock@ (Jan 5, 2013)

After booting with 9.1, press Scroll Lock and scroll back through the kernel messages looking for errors.  This could be the graid(8) problem, could be more strict checking of MBR correctness, or something else.


----------



## Ben (Jan 5, 2013)

I'm running 9.0, not 9.1.


----------



## wblock@ (Jan 5, 2013)

Okay, so it was not due to an upgrade but one day it was working and the next it was not?  What changed?


----------



## Ben (Jan 5, 2013)

Ah ok, here is the error:



> GEOM: ada0s1: geometry does not match label (255,63s != 16h,63s).



This remote console thing is driving me nuts 

UPDATE: I don't know what changed. I normally don't take care of this system. The guy said "I just mounted a partition and maybe did mount -u instead of umount"


----------



## wblock@ (Jan 5, 2013)

That's a normal GEOM warning which can be ignored.  Keep looking.  If you can, boot in single user mode, make certain that the gmirror(8) module is loaded (gmirror load) and then manually check the mirror status and mount the filesystems read-only.  If that works, run fsck(8).


----------



## Ben (Jan 5, 2013)

"Cannot add disk ada1 to gm0 (error=22). Device gm0 destroyed."

I ran fsck already. It just seems that the gmirror module is not loaded at boot. If I boot from mfsBSD (rescue console) gmirror is loaded and the raid is just fine.

UPDATE: When I mount /dev/ada0s1a and run kldstat geom_mirror.ko is loaded.


----------



## wblock@ (Jan 5, 2013)

That's after booting in single user mode and loading the module?
`# gmirror load`
or
`# kldload geom_mirror`

Is the module still present in /boot/kernel/geom_mirror.ko?


----------



## Ben (Jan 5, 2013)

Right after single user it's already loaded. I did not load it manually at that moment.


----------



## Ben (Jan 5, 2013)

During boot it says


> "Cannot add disk ada0 to gm0 (error=22). Device gm0 destroyed."


After it says


> "Cannot add disk ada1 to gm0 (error=22). Device gm0 destroyed."


My assumption that gmirror is not loaded is wrong. It is loaded but adding the drives fails. The rescue system is a 9.1 btw. Is this important here?


----------



## wblock@ (Jan 5, 2013)

It sounds like the mirror is out of sync, or possibly the mirror metadata has been corrupted on ada1.  Since you've already mounted one disk separately, the data on the two drives is already different.  So at this point, I'd remove ada1 from the mirror, reboot so the system is running off the one-drive mirror, then add back ada1.  (Back up first, of course.)
`# gmirror remove gm0 ada1`
(reboot)
`# gmirror insert gm0 ada1`

But looking at those messages now, you say it can't add either drive to the mirror.  I have no idea what would do that.  Metadata corruption on both?


----------



## Ben (Jan 5, 2013)

Strangely it works when I mount it from the rescue system. Then gm0 works, I can mount it, read it, write etc.

You think a "rebuild" works also to repair a corruption?


----------



## Ben (Jan 5, 2013)

I removed ada1 from the gm0 provider and rebooted, but it still says it can't add ada0 to gm0.

Any idea? Otherwise I need to destroy gm0 completely and recreate it which hopefully works...


----------



## wblock@ (Jan 5, 2013)

A rebuild would just copy the data.  The metadata would be created from scratch when the drive was added to the mirror.

"The guy" somehow changed/corrupted that system.  If you could identify exactly where... it's worth looking at /etc/rc.conf to see if there have been surprise settings added that somehow prevent gmirror(8) from starting.  I don't know what that would be.  If it were me, I'd be running a backup from the mfsBSD-mounted mirror right now.

Maybe he ran freebsd-update(8)?


----------



## Ben (Jan 5, 2013)

I checked the "potential spots" but nothing found. Everything looks normal. I will boot now without gm0 and start from scratch.


----------



## wblock@ (Jan 5, 2013)

Finding out what the guy did would be useful, because it's likely to happen again.


----------



## Ben (Jan 5, 2013)

I detroyed it, relabeled it and inserted it again, but no success. Same error as before.


----------



## wblock@ (Jan 5, 2013)

And uname -a still shows it as 9.0?


----------



## Ben (Jan 5, 2013)

Yes.
[CMD="uname -a"]FreeBSD HOSTNAME 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 02:52:29 UTC 2012     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64[/CMD]

I now booted from either one of the SSDs (/dev/ada0 and /dev/ada1 after changing fstab) and it works.

I really dont know whats going on here.


----------



## Ben (Jan 5, 2013)

I ran gmirror clear on both drives once again and will create the mirror a 2nd and last time.


----------



## wblock@ (Jan 5, 2013)

The only other things I can think of would be changed entries in /boot/loader.conf or possibly BIOS settings.  Can't think of a way that would prevent the drives from being added to the mirror.

Or maybe that "ignore overlapping slices" sysctl that was new with 9.x.  If that had been removed from /boot/loader.conf...

Update: http://www.freebsd.org/releases/9.0R/relnotes-detailed.html#AEN1277


----------



## Ben (Jan 5, 2013)

The loader.conf only shows
geom_mirror_load="YES"

Nothing else. There should be something more?


----------



## wblock@ (Jan 5, 2013)

Not unless there was something else there before.  But there would be a message in the kernel startup complaining about partition integrity, and running gpart(8) from mfsBSD would show it also.


----------



## Ben (Jan 5, 2013)

No nothing, but now maybe Im a step closer. I cleared the gmirror metadata once more, rebooted from ada0, created gm0 by labeling ata0 and rebooted again. This time it worked (ada1 not yet added). Now I added ada1 and it's rebuilding.

When it's finished I will reboot once more. If it works then I guess I made it.

Anyway, thanks so much for your help and input!!


----------



## wblock@ (Jan 5, 2013)

I wish we could have found a definite cause for that.  The error message comes from sys/geom/mirror/g_mirror.c, line 3072 in 9-STABLE.  At first glance, it can give an error if metadata is corrupt.  Could there be a RAID controller in that system that was switched to RAID mode?


----------



## Ben (Jan 5, 2013)

No, no RAID controller:

```
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <SMCI > on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0xdc00-0xdc1f mem 0xfbce0000-0xfbcfffff,0xfbcdc000-0xfbcdffff irq 16 at device 0.0 on pci1
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:25:90:76:d9:74
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0xec00-0xec1f mem 0xfbde0000-0xfbdfffff,0xfbddc000-0xfbddffff irq 16 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:25:90:76:d9:75
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached)
pci0: <base peripheral> at device 22.0 (no driver attached)
pci0: <base peripheral> at device 22.1 (no driver attached)
pci0: <base peripheral> at device 22.2 (no driver attached)
pci0: <base peripheral> at device 22.3 (no driver attached)
pci0: <base peripheral> at device 22.4 (no driver attached)
pci0: <base peripheral> at device 22.5 (no driver attached)
pci0: <base peripheral> at device 22.6 (no driver attached)
pci0: <base peripheral> at device 22.7 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xcc00-0xcc1f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801JI (ICH10) USB controller USB-D> on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xc880-0xc89f irq 21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801JI (ICH10) USB controller USB-E> on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xc800-0xc81f irq 19 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801JI (ICH10) USB controller USB-F> on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xfbeda000-0xfbeda3ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> on ehci0
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xc480-0xc49f irq 23 at device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801JI (ICH10) USB controller USB-A> on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xc400-0xc41f irq 19 at device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801JI (ICH10) USB controller USB-B> on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xc080-0xc09f irq 18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6: <Intel 82801JI (ICH10) USB controller USB-C> on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfbed8000-0xfbed83ff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> mem 0xf9000000-0xf9ffffff,0xfaffc000-0xfaffffff,0xfb000000-0xfb7fffff irq 17 at device 4.0 on pci6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH10 AHCI SATA controller> port 0xb480-0xb487,0xc000-0xc003,0xbc00-0xbc07,0xb880-0xb883,0xb800-0xb81f mem 0xfbed6000-0xfbed67ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
qpi0: <QPI system bus> on motherboard
pcib7: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci255: <PCI bus> on pcib7
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <INTEL SSDSA2CW120G3 4PC10302> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <INTEL SSDSA2CW120G3 4PC10362> ATA-8GEOM_MIRROR: Device mirror/gm0 launched (1/1).
 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
GEOM: ada1s1: geometry does not match label (255h,63s != 16h,63s).
Root mount waiting for: usbus7 usbus6 usbus5 usbus4 usbus3 usbus2 usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
Root mount waiting for: usbus7 usbus3
Root mount waiting for: usbus7 usbus3
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/mirror/gm0s1a [rw]...
WARNING: /: TRIM flag on fs but cannot get whether disk supports TRIM
ugen2.2: <American Megatrends Inc.> at usbus2
ukbd0: <Keyboard Interface> on usbus2
kbd2 at ukbd0
ums0: <Mouse Interface> on usbus2
em0: link state changed to UP
GEOM_MIRROR: Device gm0: rebuilding provider ada1.
GEOM_MIRROR: Device gm0: rebuilding provider ada1 finished.
```


----------



## Ben (Jan 6, 2013)

Anyway, the second time it worked. It rebuilt gm0 and booted from the mirrored RAID again.

Puh...


----------



## wblock@ (Jan 6, 2013)

Please keep good backups.  And hunt down "the guy" and find out A) what he did and B) if he ever plans on doing that again.


----------



## migle (Jan 24, 2017)

Look, I just stumbled upon this, and the original post induced me in error.

Pay attention, in the first post, the OP has (incorrectly):

/boot/loader.conf

```
geom_mirror_enable="YES"
```

And then further ahead, the OP writes (correctly):

/boot/loader.conf

```
geom_mirror_load="YES"
```

We keep reading these things and easily miss the error. I saw the thread and wrote the same error on my /boot/loader.conf. Could this be the OP's problem as well?


----------

