# Gmirror before Installation



## Mayhem30 (Apr 19, 2020)

Are there any instruction anywhere on how to Gmirror your disks before installation?

I found this how-to guide for UEFI : https://forums.freebsd.org/threads/install-uefi-freebsd-on-gmirror.71904/

Is anyone able to convert the following for non-UEFI?


```
kldload geom_mirror
dd if=/dev/zero of=/dev/ada0 bs=1M  <<< I wipe my disks before use as RAID <<<
dd if=/dev/zero of=/dev/ada1 bs=1M  <<< You better know what your doing here <<<
gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s gpt /dev/mirror/gm0
gpart add -t efi -a4k -s 200M /dev/mirror/gm0
gpart add -t freebsd-ufs /dev/mirror/gm0
newfs_msdos -F 32 -c 1 /dev/mirror/gm0p1
mount -t msdosfs /dev/mirror/gm0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt
newfs -U /dev/mirror/gm0p2
mount /dev/mirror/gm0p2 /mnt
echo "/dev/mirror/gm0p2 / ufs rw 1 1" >> /tmp/bsdinstall_etc/fstab
echo 'geom_mirror_load="YES"' >> /tmp/bsdinstall_boot/loader.conf
exit
```


----------



## Jose (Apr 19, 2020)

There's one in the Handbook:








						Chapter 20. GEOM: Modular Disk Transformation Framework
					

In FreeBSD, the GEOM framework permits access and control to classes, such as Master Boot Records and BSD labels, through the use of providers, or the disk devices in /dev.




					www.freebsd.org
				




I also found this useful:








						HOW TO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme - A-Team Systems
					

HOW TO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme




					www.ateamsystems.com


----------



## chrbr (Apr 19, 2020)

Please have a look at http://www.wonkity.com/~wblock/docs/html/gmirror.html. This article does not consider UEFI. It is quite old but it should be still valid.


----------



## Phishfry (Apr 19, 2020)

Do you want to use MBR or GPT for your legacy BIOS installation? You can use either.


----------



## Phishfry (Apr 19, 2020)

One of the problems with following someone elses instructions are they may not be the best setting for your drives.
Take this as shown in the handbook for gmirror. 





> -a 4K


This is fine for actual 4K drives but you need to research your physical drive sectors.
Blindly putting 4K is not a good solution as it will reduce your disk throughput if you have 512 sector drives..


----------



## Mayhem30 (Apr 20, 2020)

Phishfry said:


> One of the problems with following someone elses instructions are they may not be the best setting for your drives.
> Take this as shown in the handbook for gmirror.
> This is fine for actual 4K drives but you need to research your physical drive sectors.
> Blindly putting 4K is not a good solution as it will reduce your disk throughput if you have 512 sector drives..



One of the guides suggested to use "1m" for SSD's - is that the right setting?

Here is what I have so far (untested) :

Does this look right?


```
gmirror load

gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s MBR mirror/gm0
gpart add -t freebsd -a 1m mirror/gm0

gpart create -s BSD mirror/gm0s1
gpart add -t freebsd-ufs  -a 1m       mirror/gm0s1
gpart add -t freebsd-swap -a 1m -s 4g mirror/gm0s1

gpart bootcode -b /boot/mbr mirror/gm0
gpart set -a active -i 1 mirror/gm0
gpart bootcode -b /boot/boot mirror/gm0s1

newfs -t -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
```


----------



## Jose (Apr 20, 2020)

Mayhem30 said:


> ```
> gpart add -t freebsd-ufs  -a 1m       mirror/gm0s1
> gpart add -t freebsd-swap -a 1m -s 4g mirror/gm0s1
> ```


I think "gpart add" with no size (-s) parameter will allocate all the space until the end of the slice. You'll wind up with a single "a" partition that takes up the entire mirror, and the second command to allocate swap will fail. You'll have to use the -i parameter to specify the "index" of the partition you are adding. These indexes show up as letters in the partition names e.g., /dev/mirror/gm0s1a will be the partition at index 1. I dunno why gpart(8) doesn't use letters to match the partition device names. Makes it confusing.

I think what you're trying to do is in English "make one 4GB swap partition, and use the rest of the disk for a root partition". If so, the gpart commands you want are:

```
gpart add -t freebsd-swap -a 1m -s 4g -i 2 mirror/gm0s1
gpart add -t freebsd-ufs  -a 1m -i 1       mirror/gm0s1
```

N.B., You cannot use index 3, aka the "c" partition. That is reserved and always spans the entire slice.


----------



## Mayhem30 (Apr 20, 2020)

Ok, thank you.

My last question is the "1m" part .. is that th correct setting for Samsung 860 Pro 256GB SSD's?


----------



## Phishfry (Apr 20, 2020)

I believe you want 4K for those drives.
SSD's tend to lie about their geometry.
Basically you have 3 common sector sizes.
512
4K
8K
Even with a mirror you would not add two together (ie 1024)
Each mirrored disk is independant.


----------



## Mayhem30 (Apr 20, 2020)

Thank you. My final code now looks like this :


```
gmirror load

gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s MBR mirror/gm0
gpart add -t freebsd -a 4k mirror/gm0

gpart create -s BSD mirror/gm0s1
gpart add -t freebsd-swap -a 4k -s 4g -i 2 mirror/gm0s1
gpart add -t freebsd-ufs -a 4k -i 1 mirror/gm0s1

gpart bootcode -b /boot/mbr mirror/gm0
gpart set -a active -i 1 mirror/gm0
gpart bootcode -b /boot/boot mirror/gm0s1

newfs -t -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
```


----------



## gpw928 (Apr 22, 2020)

That looks very plausible.
Last time I wanted to do this on a high value live physical system, I built a disposable VM to test it first.

I see that you are placing only one primary partition on the disk, that you have turned on TRIM for the file systems, and that soft updates journaling is off.

Here are the notes I kept to clear the metadata, if you need to repeat or test the process:
	
	



```
# Get the sector count for /dev/ada0 and divide it by "1024x1024x1024".
# This is the number of GiB on the drive.  Call it NNN.  Subtract 5 => NN5.
# Nuke the start and end of the drive (partition table and gmirror metadata).
dd if=/dev/zero of=/dev/ada0 bs=1g count=2
dd if=/dev/zero of=/dev/ada0 bs=1g seek=NN5 count=999999
# You must clean ada0 of gmirror metadata *before* you load geom_mirror
```


----------



## Mayhem30 (Apr 22, 2020)

I read that it's a good idea to create a ~4GB partition and leave it empty so the drive can use those sectors for wear-leveling.

Is that still a good practice for new SSD's?


----------



## gpw928 (Apr 22, 2020)

SSDs generally have integrated (hidden) over-provisioning.  Good ones generally have more than crappy ones. 

Creating an empty partition (and TRIMming it) increases the over-provisioning of the SSD.

There's a pretty good overview of why you need over-provisioning here.

You should not need to manually over-provision unless your duty cycle is unusually heavy, or TRIM is not working, or both.  In that case you should be monitoring the SSD with sysutils/smartmontools with a view to understand what's happening.

Some years ago I saw a post on this forum from somebody who was manually setting aside 50% of the SSD capacity for additional over-provisioning.  However I don't believe TRIM was supported back then, and the duty cycle was manic.


----------



## Mayhem30 (Apr 22, 2020)

Thanks, this is great information!


----------



## gpw928 (Apr 22, 2020)

As an observation, I have just ordered all the parts I need to retire my last remaining gmirror root.

I'm replacing two spinning disks with a pair of SSDs, on which I shall install a Root-on-ZFS mirror.

I built the system before you could boot from ZFS, but the functional advantages you get from a ZFS root (including snapshots and boot environments) are just too good to ignore any longer.


----------



## Mayhem30 (Apr 22, 2020)

gpw928 said:


> Here are the notes I kept to clear the metadata, if you need to repeat or test the process:
> 
> 
> 
> ...



Could you not just do a "gpart delete/destroy" and "gmirror clear" instead?


----------



## gpw928 (Apr 22, 2020)

I would have to test it, and don't have the facilities right now.  If you are aware of metadata being at the end of the disk, then fore-warned if fore-armed.


----------



## Mayhem30 (Apr 23, 2020)

I've tried several times and everything works … until the reboot.

Then it get stuck in endless reboots and I am unable to see what the issue is - as I am using a virtual KVM to connect to the server and it's lagged.

How can I access dmesg on ada0 booting from the FreeBSD usb key installer?


----------



## D-FENS (Apr 23, 2020)

Do you necessarily want to mirror the whole disk? You could partition the disks separately but identically, for example: ada0p1/ada1p1: 512k freebsd-boot, ada0p2/ada1p2 - the rest.
You copy the boot code to both ada0p1 and ada1p1.
Then you can create a mirror on top of ada0p2 and ada1p2 and put your root on it. You can create another mirror for the swap but I never mirror swap - unless you need a hot-pluggable or a more crash resistant system, it's better to add the swap partitions separately - you get a larger swap.
The only thing that's worth mirroring is your data, the rest just complicates the boot process.

I've been using this configuration for years and rarely had problems booting. Mostly when I added or reformatted disks and forgot to put the bootcode on them.


----------



## gpw928 (Apr 24, 2020)

Mayhem30 said:


> How can I access dmesg on ada0 booting from the FreeBSD usb key installer?


Do you have physical access to the system?  If so, use a real keyboard and monitor.

If not, please describe exactly how you get to the console.

Have you done this prior to rebooting (I think you should boot from one disk without it)?

```
echo 'geom_mirror_load="YES"' >>/boot/loader.conf
```
Before you exit the installation shell, collect some status information.  

By way of example, this is what I have on my gmirror.  I have three primary partitions to accommodate two operating systems (booting from the third partition), shared swap, and logging:
	
	



```
[sherman.136] $ gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ada0 (ACTIVE)
                      ada1 (ACTIVE)
```


```
[sherman.137] $ gpart show
=>       63  488397104  mirror/gm0  MBR  (233G)
         63          1              - free -  (512B)
         64  220200960           1  freebsd  (105G)
  220201024   33554432           2  freebsd  (16G)
  253755456  220200960           3  freebsd  [active]  (105G)
  473956416   14440751              - free -  (6.9G)

=>        0  220200960  mirror/gm0s1  BSD  (105G)
          0   10485760             1  freebsd-ufs  (5.0G)
   10485760   10485760             2  freebsd-ufs  (5.0G)
   20971520  199229440             4  freebsd-ufs  (95G)

=>       0  33554432  mirror/gm0s2  BSD  (16G)
         0  25165824             1  freebsd-swap  (12G)
  25165824   2097152             2  freebsd-swap  (1.0G)
  27262976   2097152             4  freebsd-swap  (1.0G)
  29360128   4194304             5  freebsd-swap  (2.0G)

=>        0  220200960  mirror/gm0s3  BSD  (105G)
          0   10485760             1  freebsd-ufs  (5.0G)
   10485760   10485760             2  freebsd-ufs  (5.0G)
   20971520  199229440             4  freebsd-ufs  (95G)
```


```
[sherman.135] $ gmirror list
Geom name: gm0
State: COMPLETE
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3163051507
Type: AUTOMATIC
Providers:
1. Name: mirror/gm0
   Mediasize: 250059349504 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r4w4e11
Consumers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: DIRTY
   GenID: 0
   SyncID: 1
   ID: 707674990
2. Name: ada1
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: DIRTY
   GenID: 0
   SyncID: 1
   ID: 76102181
```


----------



## Mayhem30 (Apr 24, 2020)

Yes, I have the entry in loader.conf  before rebooting (after the install). I do not have physical access to the server.

I am able to uses scroll lock and see some information:


```
GEOM_MIRROR: Device mirror/gm0 launched (2/2).
Root mount waiting for : usbbus0
Root mount waiting for : usbbus0
...
...
Root mount waiting for : usbbus0
Root mount waiting for : usbbus0
...
...
Root mount waiting for : usbbus0
Root mount waiting for : usbbus0
...
mountroot> ugen0.3:  <Peppercon AG Multidevice> at usbbus0 (disconnected)
```

There is no mention of "Trying to mount root from ufs:/dev/mirror/gm0s1a".


----------



## gpw928 (Apr 24, 2020)

Have you changed your boot device in the BIOS from USB to the first SATA disk?


----------



## Mayhem30 (Apr 24, 2020)

No, but it appears my /etc/fstab was empty .. but I am still having an issue.

After booting up the installer and going to shell, I was able to mount ada0s1a and add the missing entries.

After rebooting, everything loaded up fine. The entries were missing again from fstab (gmirror sync got rid of them?).

So I added them again, and rebooted the system and now getting "the following sectors could not be read" errors and the boot fails.


----------



## gpw928 (Apr 24, 2020)

By mounting a physical disk, you have compromised the mirror.

From the emergency shell, you have to "kldload geom_mirror" and mount /dev/mirror/gm0s1a.

I'm looking at the manual here, and have tested nothing, so beware...

I would try to deactivate the second disk from the emergency shell, edit as required, boot form the mirror, and reactivate the second disk:
	
	



```
# boot from usb
kldload geom_mirror
gmirror status
gmirror deactivate gm0 ada1
gmirror status
# mount /dev/mirror/gm0s1a and edit as required
reboot # from sata
gmirror activate gm0 ada1
gmirror status
```


----------



## Mayhem30 (Apr 24, 2020)

Ok, before doing so I would like to reinstall the OS because I tried to fsck and that made things even worse.

The commands I ran were :


```
gmirrow destory gm0

gpart destroy -F ada0s1
gpart destroy -F ada1s1

gpart destroy -F ada0
gpart destroy -F ada1
```

Now gpart show shows - (unable to copy/paste from console window)






Any idea how to clear the leftover fluff on diskid/DISK-**** ?


----------



## gpw928 (Apr 24, 2020)

See my earlier post on how to clear the metadata, using dd(1).


----------



## Mayhem30 (Apr 24, 2020)

I'm confused how to calculate the number required for the second step ... (it's been a long day)


----------



## gpw928 (Apr 24, 2020)

It's 238 GiB.  To remove all metadata (the first 2 GiB and the last 8 GiB):
	
	



```
dd if=/dev/zero of=/dev/ada0 bs=1g count=2
dd if=/dev/zero of=/dev/ada0 bs=1g seek=230 count=999999
```


----------



## Mayhem30 (Apr 24, 2020)

Weird .. the commands ran fine but the partition information still shows the same as before.

Is a reboot required after running the commands?


----------



## Mayhem30 (Apr 24, 2020)

Scratch that .. everything is fine now. I'll try your new method for editing fstab in the morning.

Thanks again for all your help.


----------



## gpw928 (Apr 24, 2020)

I expect you figured it out by now, but the disks should not be in use when the metadata are cleared.

You can do it to an unused disk from a running system.

Otherwise you should be booting from USB/DVD, using an installation/emergency shell, and clearing the metadata just prior to rebuilding the mirrors.


----------



## Jose (Apr 24, 2020)

gpart destroy -F(8) did the trick for me.


----------



## Mayhem30 (Apr 25, 2020)

Ok, after much trial and error I finally got it working.

What I had to do was :

1) Boot from memstick and select "shell".

```
gmirror load
gmirror destroy gm0

gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s MBR mirror/gm0
gpart add -t freebsd -a 4k mirror/gm0

gpart create -s BSD mirror/gm0s1
gpart add -t freebsd-swap -a 4k -s 4g -i 2 mirror/gm0s1
gpart add -t freebsd-ufs -a 4k -i 1 mirror/gm0s1

gpart bootcode -b /boot/mbr mirror/gm0
gpart set -a active -i 1 mirror/gm0
gpart bootcode -b /boot/boot mirror/gm0s1

newfs -t -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt

exit
```

2) Install the OS

At this point, if you try to access "/mnt" in a shell, it will be empty.

3) Reboot and boot from memstick again and select "shell".

```
gmirror load
```

4) Ignore the "ada0 + ada1 could not be added to mirror" error message.

```
mount /dev/mirror/gm0s1a /mnt
```

5) Edit the necessary files needed for gmirror

```
echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf

vi /mnt/etc/fstab
/dev/mirror/gm0s1a  /           ufs     rw      1       1
/dev/mirror/gm0s1b  none        swap    sw      0       0
```

6) Reboot

I hope this helps someone in the future.


----------



## Phishfry (Apr 25, 2020)

Mayhem30 said:


> echo "/dev/mirror/gm0p2 / ufs rw 1 1" >> /tmp/bsdinstall_etc/fstab
> echo 'geom_mirror_load="YES"' >> /tmp/bsdinstall_boot/loader.conf


You can skip your step 5 by adding the above lines when you drop to a shell. You will need to change the partition name.
(Instead of dropping to the shell when you boot off the memstick use 'Install' and drop to shell during Disk Partition phase)
/temp/bsdinstall_etc and /temp/bsd_install_boot are inside the chroot and point to the installs /etc and /boot.


----------



## Mayhem30 (Apr 26, 2020)

I tried that originally, and for some reason the changes did not get carried over.


----------



## gpw928 (Apr 26, 2020)

The FreeBSD installation process has an option to place file systems manually.

So after you have used the initial shell to set up the mirror, and make the file systems, you exit the shell and choose "Install".

Proceed as normal, but choose manual file system partitioning.  Assign the partitions that you have created (e.g. gm0s1) to their file systems (e.g. root).  You have to use TAB and arrows to navigate (it's quite finnicky, and not terribly intuitive, but will work).

If you do that correctly, /etc/fstab will be created correctly.

Just before the installation completes, take the shell exit offered and fix up /boot/loader.conf


----------

