# Adding disks, new zpools, disk/data migration process



## jkcarrol (Mar 14, 2011)

Currently my FreeBSD 8.2-RELEASE/amd64 box is setup with the following disk structure:

- 3 1TB WD Black HDDs
- 2 500 G SATA2 HDDs

I have them sliced up as follows:

- ada0 (1 TB)
---/dev/label/slash    1.9G    488M    1.3G    27%    /
---/dev/label/tmp      3.9G    795M    2.8G    22%    /tmp
---/dev/label/usr      893G     61G    761G     7%    /usr
---/dev/label/var      1.9G    408M    1.4G    22%    /var

- ada1 (500 GB)
--- part of /backup raidz1 zpool

- ada2 (1 TB)
--- /dev/label/video       902  249   580    30%    /video

- ada3 (500 GB)
--- part of /backup raidz1 zpool

- ada4 (1 TB)
--- ada4s1a (~500 GB)
----- part of /backup raidz1 zpool
---ada4s1d (~500 GB slice)
----- /dev/label/media       451  190   224    46%    /media

I have another 1 TB WD Green drive and 2 400 G drives I'm planning to use (1 in a zpool, 1 as a spare). What I'm planning to do is set things up as follows:

- 4x1 TB (raidz) to be used for /usr and /data (all the stuff under /usr/data (~70G), /video and /media will go there after the migration)
- 3x400G (2 400G slices from 500G disks + 1 400G disk) (raidz) to be used for /backup with the other 400G drive as a spare
- 2x100G (2 100G slices from 500G disks) (gmirror, raid1) to be used for /, /var and /tmp

If I didn't have this SATA optical drive, I'd use that other 400G disk and have 2 parity disks in the backup raidz, but at least I'll have the spare.

So given that, I'm trying to plan what all I will need to do to move the data around/etc to re-organize the disks. I wanted to run my plan by you all and see if it looks good before I start. 

Currently I'm using all 6 SATA ports on this motherboard (Asus P5Q-Pro) on the Intel controller - the 5 SATA disks, then a SATA bluray/dvd/cd writer. I'm trying to decide which disks to put on the other 2-port controller (Silicon Image Sil5723). Actually, I need to see if that controller is even supported. Assuming it is, I'm thinking I'll put the bluray drive and the 400 drive on there, although perhaps I shouldn't trust one of my disks in the /backup raidz to be on that 3rd party controller?

I'm booting from ada0's MBR (I think), so that means I will need to install the boot loader on the gmirror once that's setup then move one of those disks to the first SATA port (or change the BIOS drive order so that disk is first), right? With that in mind, here's my plan of attack:

1. make sure the other SATA controller works. If not I'll have to wait until I can purchase a PCI-E controller add-on, or forgo these plans for a while. Update fstab to temporarily remove the mounts for /video and /media (will mount them manually later). Question: when is the best time to remove any glabels I have since I'll want to re-create them on the new gmirror later?

2. Install the 2 new drives. Move the disks around such that the 1st 500 G drive is ada0, 2nd 500 G drive ada1. Then the next 4 (ada2 - ada5) for the 1 TB drives. On the Silicon Image controller, I'll put the 400G drive on port0 then the bluray drive on port1. I'll confirm in the BIOS that the old ada0 that I'm currently booting from is set as the "first" disk (until later).

3. Boot normally and copy the existing stuff in /backup to /video. Then destroy the zpool to free up the two 500 G drives and the 500 G slice on the one 1TB drive.

4. Slice up the 500G disks such that I have two 400 G partitions equal to the size of a full-disk slice on the 400G disk. Then calculate whatever is left and create those for the gmirror. I think this will mean a ~100G ada0s1 and ada1s1 and 400G ada0s2 and ada1s2, right? So I'll gmirror ada0s1 and ada1s1 then create my slices in gm0s1 for /, /tmp and /var) - Question: can I create gm0s1b for swap or do I have to waste a few G on both ada0 and ada1 to create swap?

5. Format and create /, /tmp and /var on the gmirror, then install the boot loader to the gmirror (or ada0). Copy over the existing /, /tmp and /var and (temporarily) /usr (except /usr/data) to / on the gmirror since we'll need a usable /usr when creating the "data" zpool.

6. Setup the 400 G disk such that I now have three 400 G partitions (ada0s2, ada1s2, ada6s1) of equal size. Then create the backup zpool from these 3 400G chunks and mount it at /backup.

7. Copy all of my data from /usr, /media and /video (temporarily) to the /backup ZFS mount.

8. Reboot and change the disk order in the BIOS so the 500G disk is booted from (the new ada0) and boot with the gmirror setup as the root fs (/ (including temporary /usr), /tmp, /var).

9. The data that was on the 1 TB disks is now all copied to /backup, clear the partition tables on those (dd ...) then create new disk-spanning slices to use for the 4-disk "data" zpool. Once the zpool is created, copy the /usr/data, /video and /media stuff from /backup back to /usr/{data,video,media} on the new zpool and copy the old backup data from the video sub-dir back to /backup, then remove the usr/data, video and media data from the new /backup.

10. Update fstab on gmirror's /etc and (re)create glabels for slash (/), var (/var), tmp (/tmp) and swap. Ensure zpools are setup and mounting properly and set mountpoints. Delete the /usr from the gmirror / then ensure /usr is the mountpoint for the "data" zpool.


Am I missing anything major here? I think my only uncertainty is dealing with the glabels, but worst case scenario I can mount/use things directly from adaX instead of the labels.


----------



## jalla (Mar 14, 2011)

The Sil5723 isn't supported so you are limited to 6 onboard SATA ports.


----------



## jkcarrol (Mar 15, 2011)

Actually, I got them to work just fine. I enabled the Marvell IDE onboard controller, which allowed me to change the BIOS option for "Drive Xpert" to "normal" mode. It claims in the BIOS that only the first E_SATA port will work, but both drives show up in FreeBSD and work fine. They show up in camcontrol devlist along with a port multiplier:


```
# camcontrol devlist
<WDC WD4000AAKS-00A7B0 01.03B01>   at scbus0 target 0 lun 0 (ada0,pass0)
<Port Multiplier 57231095 2106>    at scbus0 target 15 lun 0 (pass1,pmp0)
<WDC WD1001FALS-00J7B0 05.00K05>   at scbus2 target 0 lun 0 (ada1,pass2)
<WDC WD5000AAKS-07TMA0 12.01C01>   at scbus3 target 0 lun 0 (ada2,pass3)
<WDC WD1001FALS-00J7B0 05.00K05>   at scbus4 target 0 lun 0 (ada3,pass4)
<WDC WD5000AAKS-00YGA0 12.01C02>   at scbus5 target 0 lun 0 (ada4,pass5)
<WDC WD1001FALS-00J7B0 05.00K05>   at scbus6 target 0 lun 0 (ada5,pass6)
<ATAPI iHBS212   2 5L06>           at scbus7 target 0 lun 0 (cd0,pass7)
```

I'm not entirely sure it's even using/coming from the Silicon Image hardware. Because there is nothing referencing Silicon Image in demsg, pciconf, nor at boot time really. I think the Silicon Image hardware acts as a way to do raid0 or raid1, but if it's set to "normal" mode, then the two extra sata (esata) ports work just fine, at least with ahci(4).

This is what I see in *dmesg*:


```
atapci0: <Marvell 88SX6121 UDMA133 controller> port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem 0xfeaffc00-
0xfeafffff irq 16 at device 0.0 on pci2
ahci0: <Marvell 88SX6121 AHCI SATA controller> on atapci0
```

So anyway, I figured I would try it just to see if somehow they'd work as generic AHCI, and it turns out they work well with what appears to be the Marvell controller.

Unfortunately for me, the 1TB disk I had is bad so I put the two 400G drives in and I'm using a 3x400G raidz1 and have the extra 400G disk out in the event of a disk failure I have a spare (not a hot spare, but still). I'm still debating if I should bother making a 3x1TB raidz for my "data" or just keeping it the way it is. I already created all the slices on ada0 and ada1 and gmirror'd them for /, swap, /var and /tmp and I'm moving the data over. Then I should be able to boot from that and then create the raidz if I want, or just leave it alone. Still haven't decided if it's worth it. There won't be any performance benefit to a 3-disk raidz1 over single disks and I have more usable space if I want to free up part of the 1TB disk which used to be part of the old backup raidz.


----------

