# I accdidentally added a disk to ZFS array



## traustitj (Jun 3, 2013)

I am running FreeBSD 9.1.

I have been looking at answers for this question and actually found a few, but all with no solution.

I have a 8 disk raidz2 array. One disk decided to die, no problem, I added a new disk to the computer, and then accidentally did


```
zpool add mysan da1
```

so now I have 8 disks, and 1 disk out of the array. As far as I have read, this means if the da1 disk dies, the whole array goes with it, which is frightening to say the least, is that true ?

Is there absolutely no way to do this without recreating the storage from scratch ? We are talking about 7 TB of data, backups and stuff. It takes days to move to another computer and days to put back.

This is how the layout looks like now :


```
[root@q ~]# zpool status mysan
  pool: mysan
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 29K in 0h0m with 0 errors on Mon May 20 19:22:06 2013
config:

	NAME                     STATE     READ WRITE CKSUM
	mysan                    DEGRADED     0     0     0
	  raidz2-0               DEGRADED     0     0     0
	    da0                  ONLINE       0     0     0
	    6676788984868813488  UNAVAIL      0     0     0  was /dev/da5
	    da2                  ONLINE       0     0     0
	    da3                  ONLINE       0     0     0
	    da4                  ONLINE       0     0     0
	    da5                  ONLINE       0     0     0
	    da6                  ONLINE       0     0     0
	    da7                  ONLINE       0     0     0
	  da1                    ONLINE       0     0     0

errors: No known data errors
```


----------



## da1 (Jun 3, 2013)

Hi,

`zpool offline mysan da1` thought offline-ing myself is not something I would normally do . then:
`zpool detach mysan da1`
`zpool replace mysan 6676788984868813488 da1`

That should do the trick I think.



			
				traustitj said:
			
		

> As far as I have read, this means if the da1 disk dies, the whole array goes with it


You have a raidz2, why should the array die if you loose 1 disk (even 2). Because it's raid2, it is supposed to withstand 2x dead disks.

PS: maybe you should take into consideration using labels.


----------



## pillai_hfx (Jun 3, 2013)

Unfortunatly this is one of the things to be watchful for when replacing a disk as far as I know. This was discussed before a few times in different forums. Here is an example: http://forums.freebsd.org/showthread.php?t=23127. 

From your `zpool status` you can see that da1 got striped with your  raidz2 vdev . I don't think you would be able to simply detach da1 and replace your failed disk with it.


----------



## kpa (Jun 3, 2013)

You can't do detach on devices in a striped pool, once the device is part of a striped pool like in this case it is there forever. The detach operation works only on  devices in mirror vdevs.


----------



## HarryE (Jun 3, 2013)

*Don't do anything without testing*

You may lose all of your pool data if/when da1 goes bad. You cannot off-line da1, there are insufficient replicas. Just buy another similar disk and `# zpool attach mysan  da1 da8`. You'll get a regular ZFS mirror for da1 and the newly bought da8.

The pool will look like 

```
[root@q ~]# zpool status mysan
  pool: mysan
 state: ONLINE
  scan: resilvered 29K in 0h0m with 0 errors on Mon May 20 19:22:06 2013
config:

	NAME                     STATE     READ WRITE CKSUM
	mysan                    ONLINE       0     0     0
	  raidz2-0               ONLINE       0     0     0
	    da0                  ONLINE       0     0     0
	    da2                  ONLINE       0     0     0
	    da3                  ONLINE       0     0     0
	    da4                  ONLINE       0     0     0
	    da5                  ONLINE       0     0     0
	    da6                  ONLINE       0     0     0
	    da7                  ONLINE       0     0     0
          mirror-1               ONLINE       0     0     0
            da1                  ONLINE       0     0     0
            da8                  ONLINE       0     0     0
errors: No known data errors
```

As a good practice, create a test zpool using memory-mapped temporary files (md) and test the zpool commands on that pool before submitting them to the real pool, such as ]

```
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
zpool create test1 raidz2 md0 md1 md2 md3
zpool add -f test1 md4
```

Now you have a similar pool to the current one. Try any zpool commands on test1 pool, safely. Don't forget to destroy the test1 pool and md devices before rebooting.


----------

