# Possible to determine disk order in raidz2?



## littlesandra88 (Feb 19, 2013)

Hi =)

I would like to create a raidz2 of 36 disks with 6 disks in each raidz2, where I determine which disks that should go in each raidz2.

If I do


```
zpool create tank3 raidz2 da0 da1 da2 da3 da4 da5 \
                         raidz2 da6 da7 da8 da9 da10 da11 \
                         raidz2 da12 da13 da14 da15 da16 da17 \
                         raidz2 da18 da19 da20 da21 da22 da23 \
                         raidz2 da24 da25 da26 da27 da28 da29 \
                         raidz2 da30 da31 da32 da33 da34 da35
```

then I get that the first raidz2 is made from

`# da0, da8, da4 ,da9, da2, da3`

instead of

`# da0, da1, da2, da3, da4, da5`

as I would have expected.

Does anyone know how the order of each raidz2 can be forced?


----------



## Sebulon (Feb 19, 2013)

Does it behave the same way if you go like:
`# zpool create tank3 raidz2 da{0,1,2,3,4,5} raidz2 da{6,7,8,9,10,11} ...`

/Sebulon


----------



## usdmatt (Feb 19, 2013)

Well that's interesting. Each vdev should be made up of the drives that directly follow the vdev type in the command. The order inside the vdev can't be controlled (i.e. a mirror may end up as da0,da1 or da1,da0 which doesn't really matter) but if it's mixing drives as you state, then it suggests there's a bug in the create command.

If all else fails, just create the pool with one raidz2 vdev using da0-5, then use 'zpool add' to add the other vdevs one by one. If it still ends up mixing them up then there's something really wrong somewhere.


----------



## kpa (Feb 19, 2013)

You can control the order in a mirror if you zpool attach a disk, the attached disk will be placed after the disk it was attached to.


----------



## phoenix (Feb 19, 2013)

Label the drives (or, create a GPT partition that's 4K-aligned and label that), then use the labels in the zpool create command.

Or, if you are bound and determined to use device nodes directly, do it in separate commands.  Create the pool with 1 raidz vdev.  Then "zpool add" another raidz vdev.  And repeat until all vdevs are added.


----------



## littlesandra88 (Feb 22, 2013)

Hello all,

And thanks a lot for all the solutions =)

I were able to reproduce it on two hosts, but after I

`# zpool destroy tank3`

and then the "zpool create tank3 raidz2 da0 da1 ..." I haven't been able to reproduce the problem.

Somehow the pool must have been corrupted, as afterwards

`# zdb -d tank3`

worked.

*UPDATE*

But I have now noticed that after a reboot, the disks are now randomized again in the pool,

`# zdb -d tank3`

fails again.


----------



## littlesandra88 (Feb 22, 2013)

I am very temped to think that my hardware is corrupt.


```
[root@nas3s ~]# zpool destroy tank3

[root@nas3s ~]# zpool status
no pools available

[root@nas3s ~]# zpool create tank3 raidz2 da{0,1,2,3,4,5}
```

Everything works as it should.


```
[root@nas3s ~]# reboot

[root@nas3s ~]# zpool status
  pool: tank3
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank3       ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da5     ONLINE       0     0     0

errors: No known data errors

[root@nas3s ~]# zdb -d tank3
zdb: can't open 'tank3': Device not configured
```

The vdev now contains random devices.


My two NAS's which makes the exact same error on FreeBSD 9-p6 is

* Supermicro CSE-847E16-R1400LPB chassis, 36 HS bays, 1400W redundant PSU
* Supermicro H8DG6-F AMD Dual G34 mainboard
* AMD Opteron 6320, 2.8GHz 8-core, 8MB L2 cache, 6400MT

What kind of hardware failure can this be?


----------



## cpm@ (Feb 22, 2013)

Switch from running 9-RELEASE-p6 to running the 9.1-RELEASE with freebsd-update(8). Read FreeBSD 9.1-RELEASE Installation Instructions. Note that ZFS will stay at pool version 28 (until Oracle releases CDDL code). For more information reads here.


----------



## littlesandra88 (Feb 22, 2013)

@cpu82

Is this a known bug? (Please let it be =) )

The reason I installed 9.0 and not 9.1 was because

`# cd /usr/ports && make fetchindex`

was broken, and the fix was to compile perl, which I think was a pretty serious bug to have in a new release. So I didn't dare to go 9.1.

If I do

`# freebsd-update -r 9.1-RELEASE upgrade`

would "make fetchindex" then work, or would I still have to recompile perl?


----------



## cpm@ (Feb 22, 2013)

littlesandra88 said:
			
		

> Is this a known bug? (Please let it be =) )
> 
> The reason I installed 9.0 and not 9.1 was because
> 
> ...



If you use the default make index target then you need to install perl in order to build an INDEX from scratch. But if you want download a pre-built INDEX by make fetchindex, then no need perl. Indeed, you can build and install ports very happily without INDEX on your system.



			
				littlesandra88 said:
			
		

> If I do
> 
> `# freebsd-update -r 9.1-RELEASE upgrade`


To upgrade the command to run is as follows:
`# freebsd-update upgrade -r 9.1-RELEASE`

To successfully update check out http://www.freebsd.org/releases/9.1R/relnotes-detailed.html#upgrade.


----------



## wblock@ (Feb 22, 2013)

fetchindex should have worked, the problem was that the index was not being rebuilt.  That was fixed a couple of weeks ago.  But most FreeBSD systems, even servers, end up with Perl installed as a dependency of something else anyway.


----------



## cpm@ (Feb 22, 2013)

wblock@ said:
			
		

> fetchindex should have worked, the problem was that the index was not being rebuilt.  That was fixed a couple of weeks ago.  But most FreeBSD systems, even servers, end up with Perl installed as a dependency of something else anyway.



wblock@

Thanks for the clarification


----------



## littlesandra88 (Feb 22, 2013)

@cpu82

Ahhh. Now it works =) Upgrading to 9.1 was really easy.

However when I wanted to upgrade the spare NAS, I did a really bad mistake, when I wanted to install "screen". I typed

`# cd /usr/ports && make install name=screen`

and it started to download and compile all kinds of things.

It ended with


```
ultrix-gcc

Then type 'make <config>' (ex: 'make linux-x86')

Or, run './configure' then 'make'
See './configure --help' for details

(ignore the following error message)
gmake: *** [configs/current] Error 1
*** Error code 1

Stop in /usr/ports/graphics/dri.
*** Error code 1

Stop in /usr/ports/x11-servers/xorg-vfbserver.
*** Error code 1

Stop in /usr/ports/accessibility/accerciser.
*** Error code 1

Stop in /usr/ports/accessibility.
*** Error code 1

Stop in /usr/ports.
```
What just happened? Can it be reversed?

*UPDATE*

Complete output log

http://pastebin.com/cwr678bp


----------



## _martin (Feb 22, 2013)

Where are those disks coming from ? Are they behind some FC adapter ? I highly doubt (I even bet money on it) pool won't get imported upon boot (and gets healthy) if some of the disks are missing. 

Did you try to check serial # of those disks afterwards ? i.e. that with different names and order, same amount of the serial # (or WWNs) are in that pool? Built-in diskinfo -v can help, I like smartctl from sysutils/smartmontools though.


----------



## cpm@ (Feb 22, 2013)

Please, show output (run in port that failed to build):
`# pwd`


----------



## littlesandra88 (Feb 22, 2013)

@cpu82

"pwd" when it stopped was "/usr/ports".

I have just updated the previous post with a complete output dump.


```
Stop in /usr/ports/accessibility.
*** Error code 1

Stop in /usr/ports.
[root@nas3 /usr/ports]# pwd
/usr/ports
[root@nas3 /usr/ports]# ls   
.cvsignore	accessibility	editors		math		shells
CHANGES		arabic		emulators	misc		sysutils
COPYRIGHT	archivers	finance		multimedia	textproc
CVS		astro		french		net		ukrainian
GIDs		audio		ftp		net-im		vietnamese
INDEX-9		benchmarks	games		net-mgmt	www
KNOBS		biology		german		net-p2p		x11
LEGAL		cad		graphics	news		x11-clocks
MOVED		chinese		hebrew		palm		x11-drivers
Makefile	comms		hungarian	polish		x11-fm
Mk		converters	irc		ports-mgmt	x11-fonts
README		databases	japanese	portuguese	x11-servers
Templates	deskutils	java		print		x11-themes
Tools		devel		korean		russian		x11-toolkits
UIDs		distfiles	lang		science		x11-wm
UPDATING	dns		mail		security
[root@nas3 /usr/ports]#
```


----------



## littlesandra88 (Feb 22, 2013)

@matoatlantis

You are absolutely right. No FC adapter =)

Going through all the serial numbers, they are all unique.

Upgrading to FreeBSD 9.1-p1 fixed the problem when I destroyed the pool, and created it again.


----------



## _martin (Feb 22, 2013)

littlesandra88 said:
			
		

> @matoatlantis
> Going through all the serial numbers, they are all unique.



Hm, so are they the same ? Same "pool" of serial numbers if you will. It might be they got discovered in different order upon boot. But as you're saying it got fixed in one of the prerelease, it might have been a bug then.


----------



## cpm@ (Feb 22, 2013)

Seems a bug, I need to take a look to /usr/ports/graphics/dri/Makefile.


----------



## littlesandra88 (Feb 22, 2013)

@cpu82

Here you go =)


```
# New ports collection makefile for:    dri
# Date created:         8 Nov 2003
# Whom:                 anholt@FreeBSD.org
#
# $FreeBSD: ports/graphics/dri/Makefile,v 1.36 2011/02/25 16:52:06 miwi Exp $
#

PORTNAME=       dri
PORTVERSION=    ${MESAVERSION}
PORTEPOCH=      2
CATEGORIES=     graphics

COMMENT=        OpenGL hardware acceleration drivers for the DRI

LIB_DEPENDS=    drm:${PORTSDIR}/graphics/libdrm \
                expat.6:${PORTSDIR}/textproc/expat2
BUILD_DEPENDS=  makedepend:${PORTSDIR}/devel/makedepend

CONFLICTS=      dri-6.2.2005* dri-6.5.2006*
MAKE_JOBS_UNSAFE=       yes

USE_XORG=       glproto x11 xext xxf86vm xdamage xfixes dri2proto

EXTRA_PATCHES+= ${FILESDIR}/patch-mach64_context.h \
                ${FILESDIR}/patch-sis_context.h


do-install:
        cd ${WRKSRC}/src/mesa; ${GMAKE} install-dri

.include "${.CURDIR}/../../graphics/libGL/bsd.mesalib.mk"

.include <bsd.port.pre.mk>

.if ${ARCH} == "ia64"
BROKEN=         Does not install on ia64
.endif

.include <bsd.port.post.mk>
```


----------



## cpm@ (Feb 22, 2013)

Try following (e.g. freebsd-dri if uses kernel FreeBSD) :
`# cd /usr/ports/graphics/dri && make clean && [b]g[/b]make freebsd-dri && make install`

Instead, freebsd-dri-x86 if uses x86-fbsd or freebsd-dri-amd64 if uses amd64-fbsd. Anyway, the version you want to install of graphics/dri is old. 

Have updated the ports tree using portsnap(8), don't you?
`# portsnap fetch update`


----------



## littlesandra88 (Feb 22, 2013)

@cpu82


```
[root@nas3 /usr/ports]# cd /usr/ports/graphics/dri && make clean && make freebsd-dri && make install
===>  Cleaning for makedepend-1.0.3,1
===>  Cleaning for gmake-3.82
===>  Cleaning for glproto-1.4.12
===>  Cleaning for dri2proto-2.3
===>  Cleaning for libXxf86vm-1.1.1
===>  Cleaning for libdrm-2.4.12_1
===>  Cleaning for xf86vidmodeproto-2.3.1
===>  Cleaning for dri-7.4.4,2
make: don't know how to make freebsd-dri. Stop
[root@nas3 /usr/ports/graphics/dri]#
```

I don't really know what I want =) I am still on FreeBSD 9.0 on this one. I didn't dare to upgrade to 9.1 before damage control had been done on my bad typo =)

Or wouldn't you worry about it, and I should just upgrade to 9.1?

UPDATE

I did 


```
portsnap fetch update
portsnap extract
```

which I probably shouldn't have done I suppose. I see Quake among many other things now being extracted.

Can it be saved, or would it be better to reinstall 9.1 from scratch?


----------



## cpm@ (Feb 22, 2013)

Is preferible upgrade to 9.1-RELEASE, but using old-style can cause error like these. Why don't using portmaster(8) to update your ports? Is easy and safe.

*EDIT*

Be save install from previous 9-RELEASE-p6, is the best way to learn: correct on the fly :e


----------



## littlesandra88 (Feb 22, 2013)

@cpu82

Ok, I'll reinstall with 9.1 then. I guess I have trashed it too much to be used in production by now =)

"portmaster" looks really nice. I didn't knew about it, as I pretty much just read the FreeBSD Handbook, and I don't recall seeing it there.

From now on I will use that =)


----------



## cpm@ (Feb 23, 2013)

littlesandra88 said:
			
		

> @cpu82
> 
> Here you go =)
> 
> ...



It doesn't build, apparently because it's using "default" as a make target when it needs to be something else from a list it gives, in other words, fails to detect ARCH because is not defined in Makefile.

It is appropriate for this case, open a "new thread". I will be happy to help with whatever arises, even if it is necessary, starting from zero up the end


----------



## wblock@ (Feb 23, 2013)

I have a question that is on-topic for this thread: other than trying to identify which drive has failed, what difference does the order in the pool make?


----------



## littlesandra88 (Feb 25, 2013)

@wblock@

In this case it meant that the data got corrupt as the devices in each raidz2 got shuffled.


----------



## wblock@ (Feb 25, 2013)

But that is not supposed to happen.  The metadata on each drive identifies it.  I have intentionally disconnected and reconnected a RAIDZ1 without keeping track of the drive order.  It's possibly I put them back in the same way, I guess.  But again, that's supposed to be the point of the metadata: ZFS can figure out which drive is which, regardless of the port.


----------



## _martin (Feb 25, 2013)

littlesandra88 said:
			
		

> @wblock@
> 
> In this case it meant that the data got corrupt as the devices in each raidz2 got shuffled.



This is what I meant by my post and @wblock said now a post below: this should really not happened. That's why I asked if, even though different names, same serial numbers are in the pool. It is possible to move those disks around (even among Solaris/OpenSolaris and FreeBSD if the ZFS version is the same). 

I'm not that familiar with zdb command (frankly I used it only to check the ashift values), but when I looked on my system: 

`# zpool list`

```
portal  10.9T  8.29T  2.59T    76%  1.00x  ONLINE  -
rpool    183G  6.76G   176G     3%  1.00x  ONLINE  -
temple  3.97T  2.49T  1.48T    62%  1.00x  ONLINE  -
```

`# zdb -d portal`

```
zdb: can't open 'portal': Device not configured
```

I've the same error. It works on remaining pools. I can work with the pool portal without problem, it survived some reboots already.


----------

