# Fatman vs ZFS Ã¢Â€Â“ Who will win



## Fatman (Dec 2, 2009)

*Fatman vs ZFS â€“ Who will win*

Hello Everyone,

I have decided to switch over my NAS to Freebsd from OpenSolaris now that ZFS has been ported over. This sudden change comes from the fact that I finally have some spare time and I noticed that I canâ€™t shutdown my open solaris system. It just seems to hang for hours. 

Hereâ€™s what Iâ€™m working with:


```
CHASSIS:  NORCO RPC-4020 
Mother;) Board: ASUS M2N-LR NF PRO3600 R 	
Power Supply:  CORSAIR CMPSU-1000HX 1000W RT 	
CPU: AMD A64 X2 5050E 2.6G AM2 RT 	
Memory: 2 x 2G PATR PVS24G6400LLKNB R 	
Hard Drives:  6x 1T WD 32M WD10EADS, 1 40g IDE Seagate
```

Iâ€™m currently in the process of downloading the 8.0-RELEASE-amd64-disc1.iso and will most likely start the install tomorrow (possibly tonight). Iâ€™ve opted to try installing zfs on the 40g IDE drive with the following wiki postings:

http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
http://wiki.freebsd.org/ZFSTuningGuide

If Iâ€™ve read correctly from the OpenSolaris Bible, you need to use the export command to ready a pool to be moved. ZFS records the hostname and hostid of the system that owns the pool so I need to boot back into OpenSolaris and execute:


```
#zfs export tank
#zfs status tank â€“ checks to make sure Iâ€™ve release ownership
```

Iâ€™m also looking for a little bit of advice. Should I bother installing zfs on the ide 40G drive? I could always do a regular base install, make the necessary tunings and kernel changes than load my old pools using:


```
#zpool import â€“ finds pools available for import
#zpool import [myoldpoolname]
```

Any input would be much appreciated. I'm sure more questions will follow once the install process has started. 

Sincerely,

Fatman


----------



## Voltar (Dec 2, 2009)

You don't have to export the pool on the old system, you can always force import it (the command you're looking for is zpool btw, not zfs on that one)
`# zpool import -f poolname/id`

As for installing on the 40gb drive, that is totally up to you. Personally I have an OS only drive that contains the base OS. 

There are a few really good threads about ZFS around these forums, might have a look at some of them.

Edit: You may not need to do all or any of the steps listing in the ZFS Tuning Guide. There has been a lot of improvements in the ZFS code since that was originally written.


----------



## Fatman (Dec 2, 2009)

Thanks Voltar. 

Ya the original plan was to use the 40gigs for the os and have the 21 drive bay used for zfs. I thought it would be interesting to try to boot from zfs to have the same features. 

So from what I gather, the following section does not apply to me:



> amd64
> 
> FreeBSD 7.2+ has improved kernel memory allocation strategy and no tuning may be necessary on systems with more than 2 GB of RAM.
> 
> ...



Hmm... I'm starting to think twice about zfs for the os drive. I really don't care if i lose the 40g hd.


----------



## Voltar (Dec 2, 2009)

Correct, since you have more than 2GB of RAM, I would see how things go for you without tuning. My fileserver runs 7.2-STABLE (up to date as of 17-Nov, and 7-STABLE has ZFS v13 like 8.0 does), and I've been able to remove all the optimizations I previously had to use with the older versions of ZFS (I have a thread or two about ZFS issues somewhere). 

As for the boot drive, I haven't really played around with ZFS boot/root. I know a few here are using it with great results though. However, I've been contemplating looking into going with a ZFS boot when I upgrade that box to 8.0 and add another four drives as ZFS is no longer considered experimental in 8.


----------



## Fatman (Dec 2, 2009)

Thanks again,

I'll do you the favor and give it a try which will hopefully help you make your decision. However, I will not try adding it to my raidz2 (6 x 1tb) for fear of losing the data that's on it. :\

When you mentioned there's some good zfs threads, are you recommending using those over the wiki?

This question is a little off topic but how much space should I attribute to swap. From what I've read the double rule no longer applies:

http://www.cyberciti.biz/tips/linux-swap-space.html

Is 4gigs what I should be using for my setup? 

Your input is very much appreciated considering that I'm a newb trying to find his way.

Fatman


----------



## Voltar (Dec 2, 2009)

The threads might have some helpful info in case you run into any problems, and how a specific user may have set something up/fixed something. I would go by the wiki starting out. 

As for swap, I've been using a straight 4GB in all my servers, and I've never seen it used either.


----------



## Blueprint (Dec 2, 2009)

I was running 8-RC3 with 4GB ram and root on zfs and everything was fine with no tuning until i decided to transfer about 250GB over to my storage pool via nfs. It panicked halfway through the transfer. After adding some tuning it went through fine and didn't panic since. I do run two pools though so im not sure if that requires more memory resource? 

I dont think much would have changed for Release. So i kept my tuning parameters. I have had no problems with Release with root on zfs thus far.


----------



## Fatman (Dec 2, 2009)

I noticed that you were unsuccessful using the following wiki:

http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

Can you please provide your system specs? Are you using 32 or 64bit system? 

Unless someone tells me otherwise, I will attempt to use this wiki since I feel someone should validate it. Hopefully any problems found can be resolved here and the wiki can be updated.

I definitely for see many questions coming since I am fairly new to FreeBSD but eager to learn.


----------



## Fatman (Dec 2, 2009)

Can someone shed some more light as to what's going on here?



> # Create GPT Disk
> 
> Fixit# gpart create -s gpt ad0
> 
> ...



The man pages really helped me but I also had to research GPT on wikipedia to understand the commands.

http://en.wikipedia.org/wiki/GUID_Partition_Table
http://www.freebsd.org/cgi/man.cgi?...ion=0&manpath=FreeBSD+8.0-RELEASE&format=html

Below creates GPT scheme:


```
Create GPT Disk
Fixit# gpart create -s gpt ad0
```



> "create"
> Create a new partitioning scheme on a provider given by provider.  The -s scheme option determines the scheme to use.The kernel needs to have support for a particular scheme before that scheme can be used to partition a disk.



The above is still unclear to me.

I understand the part below and thought to share what I grasped from this section and how I'm applying it. Please correct me if I've made a mistake.


```
# 
Creating boot partition:
Fixit# gpart add -b 34 -s 128 -t freebsd-boot ad0
```



> "add"
> Adds a new partition to the partitioning scheme given by geom. The partition begins on the logical block address given by the -b start option. Its size is expressed in logical block numbers and given by the -s size option. The type of the partition is given by the -t type option.



Here we create the boot partition which starts at Sector 34. 



> In 64-bit Windows operating systems, 16,384 bytes, or 32 sectors, are reserved for the GPT, leaving LBA 34 as the first usable sector on the disk



The command also defines the size [in sectors (blocks of 512)] and define the type option. So if I understand correctly 128 = 65536 bytes.

The next two set of command creates your swap and zfs partition. However, I don't quite understand what a label is so i'll need to read up on it.


```
Fixit# gpart add -b 162 -s 8388608 -t freebsd-swap -l swap0 ad0
Fixit# gpart add -b 8388770 -s 125829120 -t freebsd-zfs -l disk0 ad0
```



> -l label
> The label attached to the partition.  This option is only valid when used on partitioning schemes that support partition labels.



Since I have only 40 gigs for my drive, should I use the following:


```
Fixit# gpart add -b 34 -s 128 -t freebsd-boot ad0
Fixit# gpart add -b 162 -s 4294967296 -t freebsd-swap -l swap0 ad0
Fixit# gpart add -b 4294967458 -s 38654622720 -t freebsd-zfs -l disk0 ad0
```


```
42949672960 (40 * 1024 * 1024 * 1024)
4294967296 (4 * 1024 * 1024 * 1024)
82944 (162 * 512)

HD Space Available for ZFS = 38654622720 (35.99992275238037109375 GB)
```

I assume by the following that I should not use ZFS Swap Volume...



> 1. While a ZFS Swap Volume can be used instead of the freebsd-swap partition, crash dumps can't be created on the ZFS Swap Volume.



More info on creating swap volume


> Creating a swap partition on the ZFS Filesystem using a ZFS Volume:
> Fixit# zfs create -V 2gb zroot/swap
> Fixit# zfs set org.freebsd:swap=on zroot/swap
> Fixit# zfs set checksum=off zroot/swap



Thanks again to all those helping me out. Hopefully what I learn't will help others.

Sincerely,

Fatman


----------



## Fatman (Dec 2, 2009)

This is a great video that helped me understand more about ZFS. I'm sure that it has been posted all over however here it is once more:

http://www.youtube.com/watch?v=5-CR3o-Q2CU


----------



## Voltar (Dec 2, 2009)

Fatman said:
			
		

> I noticed that you were unsuccessful using the following wiki:
> 
> http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
> 
> Can you please provide your system specs? Are you using 32 or 64bit system?



Was that for me or Blueprint?




			
				Fatman said:
			
		

> Below creates GPT scheme:
> 
> 
> ```
> ...




Hopefully I understand what you want here, You want an explanation of what this does?

We invoke gpart, and tell it to create a new partition table, of the type GPT, on disk ad0.



> ...I don't quite understand what a label is so i'll need to read up on it...



Labels make it easier to keep track of partitions and swap out whole drives in ZFS pools. For example, you could label a drive 'vdev1/disk0' which could make it easier to keep track of. See glabel() for more info.

The rest looks good, and personally I would go with a FreeBSD type swap partition just in case I need a crash dump.


----------



## Fatman (Dec 2, 2009)

Great, thanks again Voltar.

The msg was directed to blueprint but it doesn't really matter. Do you think you could answer my question in the third posting about how much space i should allocate to Swap? 

Is it still double or has that deprecated? I read that I just need to use 4 gigs since that's what I have installed.

Do I also need to leave space towards the end of the drive or can I allocate the remainder as shown at the end of my previous post?

Are there any benefits from using Swap Volume over freebsd swap?

I feel really stupid asking these questions but I need to confirm that I'm doing things right the first time. Do not feel obligated to respond to them. I'm the type of guy that tries to know the ins/outs of everything before actually even applying it.

Thanks again,

Fatman


----------



## Voltar (Dec 2, 2009)

Fatman said:
			
		

> Do you think you could answer my question in the third posting about how much space i should allocate to Swap?
> 
> Is it still double or has that deprecated? I read that I just need to use 4 gigs since that's what I have installed.





			
				Voltar said:
			
		

> As for swap, I've been using a straight 4GB in all my servers, and I've never seen it used either.



It all depends on who you ask, I've never had an issue with 4GB as I said.




> Do I also need to leave space towards the end of the drive or can I allocate the remainder as shown at the end of my previous post?



If this is going to be your OS only drive, I would go ahead and use it all up.




> Are there any benefits from using Swap Volume over freebsd swap?



I read months ago something along the lines of... ZFS is memory hungry so if you have a situation where you need swap, and it is on a ZFS volume, that it could lead to a panic. I don't honestly know how much of this still applies, if any. 



> I feel really stupid asking these questions but I need to confirm that I'm doing things right the first time. Do not feel obligated to respond to them. I'm the type of guy that tries to know the ins/outs of everything before actually even applying it.



The only stupid question is the one you don't ask. I have to say though, my dev machine in the corner is looking like a prime target to try this with.


----------



## Fatman (Dec 2, 2009)

Almost ready to start the install but I want to get an answer to my last question before proceeding.


```
Install the Protected MBR (pmbr) and gptzfsboot loader
Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad0
```

The man page explains it but I'm having a hard time understanding. I also did some research on the bootstraping process and thought I'd share it: 

http://en.wikipedia.org/wiki/Bootstrapping_(computing)
http://elearning.algonquincollege.com/coursemat/pincka/dat2343/lectures.f03/16-LMC-Bootstrap.htm



> Who Loads the Loader? - If the Operating System provides the instructions necessary to load other programs, where do the instructions come from to load the Operating System? The traditional answer to this is that instructions to load the Operating System are contained in a ROM (Read Only Memory) which receives control on initial "power up". The instructions that perform this load of the Operating System (or of enough of the Operating System so that it can complete loading itself) is called a "bootstrap" program. Other possibilities (than ROM) exist, including a hardware facility to directly load values into memory locations through "toggle switches". The bootstrap program must be able to read data from a secondary storage device into memory (typically from a non-file-structured area of a disk) into memory and "jump" to the area of memory where this data was stored.





> bootcode
> Embed bootstrap code into the partitioning scheme's metadata on the geom (using -b bootcode) or write bootstrap code into a partition (using -p partcode and -i index).	Not all partitioning schemes have embedded bootstrap code, so the -b bootcode option is scheme-specific in nature.  For the GPT scheme, embedded bootstrap code is supported.  The bootstrap code is embedded in the protective MBR rather than the GPT.  The -b bootcode option specifies a file that contains the bootstrap code.  The contents and size of the file are determined by the partitioning scheme. For the MBR scheme, it's a 512 byte file of which the first 446 bytes are installed as bootstrap code.  The -p partcode option specifies a file that contains the bootstrap code intended to be written to a partition.  The partition is specified by the -i index option.  The size of the file must be smaller than the size of the partition.



/mnt2/boot/pmbr = file that contains my bootstrap code
/mnt2/boot/gptzfsbootfile = file that contains bootstrap code and that will be written to the partition

1 ad0 = index option

What does the one stand for here? Is it referring to the first sector of the first partition?

The rest of the points in section one of the ZFS wiki is pretty straight forward. Section two if pretty much the same but I was curious to know if it's possible to modify the script below to do a minimal install with just base + man (i think that's all i need). I normally get the port collection via portsnap so i don't see the need to install it.


```
Fixit# cd /dist/8.0-*
Fixit# export DESTDIR=/zroot
Fixit# for dir in base catpages dict doc games info lib32 manpages ports; \
          do (cd $dir ; ./install.sh) ; done
Fixit# cd src ; ./install.sh all
Fixit# cd ../kernels ; ./install.sh generic
Fixit# cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/

Can the script be changed for example to the following?

Fixit# for dir in base manpages; \
          do (cd $dir ; ./install.sh) ; done
```

Thanks,

Fatman


----------



## Voltar (Dec 3, 2009)

Fatman said:
			
		

> 1 ad0 = index option
> 
> What does the one stand for here? Is it referring to the first sector of the first partition?



The '-i 1' denotes the first partition in the GUID partition table, I believe (I could be very wrong, I just skimmed the man page again). The 'ad0' is the device. 




> The rest of the points in section one of the ZFS wiki is pretty straight forward. Section two if pretty much the same but I was curious to know if it's possible to modify the script below to do a minimal install with just base + man (i think that's all i need). I normally get the port collection via portsnap so i don't see the need to install it.



At the very least you need base and the GENERIC kernel. I just go with the bare-bones install and compile from source to get updated and any features that I need.


----------



## Fatman (Dec 3, 2009)

So from what I understand, my modified version should work since you copy over the kernel after the script executes.


```
Fixit# cd /dist/8.0-*
Fixit# export DESTDIR=/zroot
Fixit# for dir in base manpages ports; \
          do (cd $dir ; ./install.sh) ; done
Fixit# cd src ; ./install.sh all
Fixit# cd ../kernels ; ./install.sh generic
Fixit# cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/
```

DVD's burnt, I'm starting the install now. :e


----------



## Voltar (Dec 3, 2009)

Fatman said:
			
		

> So from what I understand, my modified version should work since you copy over the kernel after the script executes.
> 
> 
> ```
> ...



That should work, although personally I would go with portsnap for getting ports. 

Let me know how it goes.


----------



## Fatman (Dec 3, 2009)

Voltar,

I booted up the DVD and selected fixit. I was experiencing some weird issues and it took me a while to figure out that num lock was causing it. x(

I have a feeling that I've missed something here. When i type the first command:



> gpart create -s gpt ad0



I get "geom 'ad0': File exists". I'm getting this even after fdisking the partitions and using W to write then Q to exit. When I attempt the second command I get:

gpart: start '34': invalid argument
Is this normal?


----------



## Voltar (Dec 3, 2009)

Well are you certain that your first disk (or the one you want to install to) is 'ad0'? Try something like `# ls /dev | grep ad` (I'm sure there is a better way to do that) to see what you have.


----------



## Fatman (Dec 3, 2009)

The command displays ad0 and when I go into fdisk, the disk name shows ad0.

Off to a great start..  I think you're right though, i must be specifying the wrong drive. Let me take a look at the dmesg output.


----------



## Fatman (Dec 3, 2009)

Nope, my drive is showing up as ad0 in dmesg. 


```
ad0: 38204mb <samsung sp0411n tw100-11> at ata0-master UDMA100
```


----------



## Voltar (Dec 3, 2009)

Alrighty, I just popped an unused SCSI disk into my dev machine and booted the 8.0 USB stick image. Got the same error. Had to zero the old partition table with dd, then it works

`# dd if=/dev/zero of=/dev/ad0 bs=512 count=1`


----------



## Fatman (Dec 3, 2009)

That fixed it however I'm getting this error:

```
gpart: size 4294967296 : invalid argument
```

when executing:

```
Fixit# gpart add -b 162 -s 4294967296 -t freebsd-swap -l swap0 ad0
```


I thought the -s was followed by the swap size in bytes. I have for gigs so I came up with the following:


```
4294967296 (4 * 1024 * 1024 * 1024)
```

Am I to divide the total by 512 for block size? Does this make any sense?


----------



## Voltar (Dec 3, 2009)

The '-s' is for the size, in sectors. 



			
				http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot#line-1 said:
			
		

> Note:
> 
> 1.While a ZFS Swap Volume can be used instead of the freebsd-swap partition, crash dumps can't be created on the ZFS Swap Volume.
> 2. Sizes and offsets are specified in sectors (1 sector is typically 512 bytes).



So 8388608 sectors (8388608 * 512) = 4294967296 bytes (4 GB)


----------



## Fatman (Dec 3, 2009)

Thanks, that solved that problem but I hit another road block:


```
Fixit# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad0
```

outputs: "ad0 has bootcode". Is that ok?

Steps 1, points 6 and 7 went through fine.


----------



## Voltar (Dec 3, 2009)

Fatman said:
			
		

> Thanks, that solved that problem but I hit another road block:
> 
> 
> ```
> ...



I'm going to say yes, but I can tell you for sure in about 3 minutes when I reboot.


Edit: Works great. :h


----------



## Fatman (Dec 3, 2009)

So far so good, i have a quick question:

Should I be running my make install from the i386 folder. I'm currently running on 64bit platform.


```
Fixit# echo 'LOADER_ZFS_SUPPORT=YES' > /etc/src.conf
Fixit# mount -t devfs devfs /dev
Fixit# export DESTDIR=""
Fixit# cd /usr/src/sys/boot/
Fixit# make obj
Fixit# make depend
Fixit# make
Fixit# cd i386/loader
Fixit# make install
```


----------



## Voltar (Dec 3, 2009)

Fatman said:
			
		

> So far so good, i have a quick question:
> 
> Should I be running my make install from the i386 folder. I'm currently running on 64bit platform.



Yes, there is no amd64 specific bootloader that I'm aware of.


----------



## Fatman (Dec 3, 2009)

Thanks, the install went through smoothly. How ever i was not able to unmount /dev.


```
Fixit# umount /dev
cannot unmount '/dev': device busy
```

I kept going and now seeing the same message with:


```
Fixit# zfs unmount -a
cannot unmount '/zroot': device busy
```

Any ideas?


----------



## Voltar (Dec 3, 2009)

Make sure you're trying to unmount the devfs that is mounted in the chroot, and make sure you're not in /zroot while trying to unmount it (`# cd /` first). `# pwd` will tell you where you are.


----------



## Fatman (Dec 3, 2009)

Sorry Voltar,

I don't understand. I exited chroot and I still get the same message. I even made sure i'm in /.


----------



## Voltar (Dec 3, 2009)

Run `# mount`. Do you have something like ...


```
devfs on /zroot/dev (devfs, local, multilabel)
```

If so, try `# umount /zroot/dev`


----------



## Fatman (Dec 3, 2009)

Glad you got back to me, I was going to attempt to force it with the -f option. :e

mount shows me the following:

```
devfs on /dev (devfs, label multilabel)
zroot on /zroot (zfs, local)
```

I get the following message when attempting to umount /zroot/dev:

```
umount: /zroot/dev: not a file system root directory
```


----------



## Voltar (Dec 3, 2009)

Possibly stupid question, but did you exit the chroot environment?


----------



## Fatman (Dec 3, 2009)

Haha, there's no stupid question.. just idiots like me that can't follow clear instructions. I was sure that i exited but I must of not have. I exited all the way out and re-entered fix-it. I was then able to run umount /zroot.

zfs umount -a went through as expected now.

Thanks again Voltar...


----------



## Fatman (Dec 3, 2009)

I have a dumb question for you. I'm use to doing installs via the standard option that you can select after boot up of the FreeBSD OS DVD.

In point 5:


> Exit Fixit mode, and then sysinstall. Remove the FreeBSD install CD and the system will boot using the ZFS root.



I'm not sure what I need to sysinstall. Can I just reboot and run postsnap, freebsdupdate and start using my box? I'm a little confused.

Thanks again for all your help.


----------



## Voltar (Dec 3, 2009)

When you exit the FixIt environment it will put you back in sysinstall.


----------



## Fatman (Dec 3, 2009)

oh man, do i ever feel stupid now! :\


----------



## Voltar (Dec 3, 2009)

No worries, we all start somewhere.


----------



## Fatman (Dec 3, 2009)

Voltar, I haven't tried importing my old zpools but the following messages worries me from dmesg:

```
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 13
ZFS storage pool version 13
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ad0: 38204MB <SAMSUNG SP0411N TW100-11> at ata0-master UDMA100
ugen0.1: <nVidia> at usbus0
uhub0: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <nVidia> at usbus1
uhub1: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
acd0: DVDROM <HL-DT-STDVD-ROM GDR8164B/0L06> at ata0-slave UDMA33
ad4: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata2-master SATA300
ad6: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata3-master SATA300
ad8: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata4-master SATA300
ad10: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata5-master SATA300
GEOM: ad4: corrupt or invalid GPT detected.
GEOM: ad4: GPT rejected -- may not be recoverable.
ad12: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata6-master SATA300
GEOM: ad6: corrupt or invalid GPT detected.
GEOM: ad6: GPT rejected -- may not be recoverable.
GEOM: ad8: corrupt or invalid GPT detected.
GEOM: ad8: GPT rejected -- may not be recoverable.
GEOM: ad10: corrupt or invalid GPT detected.
GEOM: ad10: GPT rejected -- may not be recoverable.
uhub0: 10 ports with 10 removable, self powered
ad14: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata7-master SATA300
GEOM: ad12: corrupt or invalid GPT detected.
GEOM: ad12: GPT rejected -- may not be recoverable.
SMP: AP CPU #1 Launched!
GEOM: ad14: corrupt or invalid GPT detected.
GEOM: ad14: GPT rejected -- may not be recoverable.
```

A sides from that, everything booted up and I can actually shutdown my system! 

The import was unsuccessful. When I run 
	
	



```
zpool import
```
 to attempt to view any available zpools, it comes up blank. According the ZFS chapter in OpenSolaris, this should return the pools available. Or maybe, I've missed something. I'm going to re-read the chapter, do I need to define a pool and attach the drives?


```
mrnas# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
zroot  33.2G   730M  32.5G     2%  ONLINE  -
```


----------



## Voltar (Dec 3, 2009)

Did you partition the ZFS disks in OpenSolaris, or did you use the entire disk in the pool?

Try running `# zdb -l drive/partition` and see if you have labels on any of the drives.


----------



## Fatman (Dec 3, 2009)

I put all 6x 1TB drive in a Raidz2 zpool called zstorage. From there I created multiple sub-folders with my pictures, videos, work stuff. I'm pretty sure I have nothing to worry about right now since I tested formatting the 40gig IDE drive and reinstalling OpenSolaris which detected my zpool without any issues in the past. 

The only thing that I can think of is that I might of changed the positions of the sata connections on the motherboard. However I thought that was the great thing about zfs, the drives are hot swappable...

The command you asked me to run provides me the following output:

```
mrnas# zdb zstorage
zdb: can't open zstorage: No such file or directory
mrnas# zdb ad6
zdb: can't open ad6: No such file or directory
mrnas# zdb -l ad6
cannot open 'ad6': No such file or directory
mrnas# zdb -l zstorage
cannot open 'zstorage': No such file or directory
```

I found the following link however it does not look good:
http://forums.freebsd.org/showthread.php?t=2139

I can blow away the data if I really have too but I rather not. There's a lot of stuff on there that I didn't back up.


----------



## Voltar (Dec 3, 2009)

Try `# zdb -l /dev/ad6`


----------



## Fatman (Dec 3, 2009)

```
mrnas# zdb -l /dev/ad4
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


```
mrnas# zdb -l /dev/ad6
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


```
mrnas# zdb -l /dev/ad8
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


```
mrnas# zdb -l /dev/ad10
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


```
mrnas# zdb -l /dev/ad12
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


```
mrnas# zdb -l /dev/ad14
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


----------



## Voltar (Dec 3, 2009)

What's the output of `# ls /dev | grep ad4`


----------



## Fatman (Dec 3, 2009)

```
mrnas# ls /dev/ | grep ad4
ad4

mrnas# ls /dev/ | grep ad
ad0
ad0p1
ad0p2
ad0p3
ad10
ad12
ad14
ad4
ad6
ad8
mrnas#
```

This posting is very interesting:
http://groups.google.com/group/mail...ad/5b6ff6729353f6a9/fef81f4daae88fdc?lnk=raot



> On Tue, Nov 17, 2009 at 12:12 PM, Robert Noland <rnol...@freebsd.org> wrote:
> 
> This should be fixed in -CURRENT.  The issue seems to be that solaris claims that the GPT header is 512 bytes, where we expected it to always be 92 bytes.  The fixes are slated to merge to 7/8 stable within a few days, however they won't make it into 8.0.
> 
> robert.




```
mrnas# uname -a
FreeBSD mrnas.citywidenetworks.com 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009     [email]root@mason.cse.buffalo.edu[/email]:/usr/obj/usr/src/sys/GENERIC  amd64
```


----------



## Voltar (Dec 3, 2009)

I was about to post the same link, and happened to refresh to see that you already found it. (I was starting to run out of ideas)

Perhaps try updating to 8-STABLE and see if it works.


----------



## Fatman (Dec 3, 2009)

I remember someone telling me a long time ago that it's always better to re-install then to upgrade to a different branch/release. Do you recommend that I download the ISO for the stable branch and repeat the same steps that I took to complete this install?


----------



## Voltar (Dec 3, 2009)

I've never had issues with upgrading. Personally I would csup the 8-STABLE sources and compile/install per the handbook.


----------



## Fatman (Dec 3, 2009)

Looks like I don't have a choice!  I can't download the stable branch from the website. I'll work on this tomorrow and let you know of the results. 

Haha.. Just my luck, nothing ever runs smoothly and works the first time. I don't know why I complicate my life so much. All this effort because I want to be able to shutdown a server that I really never shutdown....

I'm calling it a night. Thanks for all your help, it's very much appreciated.

Fatman


----------



## jb_fvwm2 (Dec 3, 2009)

I'm guessing you need more devices in /dev for 
each drive (1sa, 1sb, etc).  I just skimmed the
posts but I could be wrong.  
The following .ko (one can load before "boot" though
at the loader prompt). in /boot/loader.conf should
create more devices if they are missing (myself and
another poster fixed problems that way)
geom_bsd.ko geom_label.ko geom_mbr.ko.
..........
Sorry that while posting I haven't a clue
whether that is 10 percent or 90 percent a
chance of helping.


----------



## Fatman (Dec 3, 2009)

Thanks Jb,

Could you please break it down it to simpler terms for a newb? I don't really understand what I need to do to fix the issue.


----------



## jb_fvwm2 (Dec 3, 2009)

*IF* it fixes the issue (I don't know enough about 
the current issue in your thread) you can (maybe)
in /boot/loader.conf

```
geom_mbr_load="YES"
geom_label_load="YES"
geom_bsd_load="YES"
```
and the devices will appear if they are missing.
(IF that is the cause of the current issue).
A shorter way, (I recall from another thread, I 
could be mistaken in the syntax)
...
(escape to loader prompt > boot)

```
load /boot/loader/geom_bsd.ko
(or, load geom_bsd (or, with .ko also)
(and whichever of the above three works,
the same for the other two).
boot
```
Although I suspect only two of the three are needed.
AND, it may not be the issue.  But it fixed things
here.


----------



## Fatman (Dec 3, 2009)

Thanks for the quick response. I don't mind trying it tonight however I'm concerned about applying something that I don't know much about.

I will do my due dilligence tonight and research what you've showed me before applying it. Is there a possibily that what you presented as a fix could screw my sata drives so that the data can no longer be read in OpenSolaris. 

I'm almost tempted tonight to re-install OpenSolaris and pull off the important data that I didn't backup.


----------



## Fatman (Dec 3, 2009)

Thanks for the advice Jb but unfortunetly I still have the same problem.


```
ad4: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata2-master SATA300
ad6: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata3-master SATA300
ad8: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata4-master SATA300
ad10: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata5-master SATA300
ad12: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata6-master SATA300
GEOM: ad4: corrupt or invalid GPT detected.
GEOM: ad4: GPT rejected -- may not be recoverable.
ad14: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata7-master SATA300
GEOM: ad6: corrupt or invalid GPT detected.
GEOM: ad6: GPT rejected -- may not be recoverable.
SMP: AP CPU #1 Launched!
GEOM: ad8: corrupt or invalid GPT detected.
GEOM: ad8: GPT rejected -- may not be recoverable.
GEOM: ad10: corrupt or invalid GPT detected.
GEOM: ad10: GPT rejected -- may not be recoverable.
GEOM: ad12: corrupt or invalid GPT detected.
GEOM: ad12: GPT rejected -- may not be recoverable.
GEOM: ad14: corrupt or invalid GPT detected.
GEOM: ad14: GPT rejected -- may not be recoverable.
Root mount waiting for: usbus1 usbus0
uhub0: 10 ports with 10 removable, self powered
Root mount waiting for: usbus1
Root mount waiting for: usbus1
Root mount waiting for: usbus1
uhub1: 10 ports with 10 removable, self powered
Trying to mount root from zfs:zroot
bge0: link state changed to UP

mrnas# zdb -l /dev/ad4
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```

I think I'm stuck upgrading to the stable branch or wait until the next release. 

I'll let you guys know what happens. Thanks again,

Fatman


----------



## Voltar (Dec 3, 2009)

I still think your issue is completely explained in the mailing list thread, you'll need to upgrade to get the patch to fix it.


----------



## Fatman (Dec 4, 2009)

Voltar,

I'm updating the stable version now using the info from:

http://www.freebsd.org/doc/handbook/makeworld.html

I've changed the standard-supfile to reflect the Stable branch and synching up with RELENG_8 src now.


```
#csup -L 2 /standard-supfile
#csup -L 2 /ports-supfile
```


```
#standard-supfile
*default host=cvsup11.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=RELENG_8
*default delete use-rel-suffix
*default compress
src-all

#ports-supfile
*default host=cvsup11.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=.
*default delete use-rel-suffix
*default compress
ports-all
```


```
refuse file: I'm sure there should be more but I don't have time to start searching.

src-games
doc/bn_*
doc/da_*
doc/de_*
doc/el_*
doc/es_*
doc/fr_*
doc/hu_*
doc/it_*
doc/ja_*
doc/mn_*
doc/nl_*
doc/no_*
doc/pl_*
doc/pt_*
doc/ru_*
doc/sr_*
doc/tr_*
doc/zh_*
```

This is going to take a while.. I'll let you know of the results afterwards.


----------



## Voltar (Dec 4, 2009)

You really only need the supfile for the OS sources, portsnap can be used to update the ports tree easily.


----------



## Fatman (Dec 4, 2009)

Dam, too late. Will I run into problems if I start using portsnap from now on instead of csup?

I'm also stuck. I synched my source, make buildworld, builtkernel, installed kernel and now booted into single user mode.

However, I don't think I should be running mount -a -t ufs. Am I to replace ufs with zfs?


```
# adjkerntz -i
# mount -a -t ufs
# mergemaster -p
# cd /usr/src
# make installworld
# mergemaster
# reboot
```


----------



## Voltar (Dec 4, 2009)

Portsnap may complain that it didn't create the ports tree, and you'll have to fetch and extract a new tree. Shouldn't be an issue. 

As for mounting, you might run `# mount` and see what is already mounted, your ZFS filesystems may already be there. If not, I'd try `# zfs mount -a`.


----------



## Fatman (Dec 4, 2009)

I figured another way around, the lazy way. I restarted and booted into the os as usual. I than used shutdown now which allowed me to than executed mergemaster -p. 

Okay, i'm done. I'll need to go back and re-run mergemaster because I left them all for later because I'm dead tired.


```
FreeBSD mrnas.citywidenetworks.com 8.0-STABLE FreeBSD 8.0-STABLE #0: Fri Dec  4 23:15:45 EST 2009     root@mrnas.citywidenetworks.com:/usr/obj/usr/src/sys/GENERIC  amd64
```

Well, it worked. 



> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
> to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
> ZFS filesystem version 13
> ZFS storage pool version 13
> ...





> mrnas# zpool import
> pool: zstorage
> id: 7729955510381236292
> state: ONLINE
> ...



Hmm... This doesn't look good but I seem to be able to view my files and copy them.



> mrnas# zpool import -f zstorage
> Unsupported share protocol: 1.
> Unsupported share protocol: 1.



Should I be concerned about losing my data? I noticed a couple of post online about it. 

Anyhow, Thanks again Voltar for all your help. I really appreciate it.


----------



## Voltar (Dec 4, 2009)

```
mrnas# zpool import -f zstorage
Unsupported share protocol: 1.
Unsupported share protocol: 1.
```

I haven't seen that before, and a quick search did not turn much up. 

Pure speculation here, but were you sharing the mounts directly via NFS/CIFS? That is the only thing that I could think of that may show an error like that. 

Sorry I can't be more help on that one.


----------



## Fatman (Dec 4, 2009)

Ya, I assumed it was related to some sort of previous permission that were set in OpenSolaris. It seems to be working fine so far and I'm able to copy files (mp3s) off the server without problems.


```
mrnas# zpool status -v zstorage
  pool: zstorage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zstorage    ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            ad4     ONLINE       0     0     0
            ad14    ONLINE       0     0     0
            ad12    ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            ad8     ONLINE       0     0     0
            ad6     ONLINE       0     0     0

errors: No known data errors
```


----------



## Fatman (Dec 5, 2009)

Dam,

I just noticed that I can't use freebsd-update with the stable release... :\ 


```
mrnas# freebsd-update fetch
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching public key from update4.FreeBSD.org... failed.
Fetching public key from update5.FreeBSD.org... failed.
Fetching public key from update2.FreeBSD.org... failed.
No mirrors remaining, giving up.
```

Is there a low volume bug and security mailing-list that I can sign up for with the 8.0 Stable release? I'm also planning on installing samba so that my winblows clients can access it. Do you recommend any postings or wiki's. 

I was thinking of installing it from ports and follow this link:
http://wiki.samba.org/index.php/Main_Page


----------



## Voltar (Dec 5, 2009)

You have to manually fetch the sources and recompile on STABLE, but it does hold advantages. You can customize your make flags in /etc/make.conf specific for your machine, which can yield performance increases; See make.conf(). Likewise, you can use /etc/src.conf to exclude parts of the OS that you don't need, see src.conf().  

Mailing list for freebsd-security: http://lists.freebsd.org/mailman/listinfo/freebsd-security

Edit: freebsd-annouce might be of interest to you also: http://lists.freebsd.org/mailman/listinfo/freebsd-announce

I use net/samba33 on my fileserver with great results. I had to tweak a few samba settings and sysctl's to get the most out of my Gigabit link, and I can pull 600-800+ Mbps to my workstation from my fileserver.


----------



## Fatman (Dec 5, 2009)

Wow 800mbps+... That's great. 

Thanks again for all the help and advice. I'll look into the customization of the make flags once samba is up and running. It will be interresting to see what's going to happen. I have 160 Gigs worth of junk (videos, mp3s, pictures, software) that I need to transfer over into my zpool from a laptop. 

What's the best way to track the transfer rate/speed?


----------



## Fatman (Dec 5, 2009)

Voltar,

Why am I seeing so much disk space available to me in my zpool if i'm running 6 x 1TB HD with Raidz2? Shouldn't I lose almost half for parity? I remember seeing 3.5T in OpenSolaris.


```
mrnas# zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
zroot     33.2G  3.39G  29.9G    10%  ONLINE  -
zstorage  5.44T   262G  5.18T     4%  ONLINE  -

mrnas# zpool status
  pool: zroot
 state: ONLINE
 scrub: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        zroot        ONLINE       0     0     0
          gpt/disk0  ONLINE       0     0     0

errors: No known data errors

  pool: zstorage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zstorage    ONLINE       0     0     0
          raidz2    ONLINE       0     0     0
            ad4     ONLINE       0     0     0
            ad14    ONLINE       0     0     0
            ad12    ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            ad8     ONLINE       0     0     0
            ad6     ONLINE       0     0     0

errors: No known data errors

mrnas# zdb -l /dev/ad4
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
```


----------



## Voltar (Dec 6, 2009)

I'm not sure how zpool displays sizes on (Open)Solaris, but on all my FreeBSD machines it shows the entire capacity of the pool, including parity. 

This is from a FreeBSD system with 12x500 GB drives (3 vdevs of 4 drives in RAIDZ).

```
amnesiac# zpool list
NAME      SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
storage  5.44T  4.34T  1.10T    79%  ONLINE  -
amnesiac# zpool status
  pool: storage
 state: ONLINE
 scrub: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        storage                  ONLINE       0     0     0
          raidz1                 ONLINE       0     0     0
            label/storage_01_1   ONLINE       0     0     0
            label/storage_01_2   ONLINE       0     0     0
            label/storage_01_3   ONLINE       0     0     0
            label/storage_01_4   ONLINE       0     0     0
          raidz1                 ONLINE       0     0     0
            label/storage_02_1   ONLINE       0     0     0
            label/storage_02_2   ONLINE       0     0     0
            label/storage_02_3   ONLINE       0     0     0
            label/storage_02_4   ONLINE       0     0     0
          raidz1                 ONLINE       0     0     0
            label/storage_03_1   ONLINE       0     0     0
            label/storage_03_2   ONLINE       0     0     0
            label/storage_03_3   ONLINE       0     0     0
            label/storage_03_4   ONLINE       0     0     0

errors: No known data errors
```

And from an Ubuntu system that I've yet to reinstall (4x250 GB drives in RAIDZ).

```
admin@htpc:~$ sudo zpool list
NAME      SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
storage   928G   913G  15.0G    98%  ONLINE  -
admin@htpc:~$ sudo zpool status
  pool: storage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0

errors: No known data errors
admin@htpc:~$ df -h | grep storage
storage/home          684G  683G  346M 100% /home
storage               346M   27K  346M   1% /storage
```

I only have ~4TB and ~690GB of space, respectively.


As for testing transfer speeds, there are so many different ways, ftp, smb, http, nc/pv, and I think there is something in the ports tree who's name slips my mind right now.


----------

