# Software RAID 5 & FreeBSD



## Voltar (Dec 24, 2008)

I've been running FreeBSD for a while now, and finally want to venture into using RAID with FreeBSD. I want to add a RAID 5 array to my FreeBSD server, and can't exactly afford a hardware controller at the moment. After a bit of Google'ing I've found info on geom raid5, and it seems to be used in FreeNAS so I'm guessing it has a decent rep. 

Anyone have any personal experience with geom raid5? My array is going to be comprised of three 250 GB WD SATA II drives that I have laying around, the box is a AMD x2 3800+ with 4 GB RAM, mainly used as my development box, git/svn/cvs, ftp, file server, among other things. Data integrity is a concern as this will hold a lot of my work and backups.

Any thoughts or suggestions? Thanks in advance.


----------



## SirDice (Dec 24, 2008)

I'm using gvinum with RAID5 (4*500GB)..


```
root@molly:~#gvinum list
8 drives:
D r4                    State: up	/dev/ad7s1e	A: 0/475147 MB (0%)
D t4                    State: up	/dev/ad7s1d	A: 0/1535 MB (0%)
D r3                    State: up	/dev/ad6s1e	A: 0/475147 MB (0%)
D t3                    State: up	/dev/ad6s1d	A: 0/1535 MB (0%)
D r2                    State: up	/dev/ad5s1e	A: 0/475147 MB (0%)
D t2                    State: up	/dev/ad5s1d	A: 0/1535 MB (0%)
D r1                    State: up	/dev/ad4s1e	A: 0/475147 MB (0%)
D t1                    State: up	/dev/ad4s1d	A: 0/1535 MB (0%)

2 volumes:
V temp                  State: up	Plexes:       1	Size:       6142 MB
V raid5                 State: up	Plexes:       1	Size:       1392 GB

2 plexes:
P temp.p0             S State: up	Subdisks:     4	Size:       6142 MB
P raid5.p0           R5 State: up	Subdisks:     4	Size:       1392 GB

8 subdisks:
S temp.p0.s0            State: up	D: t1           Size:       1535 MB
S temp.p0.s1            State: up	D: t2           Size:       1535 MB
S temp.p0.s2            State: up	D: t3           Size:       1535 MB
S temp.p0.s3            State: up	D: t4           Size:       1535 MB
S raid5.p0.s0           State: up	D: r1           Size:        464 GB
S raid5.p0.s1           State: up	D: r2           Size:        464 GB
S raid5.p0.s2           State: up	D: r3           Size:        464 GB
S raid5.p0.s3           State: up	D: r4           Size:        464 GB
```

I used this article to set things up:
http://www.schmut.com/howto/freebsd-software-raid-howto

But you may also want to look into using ZFS. ZFS support was brand new when I set this up and I didn't feel comfortable. I might change it in the near future though.


----------



## vermaden (Dec 24, 2008)

Voltar said:
			
		

> I've been running FreeBSD for a while now, and finally want to venture into using RAID with FreeBSD. I want to add a RAID 5 array to my FreeBSD server, and can't exactly afford a hardware controller at the moment. After a bit of Google'ing I've found info on geom raid5, and it seems to be used in FreeNAS so I'm guessing it has a decent rep.
> 
> Anyone have any personal experience with geom raid5? My array is going to be comprised of three 250 GB WD SATA II drives that I have laying around, the box is a AMD x2 3800+ with 4 GB RAM, mainly used as my development box, git/svn/cvs, ftp, file server, among other things. Data integrity is a concern as this will hold a lot of my work and backups.
> 
> Any thoughts or suggestions? Thanks in advance.



FreeNAS uses graid5 patches, so you can addd these pathes to FreeBSD and use it as you would do with FreeNAS, I havent used it, but check lists.freebsd.org for opinions on that.

You can alsu try ZFS with RAID-Z (equivalent of RAID 5), especially with that amount of RAM (2GB for ZFS is sufficent, 4GB is even better).


----------



## Voltar (Dec 24, 2008)

SirDice said:
			
		

> I'm using gvinum with RAID5 (4*500GB)..
> 
> <snip>
> 
> ...



Thanks, I actually read that article, came up as the first or second result. Only thing that made me double take on it was that the writer mentioned gvinum to be unstable.



			
				vermaden said:
			
		

> FreeNAS uses graid5 patches, so you can addd these pathes to FreeBSD and use it as you would do with FreeNAS, I havent used it, but check lists.freebsd.org for opinions on that.
> 
> You can alsu try ZFS with RAID-Z (equivalent of RAID 5), especially with that amount of RAM (2GB for ZFS is sufficent, 4GB is even better).



Nice to know about the patches, I hadn't dug that far yet. I did come across ZFS a few times now, looks promising. After reading a bit more I saw it was only available for i386 originally (announcement wasn't too long ago it seems) but the wiki page on ZFS does show amd64 support (Forgot to mention that in the OP).


----------



## vermaden (Dec 24, 2008)

Voltar said:
			
		

> Nice to know about the patches, I hadn't dug that far yet. I did come across ZFS a few times now, looks promising. After reading a bit more I saw it was only available for i386 originally (announcement wasn't too long ago it seems) but the wiki page on ZFS does show amd64 support (Forgot to mention that in the OP).



amd64 is generally preferred for ZFS.


----------



## Voltar (Dec 25, 2008)

vermaden said:
			
		

> amd64 is generally preferred for ZFS.



That's good to hear.

My first attempt with ZFS didn't go too well, but gonna give it another shot. Thinking of migrating my /usr to the raid-z in the end now, that way everything important will have redundancy. The more I read about ZFS the more I like it, hopefully it'll scream like my RAID5/XFS does on Linux.


----------



## SirDice (Dec 25, 2008)

I suggest just using /usr/home/ in a raid 5. Pretty much everything else in /usr is easily installed if it gets nuked for some reason.

My server boots from a single IDE drive, you could use a mirror for that but i didn't bother with it. My swap is spread out on the 4 SATA disks, /tmp is a striped set (speed is more important then redundancy) and /storage raid5.


----------



## Voltar (Dec 25, 2008)

I know this is going to sound like a n00b question, but is there a way to have a software on / ? I've decided to use jails to replace a few boxes that I have so have a few extra hard drives to play around with. 

I was thinking of...

/ (RAID1)
swap (no RAID)
/tmp (RAID 0 or 10)
/usr (RAID 10)
/usr/home (RAID-Z w/ ZFS)
/var (RAID 10)


All the important stuff, jails, repos, data, etc will be stored in /usr/home. Any new suggestions?

Also, what kind of throughput does everyone see with software RAID and/or ZFS?


----------



## none (Dec 26, 2008)

how is raid5 from gvinum going ?

as it never reached the tree (AFAIK), I never got brave enough to run it ...

none


----------



## kbw (May 22, 2009)

Rather starting a new thread, I thought I'd continue this one to keep it all together.

I've read Grog's original docs on vinum and a lot of fairly recent stuff on gvinum including the referenced piece.

I'm using a 32-bit system (which rules out ZFS) with 8 x 1TB drives.  I can only manage 4.7G of useable formatted space, which I think is pretty poor.

Has anyone used gvinum with RAID5 configuration in a satisfactory way?  Any comments welcome.

Thanks.


----------



## kbw (May 22, 2009)

Correction, I get 4.7TB of RAID5 space formatted from 8 x 1TB physical space.

I'm using 7.2 (i386).


----------



## phoenix (May 22, 2009)

Voltar said:
			
		

> That's good to hear.
> 
> My first attempt with ZFS didn't go too well, but gonna give it another shot. Thinking of migrating my /usr to the raid-z in the end now, that way everything important will have redundancy. The more I read about ZFS the more I like it, hopefully it'll scream like my RAID5/XFS does on Linux.



For ZFS boxes, until booting ZFS is stable and usable by all, I'd recommend leaving / and /usr on non-ZFS storage.  That way, if you need to boot to single-user mode, you still have access to the full FreeBSD OS for troubleshooting.  Putting /usr on ZFS can cause problems if you need to boot to single-user mode and you can't import the pool.

Use gmirror for / and /usr, and put /usr/local, /home, /var, /tmp, /usr/src, /usr/ports, /usr/obj onto ZFS.


----------



## phoenix (May 22, 2009)

kbw said:
			
		

> Correction, I get 4.7TB of RAID5 space formatted from 8 x 1TB physical space.
> 
> I'm using 7.2 (i386).



Ouch!  You should have closer to 7 TB of usable space with a RAID 5 across 8 disks.  RAID 5 uses (X-1)*size space for X drives.  Either it's not using the whole disks, or something went wrong.


----------



## phoenix (May 22, 2009)

Voltar said:
			
		

> I know this is going to sound like a n00b question, but is there a way to have a software on / ? I've decided to use jails to replace a few boxes that I have so have a few extra hard drives to play around with.
> 
> I was thinking of...
> 
> ...



Way over complicated.  Simplify it:

* create a gmirror using 2 drives (or slices) for / and /usr
* create swap partitions on the same drives (or slices) as above
* put everything else into a ZFS pool using raidz or raidz2
* create ZFS filesystems for /usr/src, /usr/obj, /usr/ports, /usr/local, /home, /var, /tmp

Done.  Now you can also add compression to /usr/src and /usr/ports to save space.  

Note:  don't use more than 8 disks for a single raidz1/raidz2 vdev.  If you have more than 8 disks, split them up into smaller vdevs.  The beauty of pooled storage like ZFS is that you can add as many vdevs as you want, and it all be available to the pool.



> Also, what kind of throughput does everyone see with software RAID and/or ZFS?



Once you start using pooled storage (ZFS), you'll find it very hard to go back to figuring out how to partition disks for different uses.


----------



## phoenix (May 22, 2009)

kbw said:
			
		

> I'm using a 32-bit system (which rules out ZFS)



ZFS works best on 64-bit systems with lots of RAM.

However, ZFS works just fine on 32-bit systems, especially with 2 GB or more of RAM.  But it also works just fine on 32-bit systems with as little as 768 MB of RAM.   It just requires more fine-tuning.


----------



## kbw (May 26, 2009)

This is just a courtesy reply.  I'll have another post after the memory upgrade.

I'm now using ZFS on my physical 8 x 1TB disks on i386 1.5GB RAM.  And all looked well, I was seeing 7.2TB of formatted space.

On copying data from my backup (a ZFS amd64 host) over NFS, it crashed after copying 79GB.  The amd64 host is fine.

The copy was the only operation on the two boxes. I have property *set copies=2*.

I've ordered 3G of RAM (the maximum the old thing can take) and I'll resume when it arrives.


----------



## phoenix (May 26, 2009)

Voltar said:
			
		

> Also, what kind of throughput does everyone see with software RAID and/or ZFS?



pool comprised of 3 vdevs, each vdev is an 8-drive raidz2, using 500 GB SATA harddrives, 64-bit FreeBSD 7.2 w/8 GB RAM and 4 CPU cores.

iozone run as follows:
`# iozone -M -e -+u -T -t 128 -S 4096 -L 64 -r 4k -s 40g -i 0 -i 1 -i 2 -i 8 -+p 70 -C`
gives a sustained *write* throughput of 350 MBytes/sec (as shown by snmpd) which breaks down to ~15 MBytes/sec per drive (as shown by gstat).  (Didn't wait for it to finish to get the read speeds.)

Tweaking the iozone command a bit:
`# iozone -M -e -+u -T -t 128 -r 128k -s 4g -i 0 -i 1 -i 2 -i 8 -+p 70 -C`
gives just over 400 MBytes/sec sustained *write* throughput, or just under 20 MBytes/sec per drive.


pool comprised of 1 vdevs, which is a 3-drive raidz, using 120 GB SATA harddrives, 32-bit FreeBSD 7.1 w/2 GB RAM and 1 CPU core (HTT enabled).

iozone run as:
`# iozone -M -e -+u -T -t 32 -r 128k -s 40960 -i 0 -i 1 -i 2 -i 8 -+p 70 -C`
gives 18 MBytes/sec per drive of *write* throughput (as shown by gstat).  And iozone says it writes at 30 MBytes/sec, re-writes at 52 MBytes/sec, reads at 3.5 GBytes/sec, with a mixed workload of 63 MBytes/sec.

Overall, I'd have to say ZFS is good.


----------



## phoenix (May 26, 2009)

kbw said:
			
		

> This is just a courtesy reply.  I'll have another post after the memory upgrade.
> 
> I'm now using ZFS on my physical 8 x 1TB disks on i386 1.5GB RAM.  And all looked well, I was seeing 7.2TB of formatted space.
> 
> On copying data from my backup (a ZFS amd64 host) over NFS, it crashed after copying 79GB.  The amd64 host is fine.



On a 32-bit system, with only 1.5 GB of RAM, you will need to tune /boot/loader.conf to limit the size of vm.kmem_max and vfs.zfs.arc_max.  Otherwise, ZFS will try to use all the RAM it can grab, and crash the box with out-of-memory errors.

The values you use will depend on the system and the workload.  Here's what I use on my home system (3.0 GHz P4, 2 GB RAM):

```
vm.kmem_size_max="1G"
vfs.zfs.arc_max="256M"
```

That tells the kernel to use up to 1 GB for kernel memory, leaving at least 1 GB for user apps; and tells ZFS to only use 256 MB of kernel memory for the ARC.


----------



## kbw (Jun 5, 2009)

I just thought I'd check in.

I've tweaked the kernel memory limit as instructed, and now ZFS seems fine on a i386 with 1.5G of memory on 7.2 after being used for a couple of weeks.  It's as stable as my amd64.

Thanks for your help.


----------



## derwood (Jun 11, 2009)

I've been using GEOM_RAID5 for about 2 months now on my media server.
I've got 6 * 1TB drives in a RAID5 array.  Its pretty much been up for 2 months straight, except for kernel updates.  
GRAID5 is also used in FreeNAS.  Its much lighter weight than ZFS and as far as I can tell, seems to be just as reliable.  The big plus for me was that I didn't have to put 4 gigs on a box that's just a fileserver.


----------



## phoenix (Jun 11, 2009)

derwood said:
			
		

> Its much lighter weight than ZFS and as far as I can tell, seems to be just as reliable.  The big plus for me was that I didn't have to put 4 gigs on a box that's just a fileserver.



(Sigh, it'll be such a nice day when people stop resorting to FUD.)

You don't *need* 4 GB of RAM to run ZFS.  People have run it on 32-bit systems with as little as 768 MB of RAM.  People run it on laptops.

The ideal setup is a 64-bit system with 4 GB of RAM ... but that's not the minimum requirement.

I run it on my home media server which is just a 32-bit P4 @ 3 GHz with 2 GB of RAM.  Runs just fine.  You just need to do a bit of tuning of /boot/loader.conf.

Note:  the more RAM you can put into a fileserver, the better things will run, as all "free" RAM will be used as a filesystem cache, thus speeding things up immensely.


----------



## Voltar (Jun 28, 2009)

Well, I thought I would chip in here and say that I've had fantastic experiences thus far with ZFS, and now that I've upgraded my fileserver to FreeBSD 7.2 it looks like it doesn't require a lot of manual tuning as it did before (Per ZFSTuningGuide)

It also seems that I can't just add a single drive into my RAIDZ pool? Trying to figure that one out currently as I just got a few RMA'd drives back from manufacturers.


----------



## phoenix (Jun 29, 2009)

You can't expand a raidz vdev (ie turn a 3-drive raidz vdev into a 4-drive raidz vdev).

However, you can replace the individual drives in the raidz vdev with larger drives, to expand the total amount of storage space in the pool.  You have to replace 1 drive at a time, and let it finish the resilver for each drive in turn.  After all the drives in the raidz vdev are replaced, you drop to single-user mode and do a zpool export, and then zpool import.  After that, all the extra space will be available in the pool.

There are conflicting reports online on whether or not you can mix vdev types in a single pool (ie have a raidz1 vdev, a mirror vdev, a single-drive vdev, all in the same pool).  Some sites say you can't, other show that you can.  Haven't tested this yet, as all our systems have multiple, identical, raidz vdevs.


----------



## Voltar (Jun 29, 2009)

Thanks phoenix, I might just copy the data off and recreate the RAIDZ with the four drives since I don't plan on getting any more 250 GB drives.


----------



## dougs (Sep 12, 2012)

phoenix said:
			
		

> (Sigh, it'll be such a nice day when people stop resorting to FUD.)
> 
> You don't *need* 4 GB of RAM to run ZFS.  People have run it on 32-bit systems with as little as 768 MB of RAM.  People run it on laptops.
> 
> ...



I'm putting together a x386 file server containing 4GB RAM using FreeBSD 9.0-RELEASE and Samba 3.5.6. I've built a ZFS array using three 250GB SATA drives. The OS drive is formatted UFS and has all the file systems except for /data which resides on the ZFS array.

The issue is that after a day or two of running, the server will cough up and reboot. Sometimes it'll crash and dump data onto the console. Other times it'll simply reboot without warning. Usually it'll crash/reboot while I'm doing a huge file copying to the server which has been ruuning for a day or two.

I've read up on the ZFSTuningGuide on the FreeBSD wiki and adjusted the following files:

/etc/sysctl.conf:

```
aries# less /etc/sysctl.conf
# $FreeBSD: release/9.0.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.maxvnodes=400000
vfs.zfs.write_limit_override=268435456
```

/boot/loader.conf:

```
aries# less /boot/loader.conf
vfs.zfs.prefetch_disable="1"
vm.kmem_size="512M"
vm.kmem_size_max="1G"
vfs.zfs.arc_max="256M"
#vfs.zfs.vdev.cache.size="10M"
zfs_load="YES"
#vfs.zfs.txg.timeout="5"
```

I've installed ZFS on a AMD64 FreeBSD server with good success so I'm pretty sure it's related to using ZFS on a 32-bit machine. Phoenix claims above that he has good success running on 32-bit machines with 2GB. Exactly what kind of additional tuning do I need to perform in order to have a stable 32-bit ZFS/Samba file server? I'm expecting to have hundreds of files being opened/closed each day on the server running in a small business environment and I'd like to ensure this server runs uninterrupted for months on end and even for years.

Do I need to recompile the kernel instead of using the stock 9.0-RELEASE kernel and adjust the KVA_PAGES parameter to a larger value? Would this alleviate the crashes?

Or do I need to consider using gvinum raid5 capability instead?

~Doug


----------



## phoenix (Sep 13, 2012)

I'll post my loader.conf and sysctl.conf later today.


----------



## dougs (Sep 13, 2012)

So this morning I tried to copy over 17.2 GB worth of data consisting of 10.664 files and folders over and predictably, the server crashed. I took a picture of the crash dump on the monitor. I'm not sure if it's worth looking at but here it is.

https://plus.google.com/photos/1065...s/5787741631231769793?authkey=CKyrmZ2R2aSOqwE

Look forward to the config files, phoenix.


----------



## dougs (Sep 13, 2012)

One other thought- even though I mentioned that the ZFS array has three 250GB Seagate drives, the models aren't identical in terms of number of sectors. All three models have different model numbers as follows:

WD2500KS-00M      232.9GB
WD2500YD-01N      233.8GB
WD2500YS-01S      233.8GB

The last two of these contain identical sectors but the first doesn't. I had to use the -f parameter to force creation of the ZFS array due to this fact.

Would this be a very good reason why I'm experiencing the crashes? The array wasn't close to being 25% full at the time the copying job caused the array to stop functioning earlier today.

Here's the latest zfs-stats output prior to this morning's crash:


```
aries# zfs-stats -a

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Sep 13 11:57:30 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      i386
        Processor Architecture:                 i386

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 01:47:53 UTC 2012 root
11:57AM  up 21:21, 2 users, load averages: 3.99, 2.17, 0.93

------------------------------------------------------------------------

System Memory:

        0.56%   20.19   MiB Active,     5.58%   202.19  MiB Inact
        11.97%  433.74  MiB Wired,      0.00%   184.00  KiB Cache
        81.88%  2.90    GiB Free,       0.02%   568.00  KiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 90.09%  3.60    GiB
        Real Managed:                   98.22%  3.54    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   22.61%  925.91  MiB
        Logical Free:                   77.39%  3.10    GiB

Kernel Memory:                                  245.58  MiB
        Data:                           93.40%  229.37  MiB
        Text:                           6.60%   16.21   MiB

Kernel Memory Map:                              466.16  MiB
        Size:                           75.98%  354.21  MiB
        Free:                           24.02%  111.95  MiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                46.12k
        Recycle Misses:                         3.31k
        Mutex Misses:                           1
        Evict Skips:                            3.42k

ARC Size:                               91.46%  234.13  MiB
        Target Size: (Adaptive)         91.43%  234.06  MiB
        Min Size (Hard Limit):          12.50%  32.00   MiB
        Max Size (High Water):          8:1     256.00  MiB

ARC Size Breakdown:
        Recently Used Cache Size:       80.88%  189.36  MiB
        Frequently Used Cache Size:     19.12%  44.77   MiB

ARC Hash Breakdown:
        Elements Max:                           9.59k
        Elements Current:               100.00% 9.59k
        Collisions:                             6.94k
        Chain Max:                              4
        Chains:                                 658

------------------------------------------------------------------------

ARC Efficiency:                                 157.52k
        Cache Hit Ratio:                92.50%  145.71k
        Cache Miss Ratio:               7.50%   11.82k
        Actual Hit Ratio:               92.49%  145.70k

        Data Demand Efficiency:         99.73%  8.44k

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           39.62%  57.72k
          Most Frequently Used:         60.38%  87.97k
          Most Recently Used Ghost:     3.54%   5.15k
          Most Frequently Used Ghost:   0.35%   512

        CACHE HITS BY DATA TYPE:
          Demand Data:                  5.78%   8.42k
          Prefetch Data:                0.00%   0
          Demand Metadata:              94.20%  137.26k
          Prefetch Metadata:            0.02%   30

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.19%   23
          Prefetch Data:                0.00%   0
          Demand Metadata:              99.66%  11.78k
          Prefetch Metadata:            0.14%   17

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------


------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            536870912
        vm.kmem_size_scale                      3
        vm.kmem_size_min                        0
        vm.kmem_size_max                        1073741824
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            97386496
        vfs.zfs.mfu_ghost_metadata_lsize        11758080
        vfs.zfs.mfu_ghost_size                  109144576
        vfs.zfs.mfu_data_lsize                  0
        vfs.zfs.mfu_metadata_lsize              0
        vfs.zfs.mfu_size                        7155712
        vfs.zfs.mru_ghost_data_lsize            95168000
        vfs.zfs.mru_ghost_metadata_lsize        10958848
        vfs.zfs.mru_ghost_size                  106126848
        vfs.zfs.mru_data_lsize                  137367552
        vfs.zfs.mru_metadata_lsize              950272
        vfs.zfs.mru_size                        155123712
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       39896064
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  67108864
        vfs.zfs.arc_meta_used                   68315340
        vfs.zfs.arc_min                         33554432
        vfs.zfs.arc_max                         268435456
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            11608596480
        vfs.zfs.write_limit_max                 483691520
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                1
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------

aries#
```

See anything amiss in there?


----------



## kpa (Sep 13, 2012)

dougs said:
			
		

> /etc/sysctl.conf:
> 
> ```
> aries# less /etc/sysctl.conf
> ...



First off, comment out the stuff in /etc/sysctl.conf, you want to start with as few changes to settings as possible. Second, change the vm.kmem_size_max in /boot/loader.conf to 512M:


```
vm.kmem_size_max="512M"
```

If you need more kmem you must recompile the kernel with increased KVA_PAGES. I'm pretty sure the crashes you're now getting are from incorrect vm.kmem_size_max setting.


----------



## dougs (Sep 13, 2012)

kpa-

I commented out all of the contents of /etc/sysctl.conf and changed vm.kmem_size_max from 1G to 512M. I copied over the same data set (~17.2GB) without incident. Now we wait for tomorrow to see if it crashes.


----------



## phoenix (Sep 14, 2012)

/boot/loader.conf:

```
# ZFS settings
#vm.kmem_size="1G"
vm.kmem_size_max=1G
vm.kmem_size_scale=1
vfs.zfs.arc_max="768M"
vfs.zfs.arc_min="128M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zil_disable="0"
#vfs.zfs.vdev.max_pending="4"
#vfs.zfs.txg.timeout="7"
```

`$ zpool status`

```
pool: newpool
 state: ONLINE
  scan: resilvered 330G in 2h4m with 0 errors on Sat Aug 18 23:29:52 2012
config:

        NAME           STATE     READ WRITE CKSUM
        newpool        ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            gpt/disk3  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            gpt/disk4  ONLINE       0     0     0
            gpt/disk2  ONLINE       0     0     0

errors: No known data errors
```
`$ zpool list`
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
newpool  1.36T   640G   752G    45%  1.00x  ONLINE  -[/code]

`$ uname -a`

```
FreeBSD rogue.ashesofthe.net 9.0-STABLE FreeBSD 9.0-STABLE #0 r236880: Mon Jun 11 05:49:06 PDT 2012     root@rogue.ashesofthe.net:/usr/obj/usr/src-9/sys/ROGUE90  i386
```

Runs KDE4, Samba, NFS, basically 24/7.  Torrents downloaded via ktorrent.  Videos played via XBMC on an HTPC over SMB, or via Dragon Player on a laptop over NFS.  2 other laptops access shared folders via SMB.  Also has a bridge interface configured (one NIC connected to downstairs router that all clients use for Internet access, one NIC connected to upstairs router that only this box uses for Internet access).  The split network allows me to download torrents (which occasionally causes Shaw to throttle all traffic to that public IP) without affecting everyone else's Internet connection (via a separate public IP).

The only time the box is rebooted is when the USB ports act up, and I can no longer access the keyboard/mouse.  Something to do with the USB-based KVM (wife's laptop is also connected to it so she can use the big monitor).

RAM usage sits at about 95% all the time.  ARC is generally always full.  No longer have an L2ARC, as the USB stick I was using was slower than the pool.

Anything else you'd like to know about the setup?


----------



## dougs (Sep 14, 2012)

This morning I copied over the same large 17.2GB data set and it completed without crashing. I went ahead and deleted data from the array and recopied over. Still no problems. I'm almost ready to declare this issue solved.

However, I have this 4GB of RAM on this machine and I'm debating whether I should compile a new kernel with a higher KVA_PAGES value and thus be able to increase the vfs.kva_size_max value to a higher value like 1GB. This server will also run foswiki/apache on top of samba and that's it. Currently the server shows 2,993MB free memory. Any suggestions/thoughts?

kpa- thanks for the config information. What was the value you assigned to KVA_PAGES in your kernel? Your server has 2GB of RAM, correct? Is there a formula one should follow to properly size the KVA_PAGES option and also the configuration in both /boot/loader.conf and /etc/sysctl.conf (if applicable)?

Are there any feedback in regards to the mention of the fact that not all disks in the ZFS array are sized identically. See this:


```
ada1: <WDC WD2500YD-01NVB1 10.02E01> ATA-7 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 239372MB (490234752 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad5
ada2 at ata3 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD2500KS-00MJB0 02.01C03> ATA-7 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 238474MB (488395055 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad6
ada3 at ata3 bus 0 scbus2 target 1 lun 0
ada3: <WDC WD2500YS-01SHB0 20.06C03> ATA-7 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 239372MB (490234752 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad7
```

As you can see /dev/ada2 has slight less sectors than /dev/ada1 and /dev/ada3. Will I run into trouble eventually as more and more data is fed to the ZFS array and approaching the physical hard disk limits?

~Doug


----------



## kpa (Sep 14, 2012)

I no longer have an i386 system with ZFS but based on what is written in the tuning guide http://wiki.freebsd.org/ZFSTuningGuide#i386, I would say go with KVA_PAGES set to 512 and see if you can use a full 1GB for vm.kmem_size_max. Phoenix can maybe comment on this more


----------

