# mirror (ZFS, gmirror) on USB sticks



## raVen (May 31, 2014)

Hello. I am trying to build a bootable ZFS mirror on two USB-sticks. The sticks are nothing special:
	
	



```
May 31 07:47:28 vmstor kernel: da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
May 31 07:47:28 vmstor kernel: da0: <UFD 3.0 Silicon-Power16G PMAP> Removable Direct Access SCSI-6 device
May 31 07:47:28 vmstor kernel: da0: Serial Number P1312160070B3C53D05D7171
May 31 07:47:28 vmstor kernel: da0: 40.000MB/s transfers
May 31 07:47:28 vmstor kernel: da0: 15120MB (30965760 512 byte sectors: 255H 63S/T 1927C)
May 31 07:47:28 vmstor kernel: da0: quirks=0x2<NO_6_BYTE>
```
Scrub single stick is fine:
	
	



```
scan: scrub in progress since Sat May 31 08:36:45 2014
        461M scanned out of 2,87G at 25,6M/s, 0h1m to go
        0 repaired, 15,71% done
```
But I got the drama, when I attach second stick:
	
	



```
scan: resilver in progress since Sat May 31 07:49:55 2014
        242M scanned out of 2,87G at 122K/s, 6h17m to go
        242M resilvered, 8,25% done
```
Both sticks are new. I tested it both with dd: single speed is about 33 Mbytes/s read and 25 Mbytes/s write. Simultaneously to both sticks read/write, read/read, write/write by dd() is lower, but still acceptable: about 17-20 Mbytes/s. My system is FreeBSD 10.0-STABLE #0 r266463: Tue May 20 18:24:03 UTC 2014. Kernel configuration is pretty simple:
	
	



```
include GENERIC
ident vmstor
nooptions INET6
nooptions AHC_REG_PRETTY_PRINT
nooptions AHD_REG_PRETTY_PRINT
options CONSPEED=19200
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=512
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPDIVERT
options DUMMYNET
options SHMMAXPGS=65536
options SEMMNI=128
options SEMMNS=32000
options SEMUME=40
options SEMMNU=120
options SEMOPM=250
options SEMMSL=250
options SHMMNI=4096
options SHMSEG=4096
options MAXDSIZ="(1024*1027*1024)"
options MAXSSIZ="(1024*1024*1024)"
options DFLDSIZ="(1024*1024*1024)"
options SHMMAX=2147483647
options SHMALL=2097152
```
Physcal hardware is desktop mainboards with Intel 82801G (ICH7) USB controller on first, and Intel PCH USB 2.0 controller on second computer. Also I bought VIA VT6202 USB 2.0 PCI controller to split sticks to different controllers on mainboard, but results are the same.

I also tried gmirror() with these sticks. It's rebuilding with high speed like 20-30 Mbytes/s, but dump()/restore() speed of my root filesystem to mirror of sticks (sure, after rebuild is finished) is same 100-200 KBytes per second.

Is task to mirror data on USB sticks with FreeBSD possible? What did I do wrong?


----------



## wblock@ (May 31, 2014)

I would suggest asking this on the freebsd-usb mailing list.  Adding the second controller should eliminate hardware bottlenecks, but the problem might be in the FreeBSD USB system, or even on the memory sticks themselves (background writes or wear leveling).


----------



## raVen (Jul 2, 2014)

Solution. The issue was crappy USB sticks, because of that:


```
[~@vmstor/15:13:33]
raven$ sudo dd if=/dev/zero of=/dev/da7 bs=512
^C1248+0 records in
1247+0 records out
638464 bytes transferred in 31.285647 secs (20408 bytes/sec)
[~@vmstor/15:14:33]
raven$ sudo dd if=/dev/zero of=/dev/da7 bs=1024
^C1356+0 records in
1355+0 records out
1387520 bytes transferred in 31.484652 secs (44070 bytes/sec)
[~@vmstor/15:15:48]
raven$ sudo dd if=/dev/zero of=/dev/da7 bs=2048
^C1256+0 records in
1255+0 records out
2570240 bytes transferred in 23.480575 secs (109462 bytes/sec)
```
Then I visited some webpage with USB sticks statistics, bought new USB sticks with good write speed at little block size and the issue was gone.

But I got other issue, repeats at any USB stick, including crappy, which I tried in first post. I create zvol on my zpool of usual HDDs, connected to mainboard controller, trying to make reboot() or shutdown() computer and result is sad. Computer is totaly hang, only reset button on the case is working. If I don't use USB sticks for booting, or remove zvol, then the system is sucessfully shutting down.

Where is my mistake for this issue? I still would like to boot from mirror of USB sticks, but my time to create this storage server is going to over 

Update 2014/06/03: Solution.


----------

