# ZFS pool perf problem/seeing TCP ZeroWindow all of a sudden



## aaronZZ (Feb 5, 2012)

Hello,

Have been running ZFS as a media server for more than a year. It has worked well and havenâ€™t had any problems, until about a week ago. All of a sudden, 1 of 2 ZFS pools started performing very poor, both across the network and for transfers within the machine. The pool â€œtank2tbâ€ basically went to what looked like kilobyte transfer speeds, while tests to â€œvaultâ€ remain good. 

I was on FreeBSD 7.2, but recently upgraded in an attempt to resolve this so now Iâ€™m on:

`uname -a`

```
FreeBSD filefs.local.com 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:46:30 UTC 2012    
root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
```


Looking at transfers from network machines to tank2tb I can see that it starts great, then after a burst it starts seeing the TCP window size decrease all of a sudden. Then it hits zero and I see:


```
66331	9.258478	192.168.0.102	192.168.0.197	TCP	60	[TCP ZeroWindow] microsoft-ds > 52415 [ACK] Seq=91601 Ack=56973523 Win=0 Len=0
```

It stays in zerowindow for 10's of seconds then another burst of traffic gets through...rinse...repeat. 

Reads do roughly the same thing, but just donâ€™t zerowindow. They have spikes of good perf, followed by multiple seconds of no activityâ€¦spikeâ€¦pause.

And Iâ€™ll point out again, in case it wasnâ€™t clear, that this is only occurring with the pool tank2tb. Same test from vault show no problems for read or write. There were no (intentional) changes made to the server. In fact I had not logged onto it for months when this happened. There recently was a power outage that knocked it offline, but I canâ€™t directly correlate the lost functionality to that day. 

I would appreciate any help tracking this down. Let me know what test/output might help you and Iâ€™ll get it.

Thanks,

Aaron


Some various outputs to start with. 

[CMD=""]zpool status[/CMD]


```
pool: tank2tb
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank2tb     ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    ada5    ONLINE       0     0     0
	    ada6    ONLINE       0     0     0
	    ada3    ONLINE       0     0     0

errors: No known data errors

  pool: vault
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	vault       ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    ada0    ONLINE       0     0     0
	    ada1    ONLINE       0     0     0
	    ada2    ONLINE       0     0     0

errors: No known data errors
```
`zpool get â€œallâ€ tank2tb`

```
NAME     PROPERTY       VALUE       SOURCE
tank2tb  size           5.44T       -
tank2tb  capacity       92%         -
tank2tb  altroot        -           default
tank2tb  health         ONLINE      -
tank2tb  guid           15262784380037294764  default
tank2tb  version        28          default
tank2tb  bootfs         -           default
tank2tb  delegation     on          default
tank2tb  autoreplace    off         default
tank2tb  cachefile      -           default
tank2tb  failmode       wait        default
tank2tb  listsnapshots  off         default
tank2tb  autoexpand     off         default
tank2tb  dedupditto     0           default
tank2tb  dedupratio     1.00x       -
tank2tb  free           395G        -
tank2tb  allocated      5.05T       -
tank2tb  readonly       off         -
```

[CMD=""]zpool get â€œallâ€ vault[/CMD]

```
NAME   PROPERTY       VALUE       SOURCE
vault  size           2.72T       -
vault  capacity       36%         -
vault  altroot        -           default
vault  health         ONLINE      -
vault  guid           1719042198761293107  default
vault  version        28          default
vault  bootfs         -           default
vault  delegation     on          default
vault  autoreplace    off         default
vault  cachefile      -           default
vault  failmode       wait        default
vault  listsnapshots  off         default
vault  autoexpand     off         default
vault  dedupditto     0           default
vault  dedupratio     1.00x       -
vault  free           1.73T       -
vault  allocated      1016G       -
vault  readonly       off         -
```


----------



## gkontos (Feb 5, 2012)

Your pool looks almost full. This might affect performance.


```
tank2tb  capacity       92%
```


----------



## aaronZZ (Feb 6, 2012)

I thought about that and have been moving data off of it, but the transfer speed is so slow it can't really move any quantity that matters. The one thing that makes me feel that is not the problem is that read speed is also bad for this drive. I can understand if it was just write, but why would read be bad because it's full. But I am still trying to pull some large quantity off of it to see if it helps.

Thanks.


----------



## phoenix (Feb 6, 2012)

Check the output of gstat(8) and *zpool iostat* to see if one of the drives is working harder than the other (a sign it's dying):
`# gstat -I 1000000`
`# zpool iostat -v 1`

Are there are messages on the console or in /var/log/messages when things are slow?


----------



## aaronZZ (Feb 6, 2012)

For gstat, on write mostly ada5 and ada6 were getting activity. ada3 pretty much stays idle. During reads, pretty same but a little more spread onto ada3. I'm not familiar with gstat but the %busy column was basically going from 0 to 180% on 5/6. 3 never was above a few percent. 


```
dT: 1.000s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
   10     24     24   3071  779.5      0      0    0.0  187.3  ada5
    1     24     24   2688  779.4      0      0    0.0  187.3  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  cd0
    0      0      0      0    0.0      0      0    0.0    0.0  ada4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ada4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ada4s1d
    0      0      0      0    0.0      0      0    0.0    0.0  ada4s1e
    0      0      0      0    0.0      0      0    0.0    0.0  ada4s1f
```

Looking at some posts on gstat:


> The reason gstat shows >100% busy is that there are some outstanding
> requests. (the 2 in the left hand column).



Writes to vault don't show any L(q) build up and %busy always stays below ~70%.

Does the high gstat L(q) count and %busy 5/6 imply the drives are dying? Nothing similar is occuring on ada3. Or is there further digging to do?

Thanks for helping me get to the bottom of this. And thanks for introducing some new debugging tools. 

Aaron


iostat output below, both read and write:

This is a large read from tank2tb, *iostat -v 1* output.


```
capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      3      0   390K      0
  raidz1    5.05T   395G      3      0   390K      0
    ada5        -      -      2      0   192K      0
    ada6        -      -      3      0   195K      0
    ada3        -      -      8      0   767K      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0      0      0      0
  raidz1    5.05T   395G      0      0      0      0
    ada5        -      -      0      0      0      0
    ada6        -      -      0      0      0      0
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G     15      0  2.00M      0
  raidz1    5.05T   395G     15      0  2.00M      0
    ada5        -      -      7      0   959K      0
    ada6        -      -      7      0   831K      0
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0      0      0      0
  raidz1    5.05T   395G      0      0      0      0
    ada5        -      -      0      0      0      0
    ada6        -      -      0      0      0      0
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G    369      0  45.8M      0
  raidz1    5.05T   395G    369      0  45.8M      0
    ada5        -      -    145      0  14.0M      0
    ada6        -      -    163      0  17.1M      0
    ada3        -      -    139      0  14.3M      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      7      0  1023K      0
  raidz1    5.05T   395G      7      0  1023K      0
    ada5        -      -      5      0   511K      0
    ada6        -      -      5      0   511K      0
    ada3        -      -      6      0   575K      0
----------  -----  -----  -----  -----  -----  -----
```




This is a large write to tank2tb, *iostat -v 1* output.


```
capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0     29      0  3.75M
  raidz1    5.05T   395G      0     29      0  3.75M
    ada5        -      -      0     17      0  2.00M
    ada6        -      -      0     16      0  1.87M
    ada3        -      -      0     57      0  6.65M
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      6    311  7.49K  10.8M
  raidz1    5.05T   395G      6    311  7.49K  10.8M
    ada5        -      -      3    125  3.50K  5.28M
    ada6        -      -      1    122   1022  5.40M
    ada3        -      -      3     81  3.00K   647K
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0      0      0      0
  raidz1    5.05T   395G      0      0      0      0
    ada5        -      -      0      0      0      0
    ada6        -      -      0      0      0      0
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0      0      0      0
  raidz1    5.05T   395G      0      0      0      0
    ada5        -      -      0      3      0  4.00K
    ada6        -      -      0      3      0  4.00K
    ada3        -      -      0      3      0  4.00K
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0     25      0  3.24M
  raidz1    5.05T   395G      0     25      0  3.24M
    ada5        -      -      0     15      0  1.73M
    ada6        -      -      0     14      0  1.62M
    ada3        -      -      0    174      0  21.1M
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0     51      0  6.46M
  raidz1    5.05T   395G      0     51      0  6.46M
    ada5        -      -      0     25      0  3.23M
    ada6        -      -      0     25      0  3.23M
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0      7      0  1023K
  raidz1    5.05T   395G      0      7      0  1023K
    ada5        -      -      0      3      0   511K
    ada6        -      -      0      3      0   511K
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0      0      0      0
  raidz1    5.05T   395G      0      0      0      0
    ada5        -      -      0      0      0      0
    ada6        -      -      0      0      0      0
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2tb     5.05T   395G      0     31      0  4.00M
  raidz1    5.05T   395G      0     31      0  4.00M
    ada5        -      -      0     15      0  2.00M
    ada6        -      -      0     15      0  2.00M
    ada3        -      -      0      0      0      0
----------  -----  -----  -----  -----  -----  -----
```


----------



## aaronZZ (Feb 20, 2012)

Tried replacing one of the drives with a new drive. Currently resilvering at ~24.6K/s. 2 days and only 1.26G has resilvered (0.07% done), out of 5.05T. Although there are 4 other SATA drives working perfectly in the computer, I'm beginning to think this is more of a motherboard problem. Is there any way to further pin this on the motherboard without just swapping everything into a new computer? There are not any other free SATA slot on this MB to try moving a drive to. Thoughts? Next steps?


----------

