# NFS slow/strange performance



## magmos_019 (Apr 30, 2013)

Hi!
FreeBSD 9.1-RC2 fileserver, referenced by F1, with ZFS. 2 NICs.
2 workstations are connected via 2 separate network cards on F1. Cross over cables. No more external connections from F1 (but the iDrac). Workstation nr 2 we call W2. I am using NFS (not ZFS sharenfs) to share a ZFS file system. Client is using Linux Mint v11. When I rsync files from F1 to W2 it goes very slow, 1.9 MB/s. Running `zpool iostat -1` it looks like no activity for about 10-15 seconds and then some activity for a couple of seconds. If I SSH into F1 and press Enter `zpool iostat -1` shows activity. To speed up the copy procedure I better press Enter constantly when SSHÂ´d into F1. Speed will be 36 MB/s. How come? What can be wrong set?

Best regards, Magnus

Enter not pressed

```
# zpool iostat 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     13.3T  8.41T      0      0      0      0
--"--
10-15 times the same output
--"--
storage     13.3T  8.41T    241      0  30.0M      0
storage     13.3T  8.41T     10      0  1.37M      0
storage     13.3T  8.41T    278      0  34.4M      0
storage     13.3T  8.41T      0     95      0   201K
storage     13.3T  8.41T      0      0      0      0
storage     13.3T  8.41T      0      0      0      0
```

Enter pressed

```
storage     13.3T  8.41T    537      0  66.5M      0
storage     13.3T  8.41T    326      0  40.1M      0
storage     13.3T  8.41T    101      0  12.6M      0
storage     13.3T  8.41T     94    105  11.6M   253K
storage     13.3T  8.41T    187      0  23.1M      0
storage     13.3T  8.41T    279      0  34.4M      0
storage     13.3T  8.41T    246      0  30.6M      0
storage     13.3T  8.41T    368      0  45.6M      0
storage     13.3T  8.41T    451    101  55.7M   234K
storage     13.3T  8.41T     57      0  7.24M      0
storage     13.3T  8.41T    449      0  55.5M      0
storage     13.3T  8.41T     19      0  2.50M      0
storage     13.3T  8.41T    185      0  23.1M      0
```

INFO F1:

`nfsstat`

```
# nfsstat
Client Info:
Rpc Counts: Every field shows 0
Rpc Info: Every field shows 0
Cache Info: Every field shows 0
Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
  3844693      7191     36174       740  23505240  17171578         0      4305
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
     1335       216       351       457      1168         0     28956    255733
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0      7510         4         2     64504
Server Ret-Failed 0
Server Faults 0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0  44937671
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
 17171578  17171578         0
```
`ifconfig`

```
# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:ae:52:88:f7:65
        inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
        inet6 fe80::d6ae:52ff:fe88:f765%bce0 prefixlen 64 scopeid 0x2 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:ae:52:88:f7:66
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::d6ae:52ff:fe88:f766%bce1 prefixlen 64 scopeid 0x3 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
```
ZPOOL status: everything is online and no known data errors.
`zpool list` on F1

```
# zpool list
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
storage  21.8T  13.3T  8.41T    61%  1.00x  ONLINE  -
```
`zpool get all` on F1

```
# NAME            PROPERTY              VALUE                  SOURCE
storage/shared  type                  filesystem             -
storage/shared  creation              Sun Oct  7  1:29 2012  -
storage/shared  used                  11.0T                  -
storage/shared  available             6.68T                  -
storage/shared  referenced            11.0T                  -
storage/shared  compressratio         1.00x                  -
storage/shared  mounted               yes                    -
storage/shared  quota                 none                   default
storage/shared  reservation           none                   default
storage/shared  recordsize            128K                   default
storage/shared  mountpoint            /storage/shared        default
storage/shared  sharenfs              off                    local
storage/shared  checksum              on                     default
storage/shared  compression           off                    default
storage/shared  atime                 on                     default
storage/shared  devices               on                     default
storage/shared  exec                  on                     default
storage/shared  setuid                on                     default
storage/shared  readonly              off                    default
storage/shared  jailed                off                    default
storage/shared  snapdir               hidden                 default
storage/shared  aclmode               discard                default
storage/shared  aclinherit            restricted             default
storage/shared  canmount              on                     default
storage/shared  xattr                 off                    temporary
storage/shared  copies                1                      default
storage/shared  version               5                      -
storage/shared  utf8only              off                    -
storage/shared  normalization         none                   -
storage/shared  casesensitivity       sensitive              -
storage/shared  vscan                 off                    default
storage/shared  nbmand                off                    default
storage/shared  sharesmb              off                    default
storage/shared  refquota              none                   default
storage/shared  refreservation        none                   default
storage/shared  primarycache          all                    default
storage/shared  secondarycache        all                    default
storage/shared  usedbysnapshots       0                      -
storage/shared  usedbydataset         11.0T                  -
storage/shared  usedbychildren        0                      -
storage/shared  usedbyrefreservation  0                      -
storage/shared  logbias               latency                default
storage/shared  dedup                 off                    default
storage/shared  mlslabel                                     -
storage/shared  sync                  standard               default
storage/shared  refcompressratio      1.00x                  -
storage/shared  written               11.0T                  -
```
exports

```
# cat /etc/exports 
/storage/shared 192.168.1.2
/storage/shared 192.168.2.2
```
rc.conf

```
root@storage:/root # cat /etc/rc.conf 
hostname="storage"
ifconfig_bce0="inet 192.168.2.1 netmask 255.255.255.0"
ifconfig_bce1="inet 192.168.1.1 netmask 255.255.255.0"
dumpdev="AUTO"
zfs_enable="YES"
nfs_server_enable="YES"
rpcbind_enable="YES"
mountd_flags="-r"
nfs_flags="-t -u -n 32"
```
`rpcinfo`

```
# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  rpcbind
    100000    3   tcp    111  rpcbind
    100000    2   tcp    111  rpcbind
    100000    4   udp    111  rpcbind
    100000    3   udp    111  rpcbind
    100000    2   udp    111  rpcbind
    100000    4 local    111  rpcbind
    100000    3 local    111  rpcbind
    100000    2 local    111  rpcbind
    100005    1   udp    686  mountd
    100005    3   udp    686  mountd
    100005    1   tcp    686  mountd
    100005    3   tcp    686  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
```
`showmount`

```
# showmount -e
Exports list on localhost:
/storage/shared                    192.168.1.2 192.168.2.2
```


INFO W2:

`ifconfig`

```
# ifconfig
eth1      Link encap:Ethernet  HWaddr 68:05:ca:0c:ff:46  
          inet addr:192.168.2.2  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::6a05:caff:fe0c:ff46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:356782238 errors:0 dropped:0 overruns:0 frame:0
          TX packets:191159483 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:534889953476 (534.8 GB)  TX bytes:13761998824 (13.7 GB)
          Interrupt:48 Memory:dfec0000-dfee0000
```
fstab

```
# Connection to F1 NFS
192.168.2.1:/storage/shared /media/disken nfs timeo=14,intr
```
`rpcinfo` W2:

```
procardis2 ~ # rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  57079  status
    100024    1   tcp  48076  status
    100021    1   udp  57635  nlockmgr
    100021    3   udp  57635  nlockmgr
    100021    4   udp  57635  nlockmgr
    100021    1   tcp  58359  nlockmgr
    100021    3   tcp  58359  nlockmgr
    100021    4   tcp  58359  nlockmgr
```


----------

