# Interesting NFS issue



## frijsdijk (Oct 1, 2014)

I have a strange issue when using FreeBSD 10.0 (root on ZFS, physical server, 3 2TB disks in raidz2) as an NFSv4 server for (initialy) 4 Linux (14.04 Ubuntu, VMware VM's with lighttpd streaming mp4 video files) clients.

NFS settings in /etc/rc.conf:

```
nfsv4_server_enable="YES"
nfs_server_flags="-u -t -n 8"
nfs_server_enable="YES"
mountd_flags="-r -p 2345"
mountd_enable="yes"
rpcbind_enable="YES"
```

`shell> zfs set mountpoint=/home/mp4 tank/mp4`
`shell> zfs set sharenfs=yes tank/mp4`

Mount on Linux client:

```
1.1.1.1:/home/mp4 /var/www/mp4 nfs4 _netdev,proto=tcp,port=2049,rw,rsize=16384,wsize=16384,intr,soft,sec=sys,timeo=5 0 0
```

Works fine, nfs server can push out bursts of 300MB/sec (lagg0 with 4 1Gbit nics).

HOWEVER .. 

Let's say on the NFS server we have the file /home/mp4/test.mp4, which translates to /var/www/mp4/test.mp4 on the NFS client, and maps to let's say http://streaming.domain.com/test.mp4

1) `wget [url=http://streaming.domain.com/test.mp4]http://streaming.domain.com/test.mp4[/url]` -> 200 [OK]
2) `rm /home/mp4/test.mp4` (on NFS server)
3) `wget [url=http://streaming.domain.com/test.mp4]http://streaming.domain.com/test.mp4[/url]` -> 404 [ as expected ]
4) `touch /home/mp4/test.mp4` (on NFS server)
5) `wget [url=http://streaming.domain.com/test.mp4]http://streaming.domain.com/test.mp4[/url]` -> 404 [ what? ]

It does not recover. Not even by restarting lighttpd on the client. When I do a `ls -l /var/www/mp4` on the NFS client, that one will 200 - OK again. Each client then 'needs' that, to recover.

Tested with FreeBSD 10 as an NFS client, same issue (so it's not something Linux - FreeBSD)
Tested with other webservers (nginx, apache), same issues
"Basic" NFS mount options on client side, same issues
Perform file manipulation on NFS clients, in stead of on the NFS server, same issue (except for local)

Then switched to oldnfs_server_enable="YES", and all my problems are gone (but of course NFSv3 - which is fine, but .. )

... what is happening here?


----------



## SirDice (Oct 2, 2014)

It could be some weird caching interaction. I mean the NFS host has a filesystem cache, I think NFS itself has one too, the NFS client has a cache and lighttpd also uses cache. We can probably rule out lightttpd as it didn't improve when you restarted it. It sounds like some caching on the NFS client isn't picking up the changes, or at least not all of them. Try running tcpdump(8) and see if loading the file (after it has been touched) produces NFS traffic. If there's no traffic the issue is most likely at the NFS client.


----------



## frijsdijk (Oct 2, 2014)

If it's the client .. then it's an issue in Ubuntu as well as FreeBSD (I've tested FreeBSD as client as well).

I'll try to produce some tcpdumps.


----------

