# Nginx-FreeBSD slow download!



## shahzaib (Jan 31, 2016)

Hi,

We've recently shifted to FreeBSD 10.1-RELEASE due to its robust asynchronous performance for big storage based on .mp4 files. Here is the server specs :

2 x Intel Xeon X5690
96GB DDR3 Memory
12 x 3TB SATA Raid-10 (HBA LSI-9211)
ZFS FileSystem with 18TB usable space
2 x 1Gbps LACP (2Gbps Throughput)

Things are working quite well, no high I/O due to Big Ram cache and AIO performance but once network port started to go over 1Gbps, performance begins to lag, download speed started to stuck around 60-100Kbps on a 4Mbps connection (using wget) which works quite efficient under 800Mbps port (450kbps on 4Mbps). We first thought it could be network issue or LACP issue but doesn't looks like it is, we tested with download/upolading from this server on 2Gbps throughput. We also checked that if requests are in queue using command `netstat -Lan` but it was '0' .We also checked with Disk Busy rate using gstat(8) which was quite stable as well.

So it looks like either the sysctl(1) values need to tweak or NGINX configurations are not optimized.

Following is the link with all configs performed till now :

http://pastebin.com/EHUHV7SH
====================================================

Please iI need guidance to handle with this problem, iI am sure that some value needs to tweak.

Thanks in advance !!


----------



## Oko (Feb 1, 2016)

shahzaib said:


> 12 x 3TB SATA Raid-10 (HBA LSI-9211)
> ZFS FileSystem with 18TB usable space


Did you create ZFS pool on the top of the hardware RAID?


----------



## shahzaib (Feb 1, 2016)

Oko said:


> Did you create ZFS pool on the top of the hardware RAID?


No, we created using mirroring of each pair (2 x Sata) followed by rest of the drives i.e.

`zpool create storage mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror da8 da9 mirror da10 da11`


----------



## shahzaib (Feb 1, 2016)

I've also observed lots of retransmitted packets in `netstat -s` output. Here is the status :

http://prntscr.com/9xa6z2

When i*I* try to download file using *wget *on 4Mbps . This is what the behavior of download is:

- Download starts *-*-*> *speed goes up *-->* download stucks for 2-3 seconds (much like a pause, no progress in download) *-->* process resume and speed goes up *-->* stucks again *-->* resume etc.


----------



## kpa (Feb 1, 2016)

Post your `ifconfig` output.


----------



## shahzaib (Feb 1, 2016)

kpa said:


> Post your `ifconfig` output.


Here it is :

http://pastebin.com/TTvUxYh3


----------



## kpa (Feb 1, 2016)

I would first disable all TSO options on the interfaces and see if it makes a difference.


----------



## shahzaib (Feb 1, 2016)

kpa said:


> I would first disable all TSO options on the interfaces and see if it makes a difference.


Could you please let me know if there would be downtime for disabling TSO? If you can provide me command to do that, it'd be very kind.

Do iI need to disable TSO in all Active interfaces ? i.e Igb0, Igb2, lagg0.


----------



## SirDice (Feb 1, 2016)

It should be done on the physical interfaces (igb0,igb1) and shouldn't create downtime.

`ifconfig igb0 -tso`


----------



## shahzaib (Feb 1, 2016)

SirDice said:


> It should be done on the physical interfaces (igb0,igb1) and shouldn't create downtime.
> 
> `ifconfig igb0 -tso`


Thanks, II've disabled tso on two servers to see how it goes. Will keep you updated. Thanks !!


----------



## shahzaib (Feb 12, 2016)

Hi guys, one more thing worth mentioning is: we've encountered bit high %interrupt rate over 1.2+Gbps realtime port :

http://prntscr.com/a2d884

Supermicro server has built-in 4 x 1Gbps NIC ports. Following are there details, please let me know if these NIC are good enough ? 

http://prntscr.com/a2e0vc

igb0 & igb2 are emerged to spawn 2Gbps Lacp


----------



## kpa (Feb 12, 2016)

You could ask on the freebsd-net mailing list, many of the NIC driver developers are there including Jack Vogel who (as I understand) is the main contact at Intel for FreeBSD driver issues.


----------

