# (nginx 1.9) Enable reuseport then all load on 1 worker?



## meteor8488 (Feb 28, 2016)

Hi all,

I just upgrade nginx to 1.9.2 and also enabled reuseport for my website.


```
worker_processes  18;

...

listen  80 accept_filter=httpready reuseport;
```


But I found that all workloads are handled by the same worker. Even though this worker is pretty busy and almost 100% CPU usage. And if I remove reuseport and use accept_mutex , all my workers are working and each of them get about 10%-15% CPU usage.

Based on nginx document, it supposed to balance the work load on different workers.


_With the SO_REUSEPORT option enabled, there are multiple socket listeners for each IP address and port combination, one for each worker process. The kernel determines which available socket listener (and by implication, which worker) gets the connection.
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/_


And I check FB document it's said that freebsd is supporting SO_REUSEPORT.
So is this a bug or I need to enable SO_REUSEPORT for freebsd?

Thanks


----------



## Oko (Feb 29, 2016)

Nginx 1.9 branch is not a production version! That is developmental version. If you are using Nginx in production like me you should stick to 1.8 branch. If you chose to upgrade you are doing that on your own risk.


----------



## meteor8488 (Feb 29, 2016)

Oko said:


> Nginx 1.9 branch is not a production version! That is developmental version. If you are using Nginx in production like me you should stick to 1.8 branch. If you chose to upgrade you are doing that on your own risk.



I understand that... But 1.9 is the only version which can support http/2


----------



## drhowarddrfine (Feb 29, 2016)

meteor8488 You have 18 CPU cores? That value should be about the number of cores you are using.


----------



## Juha Nurmela (Feb 29, 2016)

I don't know if this is related, but I toyed with something like this in UDP just a while ago. Unicast packets go all to one socket, the first one to bind, the other receivers succeed in all system calls, but never receive anything. Only when the first receiver exits, the second one takes over. REUSE-options were for multicast*, maybe it hasn't changed.

I cannot see anything in in_pcblookup() that would suggest "load balancing", but that doesn't amount to much.

Juha

Doodling... There does not seem to be magic in REUSEPORT for tcp. 4 processes each creating their own sockets with the option, leaves 3 totally idle (but happy). Creating a socket and then forking off into 4 acceptors behaves as expected, all take their share.

* half-truth or worse


----------



## meteor8488 (Mar 1, 2016)

Juha Nurmela said:


> I don't know if this is related, but I toyed with something like this in UDP just a while ago. Unicast packets go all to one socket, the first one to bind, the other receivers succeed in all system calls, but never receive anything. Only when the first receiver exits, the second one takes over. REUSE-options were for multicast, maybe it hasn't changed.
> 
> I cannot see anything in in_pcblookup() that would suggest "load balancing", but that doesn't amount to much.
> 
> ...




https://forum.nginx.org/read.php?2,264913,264929#msg-264929

I asked the same question on nginx forum, it seems that NGINX doesn't support reuseport on freebsd.


----------



## Juha Nurmela (Mar 1, 2016)

It would still benefit from it. If you must restart the server for upgrade or something, you could launch the new one and then politely kill the old ones.

Juha


----------



## drhowarddrfine (Mar 1, 2016)

meteor8488 said:


> it seems that NGINX doesn't support reuseport on freebsd.


That is bizarre. Until just a couple of months ago, nginx was developed on FreeBSD and only moved to Linux through market pressure (not technical reasons).



> While SO_REUSEPORT socket option is available on FreeBSD, its
> behaviour is different from one nginx relies on: instead of
> balancing between all sockets as Linux and DragonFly do, it
> preserves historic behaviour for TCP and delivers all connections
> to one socket instead.



So is this a failing of FreeBSD? Or is there a technical advantage reason FreeBSD does not do the same as Linux and DragonFly?

EDIT: This is an area I am not familiar with but, this thread from just over two years ago, seems to indicate Linux and Dragonfly are shoehorning this into their systems in a most inappropriate way, though I may be misunderstanding what is being said.

_



			That is what multicast is for.
If you want the same data sent to all listeners, then
that is multicast behavior and you should be using
a multicast socket.
		
Click to expand...

_


----------



## Juha Nurmela (Mar 1, 2016)

Is state of the art really so fast, that 1 socket, N acceptors does not cut it anymore? That soinherit() looks really hairy to be worth it.

Juha


----------



## dmoran85 (Mar 1, 2016)

Juha Nurmela said:


> I don't know if this is related, but I toyed with something like this in UDP just a while ago. Unicast packets go all to one socket, the first one to bind, the other receivers succeed in all system calls, but never receive anything. Only when the first receiver exits, the second one takes over. REUSE-options were for multicast*, maybe it hasn't changed.



I've tried that on linux and there is a perfect RR load balancing.


----------



## Juha Nurmela (Mar 1, 2016)

Going on a tangent...

It's hard to stop listening cleanly. How to handle any remaining queued connections before exiting, but hide from new connection attempts. I could only think of shutdown(rendezvous_fd, ) or listen(rendezvous_fd, 0). Neither worked as one might hope, though both returned "No error".

shutdown with any SHUT_XX makes next accept return "Software caused connection abort", any or none pending connections has no effect.
listen 0 has no effect.

No practical value, as always,
Juha

bind to some other port, hiding, is EINVAL.
SO_RCVTIMEO {0,0} or [0,1000} has no effect (and an improper solution)


----------



## meteor8488 (Mar 2, 2016)

Based on some test, reuseport can reduce Latency and reduce server work load.

https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/
http://tengine.taobao.org/document/benchmark.html

Reuseport is the best feature for nginx 1.9. But it seems that we can't use it at this moment.

Is there any way we can do to enable it?


----------



## drhowarddrfine (Mar 2, 2016)

The problem, as I stated earlier, is using reuseport this way appears to be an inappropriate use of that method. Linux hacked the wrong thing to make this work that way instead of using what they should have. Asking FreeBSD to do the wrong thing is asking a bit much, though I know you just want to use this thing.

What you should do is ask this on the FreeBSD mailing list where the developers are and can properly answer this question.


----------



## Oko (Mar 2, 2016)

drhowarddrfine said:


> The problem, as I stated earlier, is using reuseport this way appears to be an inappropriate use of that method. Linux hacked the wrong thing to make this work that way instead of using what they should have. Asking FreeBSD to do the wrong thing is asking a bit much, though I know you just want to use this thing.


I am glad I read your post. Frankly my instinctive  reaction to somebody's request to do the wrong thing is to show middle finger. I hope that will be the reaction of FreeBSD developers.


----------

