# Server behind DSR load balancer



## einsibjani (Oct 7, 2021)

I'm trying to migrate an old system that uses relayd to redirect http connections to five backends. The backends are currently running Linux and using arptables and nat to do DSR.

Google led me to believe that the only thing I needed to do on FreeBSD was to add the virtual IP address as an alias on the loopback interface and it would work.
Unfortunately that isn't working in my vagrant test environment. Relayd rewrites the destination mac address, and tcpdump shows the packets hitting the backend server but it never reaches the http server (nothing in access logs). There's nothing in pf logs and I've tried disabling pf altogether on the backends.

Is there something I'm missing?


----------



## SirDice (Oct 7, 2021)

einsibjani said:


> I'm trying to migrate an old system that uses relayd to redirect http connections to five backends.


I would replace that with net/haproxy. It's HTTP traffic, which is quite easy to load-balance. You get a bunch of extra features too, like being able to off-load SSL to the HAProxy host. Or add filtering based on URL or paths. Or being able to dynamically remove hosts from a pool (useful when updating the backends). With HAProxy you can load-balance on layer 7, while relayd works on layer2. Relayd certainly has its uses, for non-HTTP traffic, but for HTTP there are better suited solutions.


----------



## einsibjani (Oct 7, 2021)

The application is a web traffic analyzer, so it needs the correct source address and it can't overwrite X-FORWARDED-FOR since the application also tracks proxy usage


----------



## SirDice (Oct 7, 2021)

einsibjani said:


> it can't overwrite X-FORWARDED-FOR since the application also tracks proxy usage


Then use a different label for it. You can configure mod_realip to look for that label. All the X- labels are custom, they're not exactly standardized, although X-Forwarded-For is often used for proxies. You could use X-Real-IP for the HAProxy. HAProxy doesn't actually _need_ this, it can base the load-balancing on different things so visitors keep getting directed to the same backend (if it's available). You can base this on source address or a cookie for example. You typically use X-Forwarded-For or X-Real-IP to get the 'real' address of the client in your logs but as the traffic is already proxied by the upstream proxy you don't have to set this on the HAProxy.


----------



## einsibjani (Oct 7, 2021)

SirDice said:


> Then use a different label for it. You can configure mod_realip to look for that label. All the X- labels are custom, they're not exactly standardized, although X-Forwarded-For is often used for proxies. You could use X-Real-IP for the HAProxy. HAProxy doesn't actually _need_ this, it can base the load-balancing on different things so visitors keep getting directed to the same backend (if it's available). You can base this on source address or a cookie for example. You typically use X-Forwarded-For or X-Real-IP to get the 'real' address of the client in your logs but as the traffic is already proxied by the upstream proxy you don't have to set this on the HAProxy.


I'll test it. I was hoping there was just a sysctl I was missing to get the DSR with loopback address working, but this could work instead


----------



## SirDice (Oct 7, 2021)

In haproxy.conf set something like this:

```
option forwardfor header X-My-Custom-Header
```
Then (assuming nginx here but you can do the same with Apache):

```
set_real_ip_from x.x.x.x; # IP address(es) or range of your HAProxy
    real_ip_header   X-My-Custom-Header;
```

That will leave the X-Forwarded-For header intact.


----------



## einsibjani (Oct 7, 2021)

I tested it with vagrant and it seems to work. The PHP app at least gets the overridden $_SERVER["REMOTE_ADDR"] and changing LogFormat in apache to use %a gives the real client IP in logs.
Took me a while to figure out you need the RemoteIPHeader function in the virtual host definition. It's not enough to have it in modules.d/mod_remoteip.conf


----------



## SirDice (Oct 7, 2021)

Make a status page in PHP for the backends. You can set that as the `check` URL in the backend definition in haproxy.conf. If that status page just returns 200 OK then that backend server will be active. If you let that status page return a 503 (or anything other than 200) HAProxy will disable that host in the pool. We use this so we can just do `fetch http://localhost/proxy_status.php?down` on the backend server to have it taken out of the pool. Then we can do management of that backend server (updates for example). When we're done we issue a `fetch http://localhost/proxy_status.php?up` and it'll be activated in the pool again.

That proxy_status.php is a fairly simple thing, it just sets it's status in a database table, a status.php reads the status from that table and returns 200 if it should be up and 503 if it should be down.

You can even get really fancy and have that status.php also return a 'weight' based on its current load. You can use that to have HAProxy dynamically send more requests to servers that have little to do and send less requests to servers that are busy.


----------

