# please help with NIC bonding.



## darkangel (Apr 19, 2022)

how do I create or how to do a NIC bonding between two NIC device that has their own separate IP address and from setting it up what are the other step to do? Like, do I need to add both IP addresses to my DNS record?

I have IP's assign to each NIC card and the server has a NGINX setup for a domain that I manage. 

re0 172.16.100.10
re1 172.16.100.11


Any help is greatly appreciated. Thank you and have a nice day.


----------



## SirDice (Apr 19, 2022)

32.10. Common Address Redundancy Protocol (CARP)


----------



## bakul (Apr 19, 2022)

Not clear if darkangel wants link aggregation or something else. Generally NIC bonding would be lagg(4) in freebsd. If both connections are point to point and don’t need to be routed and they only want increased bandwith and fail over, lagg would be simpler than CARP. But if they need to be routed, I wouldn’t call it NIC bonding.


----------



## SirDice (Apr 19, 2022)

bakul said:


> Generally NIC bonding would be lagg(4) in freebsd.


lagg(4) is more for bundling two or more interfaces on a single machine in order to increase the overall bandwidth. carp(4) is used for fail-over between two machines running a service. It depends on what you want to do. 

Assigning two IP addresses in the same range on more than one interface is asking for trouble (routing is going to be an issue).


----------



## darkangel (Apr 19, 2022)

Thank you for replying. what I want to achieve, if in case the re0 or re1 had  physical error, like if lan cable got disconnected my site wont go down and would stay up.  Thank you again for your comments.


----------



## sko (Apr 19, 2022)

darkangel said:


> Thank you for replying. what I want to achieve, if in case the re0 or re1 had  physical error, like if lan cable got disconnected my site wont go down and would stay up.  Thank you again for your comments.


for this you usually use lagg, maybe with lacp (switch has to support it and needs to be configured accordingly) or failover (only master port is used).
However, you don't need 2 IPs for that; so you can either omit one of them or assign both to that lagg-interface, which will not give you any advantages over a single IP.


----------



## darkangel (Apr 19, 2022)

sko said:


> for this you usually use lagg, maybe with lacp (switch has to support it and needs to be configured accordingly) or failover (only master port is used).
> However, you don't need 2 IPs for that; so you can either omit one of them or assign both to that lagg-interface, which will not give you any advantages over a single IP.


I see, I read about using lagg with failover.  So I can use same IP address for both re0 and re1?


----------



## Phishfry (Apr 19, 2022)

I am using round robin on 2 ports of a Chelsio 10G card. I found it quicker on rudimentary test.
/etc/rc.conf

```
ifconfig_cxl0="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"
ifconfig_cxl1="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"
cloned_interfaces="lagg0"
ifconfig_lagg0="up laggproto roundrobin laggport cxl0 laggport cxl1 192.168.1.1 netmask 255.255.255.0"
```

Router setup for LAGG.


----------



## Phishfry (Apr 19, 2022)

I have found LAGG very flexible. You can yank one cord and it keeps working. So you get teaming and redundancy.


----------



## sko (Apr 20, 2022)

darkangel said:


> I see, I read about using lagg with failover.  So I can use same IP address for both re0 and re1?


no, you apply the IP(s) to the lagg interface, not the physical interface. (well yes, in essence the ip is then used on both physical interfaces, but you configure them on the lagg)


----------



## Phishfry (Apr 20, 2022)

Per your example assuming you have jumbo frames enabled;
/etc/rc.conf

```
ifconfig_re0="up mtu 9000"
ifconfig_re1="up mtu 9000"
cloned_interfaces="lagg0"
ifconfig_lagg0="up laggproto lacp laggport re0 laggport re1 172.16.100.10 netmask 255.255.0.0"
```

I think its best to plan for a management interface too.

You really need to test those nic's under pressure first before using lagg.
Historically Realtek interfaces have had some troubles.


----------



## gpw928 (Apr 20, 2022)

One good way to observe the throughput in real time is to use "systat -ifstat" in conjunction with a traffic generator.

For my applications, using a large MTU with lagg had no benefit.  YMMV.


----------



## sko (Apr 20, 2022)

I'd be _very_ careful with jumbo frames and "consumer grade" network gear (and mikrotik). This usually leads to lots of cursing and ruined evenings.

The impact on real-world performance also isn't that big if you don't transfer huge amounts of continuous/stream data, so I'd just stick with the default MTU (1500) and call it a day. I mostly use jumbo frames at the core infrastructure (between switches and (some) routers), but not at the access layer. (virtualized routers and their hosts are a completely different can of worms in that regard...)


----------



## darkangel (Apr 21, 2022)

Phishfry said:


> Per your example assuming you have jumbo frames enabled;
> /etc/rc.conf
> 
> ```
> ...


I read about the problem with Realtek driver with FreeBSD. I wanted to replace it with intel but I can't right now, that is why I end up looking for NIC bonding and experiment.


----------



## Phishfry (Apr 21, 2022)

You are heading in the right direction. It can't hurt to do a benchmark before and after.
Check various lagg schemes.


Phishfry said:


> ifconfig_cxl0="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"


I removed all these flags for your example leaving only jumbo frames(which I thought were common)
These are my individual tunables. You might need to find the best ones for your NIC and desired usage.

With Realtek you want to keep on eye on interrupts during a load test.
`systat -vm 1` or `vmstat -i`


----------



## Phishfry (Apr 21, 2022)

gpw928 said:


> One good way to observe the throughput in real time is to use "systat -ifstat" in conjunction with a traffic generator.


Indeed and I do 3 file transfer tests. One Big files directory, one with large amount of small files and one mixed.
I think that real world copying is better than any pktgen.


----------

