# Data going to em1 is putting bandwidth on em0



## Dex (Dec 24, 2011)

I have a server with em0 (192.168.0.100) and em1 (192.168.0.101) and two Internet IPs, x.x.x.184 and x.x.x.185 respectively. Whenever I connect and start a transfer on the .185 IP, bandwidth is shown for em0. It's the same load on em0 for the .184 IP. How do I use the two Internet connections without having everything get routed into the first connection?


----------



## Dex (Dec 24, 2011)

I thought I should add that I followed this guide with the same results: http://www.daemonforums.org/showthread.php?t=4610

Some output:

```
# cat /etc/rc.local
setfib 0 route delete default
setfib 0 route add default 192.168.0.1
setfib 1 route delete default
setfib 1 route add default 192.168.0.1

ipfw -f flush
ipfw add allow ip from any to any via lo0
ipfw add setfib 1 ip from any to any via em0
ipfw add setfib 0 ip from any to any via em1
ipfw add allow ip from any to any

# netstat -ar
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS        35   187569    em0
localhost          link#5             UH          0   164286    lo0
192.168.0.0        link#1             U           0        0    em0
192.168.0.100      link#1             UHS         0        0    lo0
192.168.0.101      link#2             UHS         0        0    lo0

# cat /etc/rc.conf
ifconfig_em0="inet 192.168.0.100 netmask 255.255.255.0"
ifconfig_em1="inet 192.168.0.101 netmask 255.255.255.0"

# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:50:56:01:14:d9
        inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:50:56:01:11:93
        inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
ipfw0: flags=8800<SIMPLEX,MULTICAST> metric 0 mtu 65536
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
```


----------



## DutchDaemon (Dec 24, 2011)

Try using non-overlapping netmasks (or separate networks) on em0 and em1. With overlapping netmasks, traffic tends to stick to one route.

Try e.g. 192.168.0.0/24 and 192.168.1.0/24 networks, or 192.168.0.100/30 and 192.168.0.104/30 IP/netmasks (or even /32 netmasks if there are no attached networks behind them), to compartmentalize the NICs within their own networks.


----------



## Dex (Dec 24, 2011)

Well, the server is connected through a virtual (VM) network/router so I need to connect to 192.168.0.1 and the two 192.168.0.100 192.168.0.101 IPs are assigned to the Internet. That leaves me with using a different submask, right? What should I use instead of 255.255.255.0?

I misunderstood, trying it now.

It seems only /24 subnet works for the rc.conf entries.

And:


```
# setfib 2 route add default 192.168.1.0
route: writing to routing socket: Network is unreachable
add net default: gateway 192.168.1.0: Network is unreachable
```

It seems that it isn't possible for this way to have outgoing connections use both interfaces, being limited to one gateway/subnet. Some clarification would be appreciated.


----------



## Abriel (Dec 24, 2011)

I think the problem is with IPFW. I never managed to set up two routes with ipfw, so I switched to pf, and it works. In my case I have two Internet connections with different gateways and one LAN connection. It should work with one gateway too. 

I wanted some users to go by first table, rest should go by second table, and it works.


```
rc.local
/usr/sbin/setfib 0 route delete default
/usr/sbin/setfib 0 route add    default 192.168.20.1
/usr/sbin/setfib 1 route delete default
/usr/sbin/setfib 1 route add    default 10.0.0.1
```


```
pf.conf
table <white> {172.21.0.190, 172.21.0.235}
pass all
pass in from 172.21.0.0/24 to any rtable 1
pass in from <white> to any rtable 0
```


----------



## Dex (Dec 24, 2011)

I set the pf.conf and turned off ipfw. I did */etc/rc.d/pf onestart && /etc/rc.d/netif restart && /etc/rc.d/local restart*. No go. I'm sure my configuration is wrong as I don't really know what I'm doing, please help.


```
# cat /etc/rc.local
setfib 0 route delete default
setfib 0 route add default 192.168.0.1
setfib 1 route delete default
setfib 1 route add default 192.168.0.1

# cat /etc/rc.conf
firewall_type="open"
pf_rules="/etc/pf.conf"
nfs_client_enable="YES"
mysql_enable="YES"
ifconfig_em0="inet 192.168.0.100/24"
ifconfig_em1="inet 192.168.0.101/24"

# cat /etc/pf.conf
table <white> {192.168.0.100, 192.168.0.101}
pass all
pass in from 192.168.0.0/24 to any rtable 1
pass in from <white> to any rtable 0

# setfib 1 netstat -r
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS         1        0    em0
localhost          link#5             UH          0        0    lo0
192.168.0.0        link#1             U           0        0    em0


# setfib 2 netstat -r
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
localhost          link#5             UH          0        0    lo0
192.168.0.0        link#1             U           0        0    em0
```


----------



## Abriel (Dec 25, 2011)

With your pf.conf, your 192.168.0.100, 192.168.0.101 are going by first route table.

First, fibs start from 0, so ...
[CMD=]netstat -r[/CMD]
is for first fib
[CMD=]setfib 1 netsat -r[/CMD]
is for second fib

In pf.conf, I think you should have:


```
pass all
pass in from 192.168.0.100 to any rtable 1
pass in from 192.168.0.101 to any rtable 0
```


----------



## Dex (Dec 25, 2011)

It didn't seem to change anything


```
# netstat -r && setfib 1 netstat -r
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS        46    23682    em0
localhost          link#5             UH          0     5195    lo0
192.168.0.0        link#1             U           0        0    em0
192.168.0.100      link#1             UHS         0        0    lo0
192.168.0.101      link#2             UHS         0        0    lo0

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS         0        1    em0
localhost          link#5             UH          0        0    lo0
192.168.0.0        link#1             U           0        0    em0
```

Is ALTQ required?


----------



## Abriel (Dec 25, 2011)

If you paste it in console, what you got?

```
setfib 0 route delete default
setfib 0 route add default 192.168.0.1
setfib 1 route delete default
setfib 1 route add default 192.168.0.1
```
It looks like in all cases it is using em0. Second fib should have em1

Are em0 and em1 LAN or they are Internet connections?


What gateways have 





> two Internet IPs, x.x.x.184 and x.x.x.185 respectively



This gatways should be assigned to your fibs.

Sorry for my english, writting in train is a pain.


----------



## Dex (Dec 25, 2011)

```
# setfib 0 route delete default
delete net default
# setfib 0 route add default 192.168.0.1
add net default: gateway 192.168.0.1
# setfib 1 route delete default
delete net default
# setfib 1 route add default 192.168.0.1
add net default: gateway 192.168.0.1
# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS        44     4043    em0
localhost          link#5             UH          0 13279023    lo0
192.168.0.0        link#1             U           0        0    em0
192.168.0.100      link#1             UHS         0      488    lo0
192.168.0.101      link#2             UHS         0        0    lo0

# setfib 1 netstat -r
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS         0        0    em0
localhost          link#5             UH          0        0    lo0
192.168.0.0        link#1             U           0        0    em0
192.168.0.100      192.168.0.1        UGHS        0        2    em0
192.168.0.101      192.168.0.1        UGHS        0        0    em0
```

Both IPs (184, 185) are on the same gateway. em0 and em1 are virtual LAN


----------



## Abriel (Dec 25, 2011)

Ok, I tested it on my server, I set up one gateway for all my Internet connections, and yes, they used the same Netif. Here is http://forums.freebsd.org/showthread.php?t=21231. Second post.


----------



## Dex (Dec 26, 2011)

Thanks for the help, however after reading it and attempting to understand it, I can't figure out what I'm supposed to do for my situation.


----------



## Abriel (Dec 26, 2011)

Both em0 and em1 are on the same subnet. Routing will not separate the traffic.


----------

