# Networking questions



## Tomonf613 (Dec 20, 2019)

Hi, I am a FreeBSD beginner.
I'm trying to build a FreeBSD router with 4 NICs: igb0,igb1,igb2 and igb3, and igb0 is set as a gateway to global network (ng0).
My network configurations is here:
            ------------
       |        |
       |        |igb1------------client1 (internet access ok)
www----igb0| fBSD router|
      (ng0)|         |igb2------------client2 (internet access failed)
       |        |
            ------------



I wish to access global network, 13.6.0.0/24 netmask 255.255.255.0, via igb1 and igb2 (I want to set the 2 NiCs in a same network).
Now I can access global network through igb1,but cannot igb2.
here is my rc.conf  (around routing):

```
defaultrouter="10.10.10.10"  #igb0
gateway_enable="YES"

static_route="136-1 136-3"
route_136-1="-net 13.6.0.0/24 13.6.0.1"
route_136-2="-net 13.6.0.0/24 13.6.0.2"

ifconfig_igb1="13.6.0.1 netmask 255.255.255.0"
ifconfig_igb3="13.6.0.3 netmask 255.255.255.0"
```


----------



## SirDice (Dec 20, 2019)

You cannot have multiple routes, routing the same thing. Well, technically you can but you're going to need a dynamic routing protocol like OSPF or BGP to make that work. Learn the basic first before diving into those. 

This is not going to work:

```
route_136-1="-net 13.6.0.0/24 13.6.0.1"
route_136-2="-net 13.6.0.0/24 13.6.0.2"

ifconfig_igb1="13.6.0.1 netmask 255.255.255.0"
ifconfig_igb3="13.6.0.3 netmask 255.255.255.0"
```

A) You have multiple interfaces in the same network (that's bad).
B) You have defined the 13.6.0.0/24 route twice (that's not going to work).
C) The routes are pointing to itself, that's pointless. These routes are already implied (directly connected network). 

This isn't really a FreeBSD issue, you need to learn more about TCP/IP and routing. 



Tomonf613 said:


> (I want to set the 2 NiCs in a same network).


No, you most likely don't. If you want to have some sort of fail-over or load-balancing use carp(4) or lagg(4). 

(thread split off)


----------



## Tomonf613 (Dec 20, 2019)

I' ll learn more. Thank you!


----------



## SirDice (Dec 20, 2019)

I was looking for something online that explains things but most of them only seem to cover the basics and don't explain routing very well.

The absolute best book to get is "TCP/IP Illustrated". All three volumes are a treasure trove for all things TCP/IP but for you only Volume 1 would already be enough. Volume 2 is more about learning to use the socket library in C and Volume 3 covers things like HTTP, NNTP and UDP. 

https://www.amazon.com/TCP-Illustrated-Protocols-Addison-Wesley-Professional/dp/0321336313


----------



## gpw928 (Dec 20, 2019)

All the W. Richard Stevens TCP/IP books are excellent.  For currency, check the edition. If price matters, there are plenty of used copied to be had (and well worth getting):

ABE
https://www.abebooks.com/servlet/Se...s=30&kn=TCP&recentlyadded=all&sortby=17&sts=t


----------



## gpw928 (Dec 20, 2019)

Oops, finger problems, add BetterWorld to that list...


----------



## Tomonf613 (Dec 22, 2019)

Nice your recommendings, thank you.
Now I set these NICs bridging.


----------



## Tomonf613 (Dec 22, 2019)

Now a problem happened, that bridging or igb3 maybe work failed.

At first, I can access www via bridge0 (igb2).
But after a few hours, connecting www via bridge0 or igb2 failed, and connecting via igb1 only goes well.

Lan cables are ok, and bridge0 and igb2 are recognized by ifconfig.
I tried rebooting and commanding 'ifconfig bridge0 (or igb2) up', those do not solve the problem.

Please tell me what arise this problem, or how can I fix it?

Here is my rc.conf around network:
```
defaultrouter="666.444.2.33 "
gateway_enable="YES"

static_route="136-1"
route_136-1="-net 13.6.0.0/24 13.6.0.1"

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb1 addm igb3 up"
ifconfig_igb1="UP"
ifconfig_igb3="UP"
ifconfig_igb1="13.6.0.1 netmask 255.255.255.0"
#ifconfig_igb3="13.6.0.3 netmask 255.255.255.0"
```

ifconfig result:
```
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:30:18:0d:4f:24
    hwaddr 00:30:18:0d:4f:24
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:30:18:0d:4f:25
    hwaddr 00:30:18:0d:4f:25
    inet 13.6.0.1 netmask 0xffffff00 broadcast 13.6.0.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:30:18:0d:4f:27
    hwaddr 00:30:18:0d:4f:27
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:30:18:0d:4f:26
    hwaddr 00:30:18:0d:4f:26
    inet 192.168.11.4 netmask 0xffffff00 broadcast 192.168.11.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:01:40:a0:d1:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
    member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 2 priority 128 path cost 2000000
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1448
    inet 666.444.2.33 --> 444.444.22.33 netmask 0xffffffff          #<-- dammy global address
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

And result of netstat -nr:
'''
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            444.444.22.33     UGS         ng0
13.6.0.0/24        link#2             U          igb1
13.6.0.1           link#2             UHS         lo0
666.444.2.33        link#8             UHS         lo0
127.0.0.1          link#5             UH          lo0
444.444.22.33     link#8             UH          ng0
192.168.11.0/24    link#3             U          igb3
192.168.11.4       link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#5                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#5                        U           lo0
fe80::1%lo0                       link#5                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
'''

With this rc.conf, error message appears "/etc/rc.conf: route_136-1=-net 13.6.0.0/24 13.6.0.1: not found".
Please tell me about this message, too.


----------



## Phishfry (Dec 22, 2019)

Tomonf613 said:


> ifconfig_bridge0="addm igb1 addm igb3 up"


I dunno if this is a typo but you have addm twice and no entry for igb2.
The way you are doing this is wrong.
My personal opinion is that bridge's are only to be used when needed.
A routed network is preferred. A bridge is like a slow funnel.

I will post the settings for my APU2 Router to show what it takes:
/etc/rc.conf
Note igb0 is connected to an upstream router. igb1 and igb2 are LAN ports.
`ifconfig_igb0="DHCP"
ifconfig_igb1="inet 192.168.101.1 netmask 255.255.255.0"
ifconfig_igb2="inet 192.168.102.1 netmask 255.255.255.0"
sshd_enable="YES"
dnsmasq_enable="YES"
gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
defaultrouter="192.168.1.1"`

Note I use dns/dnsmasq for assigning client IP leases and the pf firewall for NAT

/etc/pf.conf
`ext_if="igb0"
set skip on lo
nat on $ext_if inet from ! ($ext_if) to any -> ($ext_if)`

/usr/local/etc/dnsmasq.conf
`server=8.8.8.8
server=8.8.4.4
dhcp-range=set:igb1,192.168.101.10,192.168.101.25,255.255.255.0,24h
dhcp-range=set:igb2,192.168.102.10,192.168.102.25,255.255.255.0,24h
dhcp-option=igb1,option:router,192.168.101.1
dhcp-option=igb2,option:router,192.168.102.1`


----------



## Tomonf613 (Dec 23, 2019)

Thank you for your replies.
Finally I found that when setting rc.conf connecting www via bridge will failed, but setting in command line goes well.

For my example, bridging igb1 and igb3

Setting rc.conf seems not working around bridging (or igb3).

```
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb1 addm igb3 up"
ifconfig_igb1="up"
ifconfig_igb3="up"
ifconfig_bridge0="addm igb1 addm igb3 up"
ifconfig_bridge0="inet 13.6.0.1 netmask 255.255.255.0"
```

However setting below commands goes well.

```
ifconfig bridge0 create
ifconfig igb1 up
ifconfig igb3 up
ifconfig bridge0 addm igb1 addm igb3 up
ifconfig bridge0 inet 13.6.0.1 netmask 255.255.255.0
```
In these two pattern, "ifconfig" command shows same result.
Accordingly, in my case, bridging goes well not on rc.conf setting but command-line-setting.
Thank you.


----------



## SirDice (Dec 23, 2019)

```
ifconfig_bridge0="addm igb1 addm igb3 up"
ifconfig_igb1="up"
ifconfig_igb3="up"
ifconfig_bridge0="addm igb1 addm igb3 up"
ifconfig_bridge0="inet 13.6.0.1 netmask 255.255.255.0"
```
You're defining `ifconfig_bridge0` three times. Only the last one will 'stick'. Keep in mind that rc.conf is, in essence, just a shell script that sets variables.

For example:

```
#!/bin/sh

var1="foo"
var1="bar"

echo $var1
```
The script will output "bar" because the second assignment overrules the first. The rc.conf file works exactly the same, it's just a shell script that gets sourced a number of times by various rc(8) scripts. 

In your case you should use this:

```
ifconfig_igb1="up"
ifconfig_igb3="up"
ifconfig_bridge0="addm igb1 addm igb3 inet 13.6.0.1 netmask 255.255.255.0"
```


----------



## LVLouisCyphre (Dec 27, 2019)

SirDice said:


> You cannot have multiple routes, routing the same thing. Well, technically you can but you're going to need a dynamic routing protocol like OSPF or BGP to make that work. Learn the basic first before diving into those.
> 
> This isn't really a FreeBSD issue, you need to learn more about TCP/IP and routing.
> 
> ...


This is not just a an IP issue, but also an ISO model layer 2.  Bridging interfaces may do more harm than good.

If the connections are also at L2 then you need to have STP enabled on the switch or you'll have a network loop if there's bridging.  I haven't used FreeBSD for bridging so I don't know if it has a L2 loop guard.

The question is why does the OP want to have two connections to the same broadcast domain and subnet.  If it's for link aggregation, lagg, if it's for strictly fault tolerance carp.


----------



## SirDice (Dec 30, 2019)

LVLouisCyphre said:


> I haven't used FreeBSD for bridging so I don't know if it has a L2 loop guard.




```
The if_bridge driver implements the Rapid Spanning Tree Protocol (RSTP or
     802.1w) with backwards compatibility with the legacy Spanning Tree
     Protocol (STP).  Spanning Tree is used to detect and remove loops in a
     network topology.
```
From bridge(4).

But I agree, bridge(4) is usually not what you want to do.


----------

