# Reaching default gateway from VLAN



## guidors (Jun 2, 2019)

Hi,

On my FreeBSD 11.2 box I've setup a VLAN to be used for several jails. For that particular VLAN I've also setup a separate routing table using FIB. Now I would like to be able to access the default gateway from my VNET. However I do not get that working.

The host has ip 192.168.178.15 and part of the network 192.168.178.0/24. The VLAN has network 192.168.180.0/24 on which I have two jails running, 192.168.180.2 and 192.168.180.3. I consider 192.168.180.1 as the gateway address of the VLAN. I use routed to manage the routing however previously I tried static routes.

Now I can ping the hosts ip address from the VLAN:


```
$ ping -S 192.168.180.1 192.168.178.15
PING 192.168.178.15 (192.168.178.15) from 192.168.180.1: 56 data bytes
64 bytes from 192.168.178.15: icmp_seq=0 ttl=64 time=0.018 ms
64 bytes from 192.168.178.15: icmp_seq=1 ttl=64 time=0.045 ms
```

but not beyond to the default router:


```
$ ping -S 192.168.180.1 192.168.178.1
PING 192.168.178.1 (192.168.178.1) from 192.168.180.1: 56 data bytes
<... nothing happens here>
```

Should I be able to set a defaultrouter for the VNET/FIB1? Is that possible, I could not find anything on that.
Or should I do this with bridges or possibly even better with real NIC's?

This is the relevant rc.conf config:

```
ifconfig_em0="inet 192.168.178.15/24 fib 0"
defaultrouter="192.168.178.1"
vlans_em0="1"
ifconfig_em0_1="inet 192.168.180.1/24 fib 1"
gateway_enable="YES"
routed_enable="YES"
```


My interfaces are setup as follows:

```
$ ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 70:85:c2:29:b2:9e
        hwaddr 70:85:c2:29:b2:9e
        inet 192.168.178.15 netmask 0xffffff00 broadcast 192.168.178.255
        inet6 fe80::7285:c2ff:fe29:b29e%em0 prefixlen 64 scopeid 0x1
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

$ ifconfig em0.1
em0.1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 70:85:c2:29:b2:9e
        inet 192.168.180.1 netmask 0xffffff00 broadcast 192.168.180.255
        inet 192.168.180.2 netmask 0xffffff00 broadcast 192.168.180.255
        inet 192.168.180.3 netmask 0xffffff00 broadcast 192.168.180.255
        inet6 fe80::7285:c2ff:fe29:b29e%em0.1 prefixlen 64 scopeid 0xa
        nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        fib: 1
        vlan: 1 vlanpcp: 0 parent interface: em0
        groups: vlan
```

And these are the routing tables:


```
$ netstat -nr4 -F 0
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.178.1      UGS         em0
127.0.0.1          link#2             UH          lo0
127.0.2.1          link#4             UH          lo2
127.0.3.1          link#5             UH          lo3
192.168.178.0/24   link#1             U           em0
192.168.178.15     link#1             UHS         lo0
192.168.180.0/24   192.168.180.1      U         em0.1

and

$ netstat -nr4 -F 1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          lo0                UHS         lo0
192.168.180.0/24   link#10            U         em0.1
192.168.180.1      link#10            UHS         lo0
192.168.180.2      link#10            UHS         lo0
192.168.180.3      link#10            UHS         lo0
```

Thanks,
Guido


----------



## johnblue (Jun 5, 2019)

As you may know, the intent of a virtual LAN is, generally speaking, to segment traffic.  As an aside, I personally like to tag up my vLANs to the third octet because it makes it easier to step through when troubleshooting.  Instead of vLAN 1, as in your example, it would be vLAN 180.

I also prefer to have layer 2 devices handle vLAN's so I honestly will not be of much help .. except for my google search chi:











						FreeBSD Jails with VLAN HOWTO.md
					

GitHub Gist: instantly share code, notes, and snippets.




					gist.github.com
				




That URL has vLANs, jails and the rev level of your OS.

Hopefully there will be enough there to pick and choose from to rerail your project.


----------



## tommiie (Jun 5, 2019)

I believe `em0` will be the _default_ (untagged) VLAN while `em0.1` will be VLAN 1. However, on all switches that I know of, VLAN 1 _is_ the default (untagged) VLAN so in fact both would be the same VLAN. Could you provide some information on your switch model and VLAN configuration? That will help identify the issue and provide a solution.


----------



## guidors (Jun 7, 2019)

johnblue said:


> As you may know, the intent of a virtual LAN is, generally speaking, to segment traffic.  As an aside, I personally like to tag up my vLANs to the third octet because it makes it easier to step through when troubleshooting.  Instead of vLAN 1, as in your example, it would be vLAN 180.
> 
> I also prefer to have layer 2 devices handle vLAN's so I honestly will not be of much help .. except for my google search chi:
> 
> ...



Thanks for your response. Clever idea,that naming trick. In the example you found a bridge solution is used. I'll think  will expiriment with that if I get my original plan not working.


----------



## SirDice (Jun 7, 2019)

You need to tell the command which routing table to use, this isn't "automatic". For example `setfib 1 ping {....}` or `setfib 0 ping {....}`. Although the latter isn't needed, if not specified the default routing table is used (FIB 0).


----------



## phoenix (Jun 7, 2019)

tommiie said:


> I believe `em0` will be the _default_ (untagged) VLAN while `em0.1` will be VLAN 1. However, on all switches that I know of, VLAN 1 _is_ the default (untagged) VLAN so in fact both would be the same VLAN. Could you provide some information on your switch model and VLAN configuration? That will help identify the issue and provide a solution.



`em0` and `em0.1` both use the default vlan ID of 1.

The difference is that `em0` sends/receives untagged frames (the switch handles figuring out all the vlan stuff based on port configuration), and `em0.1` sends/receives tagged frames only, and only processes frames tagged with vlan ID 1.


----------



## guidors (Jun 8, 2019)

phoenix said:


> `em0` and `em0.1` both use the default vlan ID of 1.
> 
> The difference is that `em0` sends/receives untagged frames (the switch handles figuring out all the vlan stuff based on port configuration), and `em0.1` sends/receives tagged frames only, and only processes frames tagged with vlan ID 1.


That is what I thought myself. However, I renamed my vlan to 180 so now I have interfaces em0 and em0.180.
And TaTa:

```
$ ping -S 192.168.180.1 192.168.178.1
PING 192.168.178.1 (192.168.178.1) from 192.168.180.1: 56 data bytes
64 bytes from 192.168.178.1: icmp_seq=0 ttl=64 time=0.547 ms
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=0.732 ms
```

So Tommiie has a point! Somewhere using vlan 1 is messing things up.

I could not straight away ping from FIB 1

I first had to add a default route:

```
$ setfib 1 route add -net 192.168.178.0/24 -iface em0
```

And now I could do:

```
$ setfib 1 ping -S 192.168.180.1 192.168.178.1
PING 192.168.178.1 (192.168.178.1) from 192.168.180.1: 56 data bytes
64 bytes from 192.168.178.1: icmp_seq=0 ttl=64 time=0.235 ms
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=0.447 ms
```

Yes!

To be complete, the new routing table for fib 1 is now:

```
$ root@hazelaar:~ # netstat -nr4 -F 1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          lo0                UHS         lo0
192.168.178.0/24   70:85:c2:29:b2:9e  US          em0
192.168.180.0/24   link#10            U       em0.180
192.168.180.1      link#10            UHS         lo0
192.168.180.2      link#10            UHS         lo0
192.168.180.3      link#10            UHS         lo0
```


So recap:

It is very well possible to have a vlan and routing between them without bridging;
It is not advisable to use vlan 1 since that might be the default vlan id for some routers.
in my case a FRITZ!Box.


----------



## tommiie (Jun 9, 2019)

phoenix said:


> `em0` and `em0.1` both use the default vlan ID of 1.
> 
> The difference is that `em0` sends/receives untagged frames (the switch handles figuring out all the vlan stuff based on port configuration), and `em0.1` sends/receives tagged frames only, and only processes frames tagged with vlan ID 1.


Thanks! That is an important remark to make. And it will cause problems as it's both VLAN 1 (by default on a switch, untagged frames belong to VLAN 1) but two different subnets.


----------



## guidors (Jun 10, 2019)

So weird. I am back to square one. After my last post I added the static route to my rc.conf.


```
static_routes="xn"
route_xn="-net 192.168.178.0/24 -iface em0 -fib 1"
```

rebooted, which I do often just to be at the safe side and again I could not ping the default gateway.

```
$ ping -S 192.168.180.1 192.168.178.1
PING 192.168.178.1 (192.168.178.1) from 192.168.180.1: 56 data bytes
<... nothing happens here>
```

I tried everything to get it working, like renumbering the vlans again but I have not been able to ping the default gateway anymore. So weird. I start to believe that this is an issue with the router (FRITZ!Box 7581). Have you ever experienced something like this?


----------



## tommiie (Jun 10, 2019)

Please share your current configuration as well as your switch configuration.


----------



## aribi (Jun 10, 2019)

guidors said:


> ```
> static_routes="xn"
> route_xn="-net 192.168.178.0/24 -iface em0 -fib 1"
> .........
> ...



Maybe you should `setfib 1 ping` to ping using the routing you defined? Just as in your previous success post.


----------



## guidors (Jun 17, 2019)

Hi,

Late reaction after a busy period.. I have ordered a physical NIC since I start to believe the issue is caused by using VLAN's in my home network.
Basically I want to achieve this:

```
+-------------------+
 |     Internet      |
 +---------+---------+
           |
+----------+----------+
|     Fritz net       |
|  192.168.178.0/24   |
+----------+----------+
           |
+----------+----------+ 
|     local net       |
|     (FreeBSD)       |
|   192.168.178.15    |
|  192.168.180.0/24   |
+----------+----------+
```

And have a series of jails using the 192.168.180.0/24 network.

My /etc/rc.conf is as follows:


```
ifconfig_em0="inet 192.168.178.15/24 fib 0"
defaultrouter="192.168.178.1"

# vlans
vlans_em0="180"

# vlan 
ifconfig_em0_180="inet 192.168.180.1/24 fib 1"

routed_enable="YES"
```

And my /boot/loader.conf as follows:


```
if_vlan_load="YES"
net.fibs=2
net.add_addr_allfibs=0
```

I did add a route on my home router: 192.168.180.0/24 => 192.168.178.15 but I believe that that does not have to exist for the ping test I do.

To my surprise I always get a ping reaction from the router when I restart it. But when it finishes the restart cycle I cannot do the ping any more. From the technical log from the FRITZ!Box 7581 I noticed that the phone service also uses vlan. So may be the internal firewall from the router has some strict rules.


----------



## aribi (Jun 20, 2019)

Don't know if this will solve your problem, but *net.add_addr_allfibs* needs to be set in /etc/sysctl.conf according to this thread.
Furthermore, please note that VLAN-tagging is a datalink protocol (OSI level 2) and you seem to have only 1 link, so why bother?. 
Perhaps you should rethink your setup - do you really need VLANS?
Using a bridge inteface might be an easier (cleaner) solution. Something allong the lines:

```
# rc.conf
ifconfig_em0="inet 192.168.178.15/24"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up description jailbridge"

# jail.conf
somejail {
...
ip4.addr = "bridge0:192.168.180.2/24"
...
}
```


----------



## SirDice (Jun 20, 2019)

aribi said:


> Furthermore, please note that VLAN-tagging is a datalink protocol (OSI level 2) and you seem to have only 1 link, so why bother?.


The whole point of VLAN tagging is to allow separated networks on 1 (physical) link:

```
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 90:e2:ba:54:ff:22
        inet6 fe80::92e2:baff:fe54:ff22%em1 prefixlen 64 scopeid 0x3
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em1.1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:54:ff:22
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::92e2:baff:fe54:ff22%em1.1 prefixlen 64 scopeid 0x6
        groups: vlan
        vlan: 1 vlanpcp: 0 parent interface: em1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em1.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:54:ff:22
        inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
        inet6 fe80::92e2:baff:fe54:ff22%em1.10 prefixlen 64 scopeid 0x7
        inet6 2001:470:1f15:bcd::1 prefixlen 64
        groups: vlan
        vlan: 10 vlanpcp: 0 parent interface: em1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em1.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:54:ff:22
        inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
        inet6 fe80::92e2:baff:fe54:ff22%em1.11 prefixlen 64 scopeid 0x8
        inet6 2001:470:7989:11::1 prefixlen 64
        groups: vlan
        vlan: 11 vlanpcp: 0 parent interface: em1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em1.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:54:ff:22
        inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
        inet6 fe80::92e2:baff:fe54:ff22%em1.20 prefixlen 64 scopeid 0x9
        inet6 2001:470:7989:20::1 prefixlen 64
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: em1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

And you can then route between them:

```
% netstat -rn4 | grep em1
10.0.0.0/24        link#6             U         em1.1
10.0.1.0/24        link#9             U        em1.20
192.168.10.0/24    link#7             U        em1.10
192.168.11.0/24    link#8             U        em1.11
```

According to your definition I would have to have 4 physical interfaces and 4 network cables to my switch in order to route between the different VLANs?


----------



## guidors (Jun 22, 2019)

Ok, one last post to close this thread. First, it work! 
Secondly, I do not know how but it is most likely to to do with a bug in the Fritz.Box I am using.
I reproduce issues on virtual machines to be sure if an issue is an issue. So I did not have any luck with the vlan so I tried bridging on my VM. And after I changed the route on my router from 192.168.180.0/24 => 192.168.178.15 (my FreeBSD server) to 192.168.180.0/24 => 192.168.178.215 (my VM FreeBSD instance) it worked in bridge mode!
I changed the configuration of my VM to vlan mode and it worked as well!
I change the configuration on the Fritz.Box router back to 192.168.180.0/24 => 192.168.178.15 which is a single digit change and now it worked on my FreeBSD server as well! Maddening, really, infuriating!!


----------



## PMc (Jun 22, 2019)

Not really clear why this VLAN effort is necessary. I'm using source-based routing for my jails (and everything else) so they can use different uplinks (defaultroutes). And thinking IoT, it might be difficult to install the proper VLAN on a fridge or motor-driven curtain. But alright then, the VLAN might be more secure...


----------



## phoenix (Jun 26, 2019)

PMc said:


> Not really clear why this VLAN effort is necessary. I'm using source-based routing for my jails (and everything else) so they can use different uplinks (defaultroutes). And thinking IoT, it might be difficult to install the proper VLAN on a fridge or motor-driven curtain. But alright then, the VLAN might be more secure...



You don't need to have vlan-capable devices to use vlans.
You need to have a vlan-capable switch.  For devices that don't understand/use vlan tags, you configure the port on the switch to be part of vlan X.  It will only be able to communicate with other ports that are also on vlan X.  As packets enter that port, the switch tags them with vlan X headers and passes them around as needed.  It's how you can put Windows stations onto vlan X without making any changes to the actual system, as trying to get vlan tags working on a Windows station is ... to put it bluntly ... a Royal PITA!


----------



## PMc (Jun 26, 2019)

Yes, thats the normal thing VLAN is used for: to create logical subnets that are different than the physical subnets, e.g. if you have machines in different sites that need to be on the same subnet.


----------

