# Routing Across Subnets via 1 FreeBSD Installation?



## msullivan (May 26, 2011)

Hi all!

I'm having trouble wrapping my head around what to do to make this happen, so I'm hoping that any of you with more experience might have an idea.

Backstory: I'm working on a simulation related to some undergraduate research work.  For this simulation I've got three VMs inside of VirtualBox; all three of these are connected to the same virtual network switch, which we'll call "BLUE_SWITCH".

I have the three VMs on completely different subnets:
172.16.1.1/24 (VM_1)
192.168.1.1/24 (VM_2)
10.10.1.1/24 (VM_3)

In my perfect world, I would like to create a FreeBSD install (VM_BSD) as the default gateway for all three of those subnets.  Then, if VM_1 wanted to contact VM_2, the VM_BSD would forward those packets appropriately.

A) Can it be done at all?  I'm assuming yes, but it's probably wizardry.
B) If it can be done, can someone provide a bit of config or an example?

The preceding are the parameters which I'm able to work with; while it would be much easier to just throw them on the same subnet and call it a day, this simulation simply can't operate like that (this small issue is part of a much larger project).

Any help would be very appreciated!  I'm more of a software guy, so I'm admittedly out of my league here and anything is appreciated


----------



## SirDice (May 26, 2011)

a) yes
b) nothing more would be needed then a 
	
	



```
gateway_enable="YES"
```
 in /etc/rc.conf.


----------



## msullivan (May 26, 2011)

Sorry to be a n00bsauce here, but in addition to enabling gateway mode how would I tell FreeBSD what networks it was working with?  Aliases in each range?


```
ifconfig_em0_alias0="172.16.1.1/24"
ifconfig_em0_alias1="192.168.1.1/24"
ifconfig_em0_alias2="10.10.1.1/24"
```

And finally, would I be needing to set static routes anywhere at all?


----------



## SirDice (May 26, 2011)

msullivan said:
			
		

> Aliases in each range?


Or multiple interfaces, each in its own subnet.



> And finally, would I be needing to set static routes anywhere at all?


Nope. All "leaves" have a default gateway pointing to the FreeBSD box. The FreeBSD box itself doesn't need any static routes as all the networks are "directly connected".


----------



## msullivan (May 26, 2011)

Alrighty, I'll give that a try then.  Thanks for your help!


----------



## msullivan (May 27, 2011)

Well, I had some time to give this a try today.  We're close, but not quite working.

On the FreeBSD box I am doing:


```
ifconfig_em0_alias0="172.16.1.254 netmask 255.255.255.0"
ifconfig_em0_alias1="192.168.1.254 netmask 255.255.255.0"
ifconfig_em0_alias2="10.10.1.254 netmask 255.255.255.0"
```
VM_1 is set to be 172.16.1.1
VM_2 is set to be 192.168.1.1
VM_3 is set to be 10.10.1.1

Let's say I'm on VM_1.  I can ping 172.16.1.254, 192.168.1.254 and 10.10.1.254 just fine presently (all gateway addresses).  However, if I try to ping VM_2 at 192.168.1.1 I get no response.  Traceroutes also show my packet making the first step to the router with no issues, but after that it doesn't get forwarded anywhere and dies.

Any thoughts?  Thanks!


----------



## msullivan (May 27, 2011)

Oops, sorry for the double-post.  It's also relevant to note that I've played with the subnet masks, changing them between "255.255.255.255", "255.255.255.0", or just leaving them off entirely.  No dice when doing that either.


----------



## SirDice (May 27, 2011)

Sounds like it's not forwarding anything.
Did you add 
	
	



```
gateway_enable="YES"
```
 to /etc/rc.conf?

Don't forget to reboot or enable it by hand:

```
sysctl net.inet.ip.forwarding=1
```

Also turn off any firewalls if they're enabled. Get this working first.


----------



## msullivan (May 27, 2011)

```
gateway_enable="YES"
```
 is the first line in rc.conf; confirmed the setting is working by issuing:

```
# sysctl net.inet.ip.forwarding
sysctl net.inet.ip.forwarding: 1 ( <-- returned 'yes' )
```

No other firewalls are running anywhere in the infrastructure; the FreeBSD server is a clean install with only sshd running, nothing else.


----------



## SirDice (May 27, 2011)

Ok, can you post the output of
`$ netstat -rn`

That should show the routing table.


----------



## msullivan (May 27, 2011)

Sure thing.

```
Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
34.6.92.0/24       link#1             U           0        3    em0
34.6.92.254        link#1             UHS         0        0    lo0
35.6.92.0/24       link#1             U           0        0    em0
35.6.92.254        link#1             UHS         0        0    lo0
127.0.0.1          link#3             UH          0      117    lo0
172.16.4.0/29      link#2             U           0        0    em1
172.16.4.2         link#2             UHS         0        0    lo0
199.100.16.0/24    link#1             U           0        0    em0
199.100.16.100     link#1             UHS         0        0    lo0
199.100.16.254     link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01:3::/32                       fe80::1%lo0                   U           lo0
ff02::%lo0/32                     fe80::1%lo0                   U           lo0
```


----------



## msullivan (May 27, 2011)

Oh, I should note that my example from the original posts is simplified; the real address ranges I'm using are:
34.6.92.0/24 
35.6.92.0/24 
199.100.16.0/24

This network does not have any internet access or internet-viewing gateways, so the IP address scheme does not conflict with anything in the real world.


----------



## SirDice (May 27, 2011)

msullivan said:
			
		

> ```
> 34.6.92.0/24       link#1             U           0        3    em0
> 34.6.92.254        link#1             UHS         0        0    lo0
> 35.6.92.0/24       link#1             U           0        0    em0
> ...


This doesn't look right. The address should be bound to em0, not lo0. Similarly for the other addresses.

Or do you have your VMs bound to lo0?


----------



## msullivan (May 27, 2011)

Well for what it's worth, I quickly installed FreeBSD onto a real, physical box, then attempted to add an alias in the same way.  After a *netstat -nr*, the IP alias I gave it reported lo0 with its mask on em0, just like in my VMs - so I'm thinking that's normal and default behavior.

My rc.conf parameters are as posted a few pages up (syntax of "ifconfig_em0_aliasX=BLAH").  Does that syntax look correct to you?  From all the postings I've found it appears to be correct.

This is a doosie, but I really need to make it work, so I'm hoping for the best


----------



## wblock@ (May 27, 2011)

This might be made clearer by posting the whole /etc/rc.conf.


----------



## msullivan (May 27, 2011)

wblock said:
			
		

> This might be made clearer by posting the whole /etc/rc.conf.



Fair enough.  Nothing too special.


```
gateway_enable="YES"
hostname="proxy-int.svc1.mytestnetwork.net"
ifconfig_em0="inet 199.100.16.100 netmask 255.255.255.0"
ifconfig_em1="inet 172.16.4.2/29"
sshd_enable="YES"
squid_enable="YES"
gateway_enable="YES"
ifconfig_em0_alias0="199.100.16.254 netmask 255.255.255.0"
ifconfig_em0_alias1="35.6.92.254 netmask 255.255.255.0"
ifconfig_em0_alias2="34.6.92.254 netmask 255.255.255.0"
```

Just for kicks, here's an ifconfig as well:

```
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:d2:f4:bf
        inet 199.100.16.100 netmask 0xffffff00 broadcast 199.100.16.255
        inet 199.100.16.254 netmask 0xffffff00 broadcast 199.100.16.255
        inet 35.6.92.254 netmask 0xffffff00 broadcast 35.6.92.255
        inet 34.6.92.254 netmask 0xffffff00 broadcast 34.6.92.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:f3:96:b7
        inet 172.16.4.2 netmask 0xfffffff8 broadcast 172.16.4.7
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
```


----------



## wblock@ (May 27, 2011)

msullivan said:
			
		

> ```
> gateway_enable="YES"
> hostname="proxy-int.svc1.mytestnetwork.net"
> [B]ifconfig_em0="inet 199.100.16.100 netmask 255.255.255.0"[/B]
> ...



The main IP address and that alias are in the same net.  Just make the main address 199.100.16.254 and don't add an alias, or use a different network for the main address.

It looks wrong without a defaultrouter entry, but I guess you don't need one if it's all internal traffic.


----------



## msullivan (May 28, 2011)

wblock said:
			
		

> The main IP address and that alias are in the same net.  Just make the main address 199.100.16.254 and don't add an alias, or use a different network for the main address.
> 
> It looks wrong without a defaultrouter entry, but I guess you don't need one if it's all internal traffic.



That alias was leftover from some other testing, and although it is a good catch, removing it doesn't seem to affect anything one way or the other.  Same result... I can always ping from a VM to a gateway, but from VM to VM still fails after the packet makes it to the FreeBSD installation.


----------



## wblock@ (May 28, 2011)

Are the VMs using NAT or bridged networking?  NAT will break pings.  If the gateway is also a VM, or the VM host machine, there's "internal network" and "host-only networking" modes also.


----------



## toddnni (May 28, 2011)

I'm just wondering that does IP forwarding work using only one interface? For example on Linux forwarding works only between separate interfaces. Maybe you need to create three separate networks inside VirtualBox and use three interfaces on the FreeBSD system.


----------



## msullivan (May 28, 2011)

wblock said:
			
		

> Are the VMs using NAT or bridged networking?  NAT will break pings.  If the gateway is also a VM, or the VM host machine, there's "internal network" and "host-only networking" modes also.



Yes, the FreeBSD installation is also a VM.  All four are connected using the "host-only network" option presently, but "internal network" has also been tried with the same results.


----------



## msullivan (May 28, 2011)

toddnni said:
			
		

> I'm just wondering that does IP forwarding work using only one interface? For example on Linux forwarding works only between separate interfaces. Maybe you need to create three separate networks inside VirtualBox and use three interfaces on the FreeBSD system.



I had considered that, but eventually this *might* need to be scaled up to even more subnets (it depends on how the first set of tests goes).  If that happens, we'll be adding about 8 more VMs with separate subnets, and VirtualBox only supports 8 networking devices.

There's got to be some way...


----------



## toddnni (May 28, 2011)

msullivan said:
			
		

> I had considered that, but eventually this *might* need to be scaled up to even more subnets (it depends on how the first set of tests goes).  If that happens, we'll be adding about 8 more VMs with separate subnets, and VirtualBox only supports 8 networking devices.
> 
> There's got to be some way...



I don't know if performance is important in this simulation, but you could try daisy-chain FreeBSD routers. Unfortunately you need to set routes between routers then.


----------



## razixx (May 30, 2011)

Just on a side note,

```
gateway_enable="YES"
hostname="proxy-int.svc1.mytestnetwork.net"
[B]ifconfig_em0="inet 199.100.16.100 netmask 255.255.255.0"[/B]
ifconfig_em1="inet 172.16.4.2/29"
sshd_enable="YES"
squid_enable="YES"
gateway_enable="YES"
[B]ifconfig_em0_alias0="199.100.16.254 netmask 255.255.255.[color="DarkRed"]255[/color]"[/B]
ifconfig_em0_alias1="35.6.92.254 netmask 255.255.255.0"
ifconfig_em0_alias2="34.6.92.254 netmask 255.255.255.0"
```
Should work as well.  Whenever you have an alias ip that is in the same subnet you want to set the netmask to 255.255.255.255


----------

