# Strange network setup -- how to configure the same on FreeBSD?



## zirias@ (Jun 19, 2019)

As a bit of background, I'm trying to "depenguinate" a rented vserver. It's running Debian 9 right now, hosted on XEN. XEN provides a virtual framebuffer that guest systems can use, and my provider allows to view it through VNC. Unfortunately, grub2 doesn't seem to have a driver for this, so I never see a boot menu. I tried booting an mfsbsd ISO image from grub2 following this tip and it didn't work (but I'm not sure whether FreeBSD knows and uses XEN's virtual framebuffer).

Now, I'm starting to think that _just maybe_, my image is booting fine and I can't see anything because FreeBSD doesn't have a driver for this XEN framebuffer? It could be that it just doesn't work because of a strange network configuration.

On Debian, it looks like this in /etc/network/interfaces:

```
auto eth0
iface eth0 inet static
  address 192.168.35.42
  netmask 255.255.255.255
  up route add -host 10.19.205.1 dev eth0
  up route add default gateway 10.19.205.1
```

Note I *changed the IP addresses* here, the real interface address is a public routed address, the gateway though is in a private range. So indeed, the default gateway isn't even in the same class-a network as the address assigned to my virtual machine.

Can I do a similar configuration on FreeBSD at all? My attempt when building my mfsbsd image looked like this (in conf/rc.conf):


```
defaultrouter="10.19.205.1"
ifconfig_xnb0="inet 192.168.35.42 netmask 255.255.255.255"
static_routes="defgw"
route_defgw="-host 10.19.205.1 192.168.35.42"
```

Does this look somewhat correct? One thing I noticed is that I can't set a route on an interface like with Linux, just on a gateway, therefore I picked the address of my interface here ...

Thanks for hints!


----------



## Martin Paredes (Jun 20, 2019)

After reading route(8), I think you should try with


```
route_defgw="-host 10.19.205.1 -interface xnb0"
```


----------



## zirias@ (Jun 20, 2019)

Normally, I'm very happy with the documentation quality of FreeBSD. But I know had another look at route(8) myself, and it is just confusing:

SYNOPSIS reads `route [-dnqtv] command [[modifiers] args]`, indicating any "modifiers" have to directly follow the command (like e.g. "add")
In the section describing modifiers, there's `-iface` mentioned .. ok, that's what I was looking for and thought it wasn't possible on FreeBSD
In the section desrcibing the "add" command, it says "[...]directly reachable via an interface [...] the *-interface* modifier should be specified;" -- so which one is correct, `-iface` or `-interface`?
"the gateway given is the address of this host on the common network" <- so, I can't just write the name of the interface instead, right?
I find very few examples with google using a syntax similar to your suggestion, but using `-iface`.

Following this manpage, I could come up with something like

```
route_defgw="-host -interface 10.19.205.1 192.168.35.42"
```
But I can't believe this is correct ... this manpage is a failure


----------



## SirDice (Jun 20, 2019)

It's not correct, it should be:

```
ifconfig_xnb0="inet 192.168.35.42 netmask 255.255.255.255"
static_routes="defgw"
defaultrouter="10.19.205.1"
route_defgw="-host 10.19.205.1 -iface xnb0"
```


----------



## zirias@ (Jun 20, 2019)

Well thanks to both of you ... I already tried the following after Martin's answer:

```
#
# You need a gateway defined for a working network setup
defaultrouter="10.19.205.1"
#
# You may configure a network interface here, you may use "DHCP" as well
ifconfig_xnb0="inet 192.168.35.42 netmask 255.255.255.255"
static_routes="defgw"
route_defgw="-host 10.19.205.1 -iface xnb0"
```

.. and it didn't work either  As I still can't see *anything* (XEN virtual framebuffer is never initialized), I'm back to assuming my mfsbsd ISO image isn't even booted by grub2  That is except someone has an idea what else could be wrong. Order should never matter in rc.conf, right?

Anyways, this particular manpage (route(8)) really is confusing ...


----------



## SirDice (Jun 20, 2019)

Thinking about it, I believe the problem is with the default gateway. On FreeBSD a gateway _must_ reside on a directly attached network, any attempt to set a gateway address to something _outside_ any one of the directly connected networks is going to fail. So I suspect that the added static route works but defaultrouter fails to apply. 



Zirias said:


> Order should never matter in rc.conf, right?


Correct. This would have the exact same results:

```
defaultrouter="1.1.1.1"
route_mystatic="2.0.0.0/8 1.1.1.2"
```


```
route_mystatic="2.0.0.0/8 1.1.1.2"
defaultrouter="1.1.1.1"
```


----------



## zirias@ (Jun 20, 2019)

SirDice said:


> On FreeBSD a gateway _must_ reside on a directly attached network, any attempt to set a gateway address to something _outside_ any one of the directly connected networks is going to fail.



Wow ... so this would mean I could never configure this strange routing setup with FreeBSD at all? Or maybe, it would require to set netmask to 0.0.0.0, manually removing the route that's automatically set and adding my routes as before? If so, can I remove a route from rc.conf or would I have to write my own custom init-script?


----------



## SirDice (Jun 20, 2019)

Can you try a couple of things on the console? Only set the IP address of the machine in rc.conf and leave out any routing. Then on the console do some experimenting with the route(8) command. 

Specifically these (in this order):

```
route add -host 10.19.205.1 -iface xnb0
route add default 10.19.205.1
```

If those are accepted and appear to work we can translate those to:

```
static_routes="static1 static2"
route_static1="-host 10.19.205.1 -ifcase xnb0"
route_static2="default 10.19.205.1"
```
Don't set defaultrouter in that case. The idea here is that we have control over the order of the static routes.


----------



## zirias@ (Jun 20, 2019)

SirDice said:


> Can you try a couple of things on the console?


Unfortunately, no  I don't have any accessible console as long as I can't login through SSH. Nothing initializes the XEN virtual framebuffer (not sure whether FreeBSD would be expected to do so?) and my provider doesn't provide access to a serial console 

*edit:* will try your suggestion anyways, maybe the rc scripts handle `defaultrouter` before adding configured static routes, so I get the idea


----------



## zirias@ (Jun 20, 2019)

Still no cake... well if that's true (a gateway MUST be directly reachable on an interface according to the netmask), I guess the way to go is indeed:

configure a netmask of 0.0.0.0
remove the default route that's automatically added by this (whole internet directly on that interface)
configure the static routes as above
I guess I'll have to write my own script for that  will try.


----------



## SirDice (Jun 20, 2019)

Do you have the resources to create two VMs somewhere? You could set up a functionally similar test network, that would make it a bit easier to experiment with various routing commands.


----------



## zirias@ (Jun 20, 2019)

SirDice said:


> Do you have the resources to create two VMs somewhere? You could set up a functionally similar test network, that would make it a bit easier to experiment with various routing commands.


I do (a basic FreeBSD vm doesn't eat up many resources after all), and I already had this idea as well. Will probably do this as a last resort. It'll be a weird exercise


----------



## SirDice (Jun 20, 2019)

I'm sure it's possible, we just need to figure out the correct route(8) commands in order to coax the system into accepting it. Being able to experiment on the command line makes this a lot easier to test.


----------



## zirias@ (Jun 20, 2019)

The problem I'm facing is that I have two possible failures, either grub2 already fails to boot my mfsbsd image, or the network setup in this image doesn't work. I have no way to figure out which one is the problem right now 

But yes, if I simulate this strange network setup in an environment I control myself, I can figure out a way to configure it on FreeBSD I know will work, so if the mfsbsd image still doesn't come up, I will know the problem is booting it from grub2 ... in case anyone is interested, I'll post the result  Thanks for assistence so far!


----------



## zirias@ (Jun 21, 2019)

SirDice said:


> It's not correct, it should be:
> [...]


This version already works as expected (unfortunate for me, as my problem must be booting the mfsbsd iso image in the first place ).

I verified by adding a new bhyve VM to the bridge for my guest network 192.168.101.0/24 and adding a static host route to 10.0.0.1 to that bridge on the router/firewall VM. In this new VM, I configured the following:

```
ifconfig_vtnet0="inet 10.0.0.1 netmask 255.255.255.255"
static_routes="defgw"
defaultrouter="192.168.101.1"
route_defgw="-host 192.168.101.1 -iface vtnet0"
```

This leads to the following routing table:

```
Destination        Gateway            Flags     Netif Expire
default            192.168.101.1      UGS      vtnet0
10.0.0.1           link#1             UHS         lo0
10.0.0.1/32        link#1             U        vtnet0
127.0.0.1          link#2             UH          lo0
192.168.101.1      58:9c:fc:00:d2:e9  UHS      vtnet0
```
and the network works just fine.

So, obviously, a gateway outside the network configured for an interface still works


----------

