# Visibility of  subnet through the gateway



## Labaman (Aug 15, 2014)

My modem  (IP 192.168.1.1), is connected to the gateway through nfe0 (IP 192.168.1.100), and the second network card ue0 (IP 192.168.0.1) gives the 'Net to LAN (192.168.0.0/24).
I want to provide access to several IP addresses from the local network, such as 192.168.0.3 to the modem (IP 192.168.1.1).
How can this be implemented?


----------



## ondra_knezour (Aug 15, 2014)

You have to enable routing, see 30.2.2. Configuring a Router with Static Routes in the Handbook.


----------



## Labaman (Aug 16, 2014)

Do I need to recompile the kernel with options MROUTING?

```
netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0     3372   nfe0
127.0.0.1          link#2             UH          0        3    lo0
192.168.0.0/24     link#3             U           0      572    ue0
192.168.0.1        link#3             UHS         0       11    lo0
192.168.1.0/24     link#1             U           1     6886   nfe0
192.168.1.100      link#1             UHS         0     3387    lo0
```
I'm trying to do:

```
route add -net 192.168.1.0/24 192.168.0.1
route: writing to routing socket: File exists
add net 192.168.1.0: gateway 192.168.0.1 fib 0: route already in table
```
What's wrong?


----------



## ondra_knezour (Aug 16, 2014)

No, MROUTING is not required. However, you need to either teach the opposite side - the modem - about the route back, it is now probably in situation like "Ouch, I got packet from somewhere, where I am supposed to send reply to?" or enable a network address translation, check also the Handbook for more info.

You can't add that route manually, because the kernel already knows about network connected locally, IP address and subnet mask says that for example everything like 192.168.1.0-255 is connected directly to interface x.

Also your IPs configured on the lo0 interface looks little unusual to me. It may (and in many situation, does) work, but in scenario you described I would expect them directly on given interfaces - ue0 and nfe0.


----------



## Labaman (Aug 16, 2014)

If you can, please give a concrete example of the possible actions in my case.
And yes, I have everything goes NAT with nfe0 :

```
ipfw add divert 8868 tcp from any to any via nfe0
```
I'm bewildered....


----------



## ondra_knezour (Aug 16, 2014)

You also have to have the NAT daemon, natd(8) running. I'm not using IPFW for years, but there is of course example in the Handbook and man page 

http://www.freebsd.org/doc/en_US.ISO885 ... -ipfw.html

Using those, following should be sufficient in your case:
`natd -interface nfe0
/sbin/ipfw -f flush
/sbin/ipfw add divert natd all from any to any via nfe0
/sbin/ipfw add pass all from any to any`

That start natd on the outgoing interface, drop any existing firewall rules, setup "routing" packets to natd and finally allow all traffic (makes firewall open).

I am leaving now for two weeks, but hopefully someone will jump in if you have more questions. However, the FreeBSD has excellent documentation including manpages with examples which should be sufficient to follow in almost copy and paste manner to get all basic functions working, always consult them.


----------



## Labaman (Aug 16, 2014)

I've already spelled out the rules. Without them, there would be no Internet in the subnet 192.168.0.0/24.
That's what I had originally in  ipfw rules:

```
add 4000 divert natd ip from any to any via nfe0
add fwd 127.0.0.1,3128 tcp from any to any 80 via ue0 #SQUID
add 65500 allow ip from any to any
```
But subnet 192.168.0.0/24 not to see the subnet 192.168.1.0/24. In this my whole problem.
And, yes, I have repeatedly read handook on ipfw and natd, and even a couple of paper books on freeBSD, but could not find the solution to his problem, so I ask for help from those who may have experienced this in practice.


----------



## ondra_knezour (Aug 16, 2014)

Oh so, I didn't understand, that you have working everything except connections between 192.168.0/24 and 192.168.1/24. In such situation I would try to see what is going on with packet inspection on the gateway using tcpdump(). Do you see any requests from the internal network on both interfaces? And responses? And what about those stations in the internal network, if there is some form of the tcpdump or Wireshark available for them?

Wild guess - most common source of such problems is wrong subnet mask. Either selecting wrong subnet size like /16 or some typo like /23 can do what you are seeing. Stations will try to seek for given target IP on local ethernet segment, you will see some arp requests like who has 192.168.1.x with the tcpdump both on the station and internal interface of the gateway. Because of no reply packet is droped, not forwarded to gateway - target should be in local network and is not responding.


----------



## Labaman (Aug 16, 2014)

`ifconfig`:

```
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8210b<RXCSUM,TXCSUM,VLAN_MTU,TSO4,WOL_MAGIC,LINKSTATE>
        ether e0:69:95:85:2e:cb
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::e269:95ff:fe85:2ecb%nfe0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80008<VLAN_MTU,LINKSTATE>
        ether fc:75:16:cf:76:06
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::fe75:16ff:fecf:7606%ue0 prefixlen 64 scopeid 0x3
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
```
/etc/rc.conf

```
hostname="proxy.local"
keymap="ru.koi8-r.win.kbd"
ifconfig_nfe0="inet 192.168.1.100 netmask 255.255.255.0"
ifconfig_ue0="inet 192.168.0.1 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
#Gateway enable
gateway_enable="YES"
#NAT
natd_enable="YES"
natd_interface="nfe0"
#IPFW firewall
firewall_enable="YES"
firewall_type="/etc/firewall.conf"
# DNS server
local_unbound_enable="YES"
sshd_enable="YES"
moused_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
mysql_enable="YES"
squid_enable="YES"
apache24_enable="YES"
```
And from internal network is not even ping the external network card IP 192.168.1.100, even more not one of 192.168.1.0/24 IP.


----------



## obsigna (Aug 16, 2014)

Three observations:


ue0 is a typical interface name for an usb/ethernet adapter. In one of my systems I got also one of it, and as a matter of fact, the usb-interface driver module was loaded only after the firewall rules got applied, and therefore the firewall was not completely configured in my case, and didn't work as expected. I resolved the issue by directing /boot/loader.conf to load the necessary drivers right at the beginning:

```
...
uether_load="YES"
if_axe_load="YES"
```

divert/natd is well outdated. You might want to use the NAT which is builtin to ipfw(8).
 
AFAIK, ipfw fwd does not work without rebuilding the kernel with options IPFIREWALL_FORWARD. So quite possible your transparent proxy installation won't see any traffic.


----------



## Labaman (Aug 16, 2014)

obsigna said:
			
		

> [*]AFAIK, ipfw fwd does not work without rebuilding the kernel with options IPFIREWALL_FORWARD. So quite possible your transparent proxy installation won't see any traffic.[/list]


I'm use FreeBSD 10 on my gateway. This option is unnecessary for 10. 
A transparent proxy is working fine. I tried to remove the rules "forwarding" on the proxy, but it did not help.
I still could not reach to any one node in the network 192.168.1.0/24\
UPD:
Tried to compile the kernel with IPFIREWALL_NAT, rewrote the rules and configs for the new format - did not help.


----------



## obsigna (Aug 17, 2014)

Labaman said:
			
		

> ...
> I'm trying to do:
> 
> ```
> ...



The point here is, that not the FreeBSD gateway but the modem is missing a route, namely that into your internal network. Your FreeBSD gateway machine knows already about all routes that are needed. If your modem would be *BSD, then your would add the necessary route like so:
`route add -net 192.168.0.0/24 192.168.1.100`.

Presumably the modem can't be configured like this.

Perhaps it would be an option for you to make nfse0 and ue0 into a bridge. Both sides of the bridge would be in the same network, even though, you could place a firewall on it, and all packets from and to your internal network behind the bridge need to pass that firewall.

Neither routing nor nating into a separate network would give additional security.

For example, modify the file /etc/rc.conf as follows:

```
cloned_interfaces="bridge0"
ifconfig_ nfse0 ="up -tso"
ifconfig_ue0="up -tso"
ifconfig_bridge0="inet 192.168.1.100/24 addm nfse0 addm ue0"
defaultrouter="192.168.1.1"
#Gateway enable
gateway_enable="YES"
#NAT disabled
#natd_enable="YES"
#natd_interface="nfe0"
#IPFW firewall
firewall_enable="YES"
firewall_type="/etc/firewall.conf"
...
```


----------



## Labaman (Aug 17, 2014)

So today, climbed into the settings linux-gateway (which allows two subnets through it without any additional settings on the modem).
Drew attention to its routing table:

```
netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.0     0.0.0.0         255.255.255.252 U         0 0          0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
```
And I have now:

```
netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0       87   nfe0
127.0.0.1          link#2             UH          0        0    lo0
192.168.0.0/24     link#3             U           0      306    ue0
192.168.0.1        link#3             UHS         0        0    lo0
192.168.1.0/24     link#1             U           0       92   nfe0
192.168.1.2        link#1             UHS         0        0    lo0
```

This automatic settings FreeBSD.
As I understand it, to see both the subnet through the gateway, I need anywhere in the table was default gateway (aka 0.0.0.0 in Linux).
Now a specific question - how me do it in FreeBSD?


----------



## kpa (Aug 17, 2014)

Could you draw a picture of what is where in your system and how exactly the addresses are assigned to interfaces and what devices are connected to what other devices? Your explanation so far makes absolutely no sense to me. If your system was a standard modem <--> FreeBSD router <--> LAN net there would nothing else to do but tell the modem that there is a another network (192.168.0.0/24) that can be reached trough the "WAN" interface of the FreeBSD router and that would be done by adding a static route on the modem's routing table.


----------



## Labaman (Aug 17, 2014)

Here is...


----------



## obsigna (Aug 17, 2014)

Labaman said:
			
		

> As I understand it, to see both the subnet through the gateway, I need anywhere in the table was defaut gateway (aka 0.0.0.0 in Linux). Now a specific question - how me do it in freebsd?



default in the destination field on FreeBSD is a shortcut for 0.0.0.0 netmask 0.0.0.0. So the default entry in the FreeBSD routing table matches exactly that of the Linux gateway. The differences are in the netmasks of the 192.168.1.0 network. On the FreeBSD box you got /24 i.e. 255.255.255.0 and on the Linux box you have /30 i.e. 255.255.255.252.


----------



## obsigna (Aug 17, 2014)

kpa said:
			
		

> Could you draw a picture of what is where in your system and how exactly the addresses are assigned to interfaces and what devices are connected to what other devices? Your explanation so far makes absolutely no sense to me. If your system was a standard modem <--> FreeBSD router <--> LAN net there would nothing else to do but tell the modem that there is a another network (192.168.0.0/24) that can be reached trough the "WAN" interface of the FreeBSD router and that would be done by adding a static route on the modem's routing table.



This problem is a quite typical one. Imagine, your ISP would force you to use his router and at the same time does not give you access to the box. For example, the netizens in Germany are very upset, because the regulatory authority is going to not forbid (i.e. allow) this. People who want to keep their control over their home network need solutions for this.


----------



## usdmatt (Aug 18, 2014)

"Double NAT" isn't exactly the nicest setup although there's no reason it shouldn't work. Are you sure you can't just replace the original modem/router with your own equipment? (Seeing as it has a private address, I assume it's running NAT itself, and so is actually a modem+router, not just a modem). In the UK this definitely isn't a typical problem. Some ISP-provided routers may have few configuration options, but I've not come across a service yet where you can't just get login details from the ISP and use your own.

To be honest, I can't see an obvious reason why it isn't working already with the config you have. If you are trying to access the modem (say a web interface) from a LAN PC, the following should happen:


Browser on 192.168.0.3 attempts to open a connection to 192.168.1.1 port 80 (source port will be something random, like 10000)
(src: 192.168.0.3:10000, dst: 192.168.1.1:80)
This isn't on the local network, so the workstation sends the packet to its gateway 192.168.0.1
FreeBSD machine should route the packet to 192.168.1.1, but will NAT the source address (new random source port, eg 20000)
(src: 192.168.1.100:20000, dst: 192.168.1.1:80)
Packet sent out to modem, which should respond back
Response - (src: 192.168.1.1:80, dst: 192.168.1.100:20000)
Server should undo the NAT, changing the destination from 192.168.1.100:20000 back to 192.168.0.3:10000
Packet should go back to client with exact same details as the first packet (just reversed), as if the NAT never happened
(src: 192.168.1.1:80, dst 192.168.0.3:10000)

Looking at tcpdump() output on the FreeBSD machine would probably be the most likely way of finding out what's happening.
What are you actually trying to do with the modem from the LAN computers? If you don't have access to it, I don't see why you'd need to be able to reach it from the LAN.

If you're trying to allow the modem to create connections to LAN machines, then it's a different problem entirely.


----------



## obsigna (Aug 18, 2014)

For some certain reasons I am also interested in solutions for the issue of the OP, and therefore I was investigating the various possibilities with a test setup which is quite similar to the one that the OP @Labaman designed for his response above. Here comes a summary of the findings and here are the options, that were all working in my test setup:

*1. Proxy on the gateway*
The innermost internal clients have at least access to certain services on the internet. The OP told us already that squid is working as a transparent proxy on his gateway, and yes, on FreeBSD 10 ipfw fwd works without building a custom kernel with options IPFIREWALL_FORWARD. This is of course not exactly the solution for the issue, however, in some usage cases a proxy might be an adequate option, and I mention it for the sake of completeness.

*2. Manipulating the Routing Table* 
This was suggested already in some posts. However, the OP seemed to understand it wrong. The routing table of the modem+router has to be manipulated or at least a DMZ option has to be configured, and not the routing table of the FreeBSD gateway. If it is for any reasons not possible to manipulate the modem+router, then this is NOT an option (of course). And even if it would be possible to manipulate the modem+router, then IMHO, it would be better to put it into bridge mode and let the FreeBSD gateway talk to the WAN directly, and so, all that hassle is avoided. Anyway, again for the sake of completeness, the correct command (*BSD style) in the actual case for adding a route to the modem/router would be: `route add -net 192.168.0.0/24 192.168.1.100`. I verified this in my test setup, and this would really solve the issue, again, provided the modem+router is configurable in that way.

*3. Bridging on the FreeBSD gateway*
I suggested this in a previous post. The advantage is, that this actually works without the need to change anything in the modem+router. The "disadvantage" is, that there is no innermost internal network. Nonetheless, all packets from/to the internet are forced to pass  the firewall, and therefore this solution with proper FW rules would NOT impose higher risks on the internal clients compared to the desired setup of the OP, having the clients in a separated internal network.   

*4. Double NAT*
As @usbmat already stated, "'Double NAT' isn't exactly the nicest setup...", however, it would bring to the OP what he was looking for. The point is, it does not work that simply as expected. The whole last sunday, I tried all possible and impossible ways to get it to work with my test setup employing ipfw+natd and ipfw+in-kernel-nat, to no avail. I told to myself, this must work, I saw it already working. VirtualBox defaults to NAT, and with the Host being on NAT this is 'Double NAT'. Mac OS X offers the so called 'Internet Sharing' among network devices which is actually a combo of  DHCP/DNS/NAT and again if the Mac is on NAT, the whole setup is 'Double NAT'. Well, today, I examined what Mac OS X actually does for setting up its NAT for 'Internet Sharing' of the WLAN (en1) to LAN-Devices on (en0). The surprise was, that for the separated internal network it creates a bridge with the only one member en0 and NAT's this via en1. I reproduced this on the FreeBSD gateway, and surprisingly, I got 'Double NAT' working finally:

In file /etc/rc.conf

```
...
# Interface Setup
cloned_interfaces="bridge0"
ifconfig_ue0="up -tso"
ifconfig_bridge0="inet 192.168.0.1/24 addm ue0 description LAN"
ifconfig_nfe0="inet 192.168.1.100/24 -tso description WAN"
defaultrouter="192.168.1.1"

# Gateway/Firewall/NAT
gateway_enable="YES"
firewall_enable="YES"
firewall_nat_enable="YES"
firewall_script="/etc/ipfw.conf"
```
In file /etc/ipfw.conf

```
#!/bin/sh

for iface in `/sbin/ifconfig -l` ; do
   desc=`/sbin/ifconfig $iface | /usr/bin/sed -n '/.description: /{s///;s/ .*//;p;}'`
   if [ "$desc" == "LAN" ] ; then
      LAN=$iface
   elif [ "$desc" == "WAN" ] ; then
      WAN=$iface
   fi
done

/sbin/ipfw -q flush
/sbin/ipfw -q nat 1 config if $WAN reset
/sbin/ipfw -q add 100 nat 1 ip from any to any via $WAN
/sbin/ipfw -q add 65534 allow ip from any to any
```
Note: Without putting ue0 into the bridge, 'Double NAT' DOES NOT WORK. And, don't blame me for this, the bridge in this NAT setup is not my invention, I copied this from Mac OS X 'Internet Sharing'.


----------



## aupanner (Aug 18, 2014)

I just did this for my home LAN.  I use PF so the configuration is different but perhaps it will be enough to help you.  

How to allow your local net to talk to your DSL modem through the gateway firewall.

1. On the external interface create an alias on the same subnet as the DSL modem.

```
ifconfig_nfe0="inet MY.STATIC.IP.ADDR netmask 255.255.255.0"
ifconfig_nfe0_alias0="inet 192.168.1.254 netmask 255.255.255.0"
```

2. In the PF firewall configuration apply a NAT translation for traffic to the DSL modem via the alias, and the normal NAT translation to the outside world.

```
EXT_IF="nfe0"
INT_IF="ue0"
LOCALNET=$INT_IF:network
DSLMODEM="192.168.1.1 port 80"
INT_HOST_DSL="192.168.1.254"
nat on $EXT_IF inet from $LOCALNET to $DSLMODEM -> $INT_HOST_DSL
nat on $EXT_IF inet from $LOCALNET to any -> ($EXT_IF:0)
```

3. In the PF firewall configuration add a rule to allow traffic from the router to the DSL modem.

```
pass out quick on $EXT_IF inet proto tcp from $INT_HOST_DSL to $DSLMODEM
```

The first NAT takes traffic from the localnet and duplicates it on nfe0_alias0.  The second NAT uses $EXT_IF:0; :0 means to use only the primary IP address and to ignore any aliases.


----------



## Labaman (Aug 20, 2014)

It was all about DLINK 2650U. Something was wrong with the routing in this model. Several complaints on the forums is only confirmed. Set old TPLINK 861 and everything went in both directions. Thank you all for your assistance.


----------



## Jimmy (Sep 12, 2014)

You will not need NAT on the FreeBSD box at all if you can add the route onto your modem.  Add a route to 192.168.0.0/24 via 192.168.1.100 on the modem and ensure you have the sysctl() net.inet.ip.forwarding set to 1 on the FreeBSD box and you are all set.


----------

