# FreeBSD 7.0 routing between 2 ISPs



## mrowcp (Nov 14, 2012)

Hello,
I have 1 FreeBSD server with 3 NICS:

le1: 192.168.100.1 (Local Network)
le2: 100.90.80.71 (ISP1) (pppoe)
le3: 100.90.80.61 (ISP2)

[cmd=]netstat -rn[/cmd]

```
def gateway 100.90.80.70
```
Now all traffic is redirected via ISP1.
If I add: 
`# route add -net 97.74.27.0/24 100.90.80.60`
and 
`# traceroute google.com` all is fine:

```
traceroute to google.com (173.194.70.101), 64 hops max, 40 byte packets
 1  100.90.80.60 (100.90.80.60)  5.479 ms  1.452 ms  3.734 ms
 2  .....
 3  .....
```
But I can't trace google.com from PC (192.168.100.2) in local network.
At the moment I'm using ipfw. Is there something that I must add in ipfw? Like divert? Because if I add:

```
ipfw add 1 divert all from 192.168.100.2 to any out via le3
ipfw add 1 divert all from any to 100.90.80.61 in via le3
```


```
sockstat |grep 8668
root     natd       842   4  div4   *:8668                *:*

COMMAND PID USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
natd    842 root    4u  IPv4 0xc67b0000      0t0 DIVERT *:8668[/B]
```
Client PC can't "see" 1st hop:


```
tracert google.com

Tracing route to google.com [173.194.70.138]
over a maximum of 30 hops:
 1 * * * timeout
```

Am I missing something?
Thanks

P.S. Thanks for the mod.Post look better now


----------



## mamalos (Nov 14, 2012)

If I were you, I'd first turn off my firewall in order to see if my routing works OK. Then, I'd make sure that routing is enabled on my host by running:


```
# sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1
```

And verify that the output is indeed 1. If not, I'd configure it either through setting gateway_enable="YES" in /etc/rc.conf, or by adding net.inet.ip.forwarding=1 in /etc/sysctl.conf. If that worked, I'd configure my firewall, which I don't think that it would need any diverting.

Of course, if you want to use a failover mechanism with your two ISP's, then your configuration will have to be changed accordingly.

PS. Out of curiosity, what are the netmasks on your le2 and le3 IP configurations?


----------



## mrowcp (Nov 15, 2012)

mamalos said:
			
		

> If I were you, I'd first turn off my firewall in order to see if my routing works OK. Then, I'd make sure that routing is enabled on my host by running:
> 
> 
> ```
> ...



First thanks for you replay.Here is what I've test:

`# ipfw delete 1`

```
ipfw add 1 allow all from any to any
```

Still can't trace google.com from user PC.Forwarding is *ON*, because it's work for another interface - le2

`# sysctl -a |grep net.inet.ip.forwarding`

```
net.inet.ip.forwarding: 1
```

Mask's are:


```
le2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
        inet 100.90.80.71 --> 100.90.80.70 netmask 0xffffffff
        Opened by PID 34493

le3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:50:56:9a:65:1b
        inet 100.90.80.61 netmask 0xff000000 broadcast 255.255.255.252
        media: Ethernet autoselect
        status: active
```

and again traceroute from router:


`# traceroute google.com`

```
traceroute: Warning: google.com has multiple addresses; using 173.194.70.139
traceroute to google.com (173.194.70.139), 64 hops max, 40 byte packets
 1  100.90.80.60-rev.host.net (100.90.80.60)  1.441 ms  1.657 ms  1.496 ms
 2  100.90.80.59-rev.host.net (100.90.80.59)  3.673 ms  3.722 ms  4.697 ms
```

traceroute to yahoo.com

`# traceroute yahoo.com`

```
traceroute: Warning: yahoo.com has multiple addresses; using 98.139.183.24
traceroute to yahoo.com (98.139.183.24), 64 hops max, 40 byte packets
 1  100.90.80.70 (100.90.80.70)  1.542 ms  1.259 ms  1.435 ms
 2  100.90.80.69 (100.90.80.69)  19.641 ms  5.531 ms  3.033 ms
```


----------



## mamalos (Nov 15, 2012)

Run:

`# /etc/rc.d/ipfw stop`

to stop your firewall. Then *ping* some IPv4 address, like 8.8.8.8. 

When you want to check your network, check each component/layer one by one. First *ping* your router to see that your local network is reachable, then *ping 8.8.8.8* to see if the router is forwarding your packets correctly, then *ping www.google.com* to check that your DNS servers on your client are configured correctly, then run *traceroute www.google.com* to see if you're following the right route. After all these have worked, *then* turn on your firewall and configure it accordingly.


----------



## mamalos (Nov 15, 2012)

mrowcp said:
			
		

> [CMD="traceroute"]traceroute yahoo.com[/CMD]



Ah,

and when you're inserting a command, type # or $ in the pop-up, not the command name, because it's prefixed to the command you're giving afterwards and it shows up twice...


----------



## mrowcp (Nov 16, 2012)

`# /etc/rc.d/ipfw stop`

I don't know why, but I thing that this script don't work (ipfw stop)

`# ping facebook.com`

```
PING facebook.com (69.171.237.16): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
```

`# route add -net 209.85.148.0/24 100.90.80.60`

```
add net 209.85.148.0: gateway 100.90.80.60
```

`# ping -S 100.90.80.61 google.com`

```
PING google.com (209.85.148.113) from 95.158.128.154: 56 data bytes
64 bytes from 209.85.148.113: icmp_seq=0 ttl=54 time=38.624 ms
64 bytes from 209.85.148.113: icmp_seq=1 ttl=54 time=33.784 ms
64 bytes from 209.85.148.113: icmp_seq=2 ttl=54 time=38.486 ms
64 bytes from 209.85.148.113: icmp_seq=3 ttl=54 time=33.933 ms
```


`# traceroute google.com`

```
traceroute: Warning: google.com has multiple addresses; using 209.85.148.101
traceroute to google.com (209.85.148.101), 64 hops max, 40 byte packets
 1  153.128.158.95 (100.90.80.60)  22.531 ms  1.460 ms  1.316 ms
 2  254.128.158.95 (100.90.80.59)  3.667 ms  25.043 ms  3.981 ms
```

But when I do it from Windows XP:

`# C:\>ping google.com`


```
Pinging google.com [209.85.148.113] with 32 bytes of data:

Request timed out.
Request timed out.
```

`# C:\>tracert google.com`


```
Tracing route to google.com [209.85.148.101]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.100.1
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
```

I'm sure the problem is somewhere in the routing, but can not figure out where.

*Something New that I test:*

I have another freebsd server with le1: 192.168.100.20 and free le2 slot.If I put ISP2 cable in le2 NIC and up interface:

`# ifconfig le2 100.90.80.61 255.255.255.252`

`# route add -net 209.85.148.0/24 100.90.80.60`

```
add net 209.85.148.0: gateway 100.90.80.60
```

than add GW 92.168.100.20 to the WindowsXP station, still I do not have ping/traceroute to google.com
*BUT* if add divert:

`# ipfw add 1 divert 8668 all from 192.168.100.2 to any out via le3`
`# ipfw add 1 divert 8668 all from any to 100.90.80.61 in via le3`

, now I can ping/trace google.com and yahoo.com


`# C:\>tracert google.com`


```
Tracing route to google.com [209.85.148.101]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.100.20
  2    <3 ms    <2 ms    <2 ms  100.90.80.60
  3    <13 ms   <9 ms   <19 ms .......
  4  ...................................
```

`# C:\>tracert yahoo.com`


```
Tracing route to yahoo.com [98.138.253.109]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.100.20
  2    <3 ms    <2 ms    <2 ms  100.90.80.60
  3    <13 ms   <9 ms   <19 ms .......
  4  ...................................
```

At the end I add route to yahoo.com:

`# route add -net 98.138.253.0/24 192.168.100.1`

and traffic is redirected correctly:

`# C:\>tracert yahoo.com`


```
Tracing route to yahoo.com [98.138.253.109]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.100.20
  2    <1 ms    <1 ms    <1 ms  192.168.100.1
  3    <13 ms   <9 ms   <19 ms  
  4  ...................................
```


*Is there any way for route debuging?I'm sure that I miss something in route/NAT between this 3 NICs (server1)*



			
				mamalos said:
			
		

> Ah,
> 
> and when you're inserting a command, type # or $ in the pop-up, not the command name, because it's prefixed to the command you're giving afterwards and it shows up twice...



Thanks, fixed


----------



## SirDice (Nov 16, 2012)

Keep in mind FreeBSD 7.0 is End-of-Life since April 2009. Please upgrade to 7.4 or 8.3.

http://www.freebsd.org/security/#unsup


----------



## mix_room (Nov 16, 2012)

Are you trying to achieve a high availability solution, where the loss of one link doesn't affect the whole connection, or are you wanting to use different ISPs to reach different parts of the internet? 

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html might be able to help you.


----------



## mamalos (Nov 16, 2012)

mrowcp, I've tried to explain to you that you need to try things one step at a time, but I think you're not getting this. The first problem I can now see from your initial post is that I see no NAT configuration whatsoever on your le1 or le2 interfaces. Setup NAT and make sure that your default policy for ipfw is allow(8)!! Both steps are paramount. If not set correctly, nothing will work. When this is done, on the FreeBSD box run:


```
# route add default 100.90.80.70
```

On your Windows box run:

```
> ping 192.168.100.1
> ping 8.8.8.8
```

and post your results. If you wish to be a bit more informative, show us what the FreeBSD box shows when running these two commands:


```
# tcpdump -neti le1 host 192.168.100.winboxIP
# tcpdump -neti le2
```


----------



## mrowcp (Nov 17, 2012)

@mix_room I want to use different ISPs to reach different parts of the internet 



			
				mamalos said:
			
		

> Run:
> 
> `# /etc/rc.d/ipfw stop`
> 
> ...



May be I miss something, but I will try to test your steps again.



> # /etc/rc.d/ipfw stop



This script do not work for me.Don't know why.May be there is random paths for some bins or... something else.I'm sure because:

`# /etc/rc.d/ipfw stop`
`# ping facebook.com`

```
PING facebook.com (66.220.152.16): 56 data bytes
ping: sendto: Permission denied
^C
--- facebook.com ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
```

`# ipfw list |grep 66.220`


> 00007 deny ip from any to 66.220.0.0/16



So, to make my tests I delete line 1 (`# ipfw delete 1`) and add allow all from any to any.

Next step is checking layers.How to check 2 dif. paths (ISPs) with this commands which you give it to me?I have ping to 8.8.8.8 and ping to google.com through ISP1 (pppoe).NAT working:

```
root    960  6.1  0.1  4984  2748  ??  Ss   12:42AM  42:41.17 ppp -quiet -background -nat pppoe
```

*BUT, what I must do for NATing  my 2nd ISP traffic?Does I need another NAT .conf and run another process or I must edit this one (proc ID 960)*

Here is my ppp.conf NAT file:

`# cat /etc/ppp/ppp.conf`

```
default:
    set log Phase Chat LCP IPCP CCP tun command
    nat enable yes
    nat same_ports yes
    nat use_sockets yes

    nat port tcp 192.168.100.225:1723 1723    #VPN-DOMAIN dostup ok: ot vsjakude
    nat port udp 192.168.100.225:1723 1723    #VPN-DOMAIN dostup ok: ot vsjakude
    nat port tcp 192.168.100.245:3306 45330    #DB_EFIR dostup ok:kostievo
    ...........
    ...........
    set redial 15 28800
    set reconnect 15 28800
pppoe:

    set device PPPoE:le0:
    set mru 1492
    set mtu 1492
    set speed sync
    enable lqr
    set lqrperiod 5
    set cd 5
    set dial
    set login
    set timeout 0
    set authname pppoe-freshlog
    set authkey alabala
    set ifaddr 100.90.80.71 100.90.80.70 255.255.255.255
    add default HISADDR
    enable dns
```


----------



## mrowcp (Nov 18, 2012)

I found out where the problem comes from but still don't know how to fix it. ... 
I think the problems comes from 1 pppoe interface (used for NAT) and 2nd one which also used wireless antenna, but connection isn't made via pppoe (have router board, cable from wireless is connected on port 2 and cable from port 4 is connected to my server).

So the main question is: is it possible to route at the same time from both interfaces - LAN (ISP2) and pppoe (ISP1).
Here is what I find:

http://forum.ru-board.com/topic.cgi?forum=8&topic=25436

,but don't know how to modify it for my situation.


----------

