# Setting up router -- stalled



## jurrie (Feb 10, 2009)

Hi,

today I got my fiber connection replacing my dsl line. I have always used a nice Linksys DSL modem / router for my dsl line, but it won't work with the fiber connection (it lacks an ethernet uplink port). I decided to use my BSD home server to be my router for the time being.

I have used this guide to set up my server as a gateway and configured it to use NAT as well. Things work fantastic and I can do all my stuff using my desktop perfectly.

The problem, however, lies with the server. When I try to scp from the server, or serve a page using apache, the connection stalls. Uploading using my desktop works fine, but on the server it uploads a few KB (increments of 64) and then stalls for a while. 

Being somewhat of a network nitwit (I can set up / maintain basic things, but when things get advanced, I drop out), I was hoping someone here could direct me to a fix.

To illustrate:

(Fiber modem)----<nfe0>(BSD Server)<em0>----(Switch)----(Desktop)

nfe0 is set to use DHCP (to work with my fiber connection) and em0 is set to 192.168.1.2 (legacy), with subnet mask 255.255.255.0, using my ISP's DNS.

My desktop is set to 192.168.1.3, subnet 255.255.255.0, also using my ISP's DNS.

The console prints no errors whatsoever. Nothing else I could say to provide more info comes to mind, so if I left something out which is required to answer my question, please let me know.

Thanks very much in advance.


----------



## honk (Feb 10, 2009)

jurrie said:
			
		

> When I try to scp from the server, or serve a page using apache, the connection stalls. Uploading using my desktop works fine, but on the server it uploads a few KB (increments of 64) and then stalls for a while.



You mean serving for internet clients? It works for internal clients? Just an idea, I've an apache reachable from the internet and my provider drops connections after transfer of about 5MB in upstream direction if my servers port is 80 or 443. Moving my server services to a high-port helps, the conections from the clients don't break after 5MB.

I propose to use tcpdump to debug this.


----------



## jurrie (Feb 11, 2009)

It's not just serving to clients, it's also using scp myself. I had a a friend connect to http://<my ip>/ where I had a phpinfo() in an index.php. He did see the output, though it took ages for the complete page to transfer. When I put a file there, he got it, verrrrry sloooowly. When scp'ing from the server to my web host, a bit was uploaded and then the -stalled- message appeared.

It's not an ISP issue, since I can use scp from my desktop to copy to my web host and it flies at around 650k/s. Speedtest.net also gives me around 18Mbit in upstream (all this using the server as gateway/router).

I think the server is somehow confused as to which network card it should use to upload to the outside network. But that's just my wild nitwit guess.


----------



## honk (Feb 11, 2009)

Not sure if I understand you exactly...
When your internal desktop transfers data >>through<< your FBSD gateway/webserver to your (any) internet webhost, it's ok? And it's also ok if you transfer from your internal desktop to your FBSD gateway/webserver (within the local net)?? But from outside(internet) to your FBSD gateway/webserver its slow? If so it might be an MTU issue. Please post the output of

ifconfig -a
netstat -rn
netstat -i

And of course your firewall and NAT configuration. You might also check how your firewall handles icmp.

Try to figure out the MTU on your fiber connection by pinging the first hop like this:

# ping -s 1472 -D 1.2.3.4
PING 1.2.3.4 (1.2.3.4): 1472 data bytes
1480 bytes from 1.2.3.4: icmp_seq=0 ttl=62 time=10.547 ms
1480 bytes from 1.2.3.4: icmp_seq=1 ttl=62 time=10.464 ms
1480 bytes from 1.2.3.4: icmp_seq=2 ttl=62 time=10.158 ms
...
# ping -s 1473 -D 1.2.3.4
PING 1.2.3.4 (1.2.3.4): 1473 data bytes
ping: sendto: Message too long


Try different values for the packet size -D until you find the largest value before your don't get an answer back or a "Message too long".


----------



## jurrie (Feb 12, 2009)

Thanks for replying again. I'll try to explain it once more before I will post the information you requested.

* Desktop -> uses server as gateway -> no problems whatsoever
* Server -> login via terminal -> scp to the internet -> stalled
* Remote user -> connects to my home IP via http -> receives data very slow

About the pings. I tried several times. "ping -s 1472 -D" works like a charm. Any number above the 1472 results in 100% packet loss. On rare occasions I do receive a ping back, but the lowest packet loss I've seen is 86.4%.

Output of the commands:


```
# ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0e:0c:b6:0f:85
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:1d:7d:d7:58:16
        inet 94.229.49.55 netmask 0xfffffe00 broadcast 94.229.49.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
```


```
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            94.229.48.1        UGS         0     3278   nfe0
94.229.48.0/23     link#2             UC          0        0   nfe0
94.229.48.1        00:0e:00:00:00:01  UHLW        2        0   nfe0    469
94.229.48.23       00:0e:00:00:00:01  UHLW        1        4   nfe0   1073
94.229.49.55       00:1d:7d:d7:58:16  UHLW        1       40    lo0
94.229.49.255      ff:ff:ff:ff:ff:ff  UHLWb       1       86   nfe0
127.0.0.1          127.0.0.1          UH          0       40    lo0
192.168.1.0/24     link#1             UC          0        0    em0
192.168.1.2        00:0e:0c:b6:0f:85  UHLW        1       16    lo0
192.168.1.3        00:18:f3:74:8c:a3  UHLW        1    60235    em0    825
192.168.1.255      ff:ff:ff:ff:ff:ff  UHLWb       1       45    em0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       link#4                        UHL         lo0
ff01:4::/32                       fe80::1%lo0                   UC          lo0
ff02::%lo0/32                     fe80::1%lo0                   UC          lo0
```


```
# netstat -i
Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs  Coll
em0    1500 <Link#1>      00:0e:0c:b6:0f:85    85568     0    60840     0     0
em0    1500 192.168.1.0   merry                84819     -    59963     -     -
nfe0   1500 <Link#2>      00:1d:7d:d7:58:16     1585     0     1803     0     0
nfe0   1500 94.229.48.0/2 5ee53137.ftth.con      680     -     1838     -     -
plip0  1500 <Link#3>                               0     0        0     0     0
lo0   16384 <Link#4>                              96     0       96     0     0
lo0   16384 fe80:4::1     fe80:4::1                0     -        0     -     -
lo0   16384 localhost     ::1                      0     -        0     -     -
lo0   16384 your-net      localhost               40     -       40     -     -
```

Perhaps these outputs point to an obvious mistake in my settings, but they mean nothing to me.

My firewall settings (I hope I did it properly):


```
# ipfw -a l
ipfw: DEPRECATED: 'l' matched 'list' as a sub-string
00050   3300   2109884 divert 8668 ip4 from any to any via nfe0
00100    192     15274 allow ip from any to any via lo0
00200      0         0 deny ip from any to 127.0.0.0/8
00300      0         0 deny ip from 127.0.0.0/8 to any
65000 151731 140144932 allow ip from any to any
65535      4       796 deny ip from any to any
```

And natd settings (only a port forward for my torrent client on my desktop):


```
# cat /etc/natd.conf
port 8668
interface nfe0
redirect_port tcp 192.168.1.3:6781 6781
redirect_port udp 192.168.1.3:6781 6781
```

I hope this clears things up. Thanks in advance for your help and effort.


----------



## jurrie (Feb 16, 2009)

Okay, the problem is solved now. Thanks to honk for helping me via email!

The evildoer was the nvidia nic (nfe0). Several topics on the web pointed to it performing badly on several occasions. It was suggested to me that I should turn off tcp checksum offloading and tcp segmentation offloading on the nvidia nic, using "ifconfig nfe0 -txcsum -rxcsum -tso4". This, however, did not work for me (though it apparantly worked for others).

Then we tried the simplest of all solutions: switch the internal and external nics. Now I use my onboard nvidia nic for the internal network and my trusty intel pci nic for the external network. Things work like a charm now! Full blazing upload speed and no noticable problems on my internal network, now being served by the nvidia one.

I'm a happy camper now. Thanks again, honk!


----------



## RaZwaN (Sep 2, 2009)

i have the same problem. 
can you help me too?
i used the txcsum -rxcsum -tso4 and it works for now but i'm wondering if it is another solution for this problem.

thanks!


----------



## jurrie (Sep 10, 2009)

It's been a while, but I recall the nvidia nic having the problems when it's not operating on 1Gbit. Before I switched the nics, the nvidia one was communicating with my fibre line (35Mbit) and my intel nic with my internal network (1Gbit). After switching the nvidia nic to the 1Gbit network, it stopped acting up. I'm afraid that's all I can say about that.


----------

