# Is routing 1000Mbs CPU intensive task?



## graudeejs (Jan 13, 2010)

I'm wondering how CPU intensive 1000Mbs routing would be.

This Friday I will get my new server, I could get 1000Mbs connection, but I will need to route it to other PC's as well, because currently I can't afford to pay for 2 separate internet connections.
But I can afford to buy another IP and use my new server as router as well.

I've never done routing so bare with me.

Thanks in advance.

P.S. I know home PC's would never use 1000Mbs but I want to know answer in general


----------



## Alt (Jan 13, 2010)

If you tune network sysctls, turn on polling, use good network cards and their abilities (for example intel cards are great for FreeBSD), enable fastforwarding, do not NAT anything, do not use long ipfw rules (10-20k or more rules) then... then you can route far more than 1000mbs... 
There is way more important how much packet per second (pps) is goin on network card, not packet length. Of course, if we have network load 1000mbs and average packet is 200 bytes, there will be much more CPU eating case than avg packet 1500 bytes


----------



## graudeejs (Jan 13, 2010)

So what do you think if 2x PC's would run torrents at 300Mbs?


----------



## Alt (Jan 13, 2010)

I think if you tune router it must be less than 10-20% cpu, on default config it may be far more


----------



## graudeejs (Jan 13, 2010)

Hmm, could using this
http://shopper.cnet.com/adapters-nics/hp-nc150t-pci-4/4014-3380_9-31135966.html#info-5
help decreasing load?

I might get one used for 82.61 USD


----------



## Alt (Jan 13, 2010)

Depends which card is alternative =) Realtek might be cheapest cards but they mostly dont support hardware things like checksum processing and/or have small own buffer. I think card you suggested is not bad, but i'd prefer intel chip. Also notice you can face bottleneck on network speed when you use PCI slot (and surely if you have 4 ports on 1 slot)


----------



## phoenix (Jan 13, 2010)

Our two core "routers" are FreeBSD boxes with the following:

Asus M2N motherboard
2 GB of RAM
dual-core AMD Opteron 1281 CPU @ 2.6 GHz
80 GB SATA drive
Intel Pro/1000MT quad-port gigabit NIC (PCIe)
One does routing across 12 vlans, with a 100 Mbps uplink to the Internet, and a gigabit link to the fibre switch that connects the board office with all the in-town secondary schools.  All Internet traffic from all the secondary schools goes through this box.  It's also doing dummynet traffic shaping to guarantee each school a minimum of 10 Mbps (up to 90 Mbps if no one else is using the line).

The other is a firewall for the board office, managing traffic for all the district server, and the board office LAN, and tech LAN (which is used for testing servers).

During the day, the schools use an almost constant 30 Mbps with spikes to 60-70 Mbps.

During the backups run each night, we peg the Internet line at almost the full 100 Mbps.

During the backup sync between backup servers, we transfer around 600-800 Mbps of traffic for several hours between the board office and another site, via the fibre link.

CPU usage on these two boxes rarely goes above 20%, and most of that is in the natd process (not using in-kernel NAT with IPFW yet).

The router box rarely goes above 10% CPU, even with all the traffic shaping.  Shuffling packets between interfaces doesn't take a lot of CPU.  Mangling packets via NAT and fwd and what not, uses more CPU.

Obviously, if you use crappy NICs that don't do offloading or hardware vLAN tagging and such, then your CPU usage will go way up.

We don't use polling on any of our firewalls or routers.  Most of our firewalls run FreeBSD 6.x, our routers run FreeBSD 7.x.  Running FreeBSD 8.x should give better performance.


----------



## DutchDaemon (Jan 13, 2010)

My little observation:

FreeBSD bridge, two fiber cards, currently 64 Mbit/sec down, 20 Mbit/sec up flowing through it (crossing both interfaces). The dual-core CPU sits at a comfortable 0.1 - 0.2, and an experiment with powerd shows it will happily throttle the CPU back to 249 (its lowest possible speedstep value). Meanwhile, it has 22M RAM active  

So, not sure how that would look in a 1 Gbit/sec scenario (haven't seen more than about 150 Mbit/sec traffic, when the load was around 0.6 - 0.7).

The server does have PF enabled and is filtering traffic and keeping state (sometimes 50,000+ states), but does not perform NAT. Under these circumstances, CPU (or even all-over) impact is negligible.


----------



## graudeejs (Jan 13, 2010)

Thanks for sharing info & experience.
You know you guys rock, don't you 

as DutchDaemon suggest in this thread, I will use switch to avoid routing on server


----------



## J65nko (Jan 13, 2010)

OpenBSD pf hacker Henning Brauer in a recent mail message on the OpenBSD misc list, mentioned that at this moment pf in OpenBSD current can handle 420Mbit/s.



> I really like the 275 -> 420MBit/s change for 4.6 -> current with pf.


----------



## DutchDaemon (Jan 13, 2010)

I wonder what the limiting factor is? It's a rather exact value, so it sounds like it's in the process, not (let's say) in the 'hardware' around it.


----------



## J65nko (Jan 13, 2010)

DutchDaemon

I also wonder, but I think you are right. It is in the process, and I suspect also the structure of OpenBSD. E.g. Henning and others have stated before that multiple CPU's are just a waste for a pf router, because switching/locking is still rather costly on OpenBSD.

What I found strange is his usage of "bits per seconds" terminology, while before, the pf hackers, insisted on "packets per seconds" as the correct way to discuss and measure pf performance.


----------



## ecazamir (Jun 23, 2010)

My numbers:
Internet connection: 200 mbit
Hardware: core 2 Quad at 2.3 GHz, 2G RAM, Intel and Broadcom GbE NICs.
On a FreeBSD router/PPPoE server with 250 active PPPoE sessions and approx. 600 ipfw counters, at 60kpps (approx. 100mbps, total forwarding rate, approx. 15kpps per interface/per direction) the network starts dropping packets. I've tried enabling DEVICE_POLLING, but the PPS rate didn't rise above 60k.
If I set net.inet.ip.process_options to zero, the maximum PPS increases to at least 80 k PPS, and network throughput rises to at least 150 Mbps.


----------



## mix_room (Jun 24, 2010)

killasmurf86 said:
			
		

> This Friday I will get my new server, I could get 1000Mbs connection, but I will need to route it to other PC's as well, because currently I can't afford to pay for 2 separate internet connections.
> But I can afford to buy another IP and use my new server as router as well.



Can you get a 1Gbps connection at home, or is it to a collocation facility? If you can get it at home I am impressed.


----------



## LypsylateX (Jun 24, 2010)

It's not that rare to be able to buy a 1Gbit/s connection for not that much money in some countries. I guess it's a combination of competition and how the state/city/whatever have seen upon investing in infrastructure. Some strange mix of capitalism and socialism seems to do miracles, atleast here.

For me a 1000/100Mbit connection is quite expensive (relativly) sinse I'll have to pay SEK999 per month for it (= 100â‚¬ or $122). That's a single operator who has done a FTTH-solution. (But it has excellent peering and they are one of 14 tier1 networks in the world, so it's nice anyhow. I've singned up for a 100/100 connection.)

My parents on the other hand get a 1000/100Mbit connection for around SEK400 a month. (= $49 or 40â‚¬) They have a FTTH solution built by the city and the operators lease capacity.

There are other countries that have even better conditions on their broadband connections, but there are alot of countries where these connections and prices seems like dreams.

I've heard that Japan has great prices. The US and GB seems to be awful when it comes to broadband, but I might be wrong.


----------



## loop (Jun 25, 2010)

LypsylateX said:
			
		

> I've heard that Japan has great prices. The US and GB seems to be awful when it comes to broadband, but I might be wrong.



You should try living in the internet backwater that is Australia ....


----------



## mix_room (Jun 25, 2010)

LypsylateX said:
			
		

> For me a 1000/100Mbit connection is quite expensive (relativly) sinse I'll have to pay SEK999 per month for it (= 100â‚¬ or $122).



~100EUR for a 1000/100 sounds like a realistic price to me, but perhaps that is because I am used to slightly higher rates (~60 EUR 100/7, which also happens to be the fastest rate available, or around 50EUR for the standard ADSL connection (12/2).


----------



## LypsylateX (Jun 25, 2010)

loop: You have the same problem as Iceland. If you want you traffic to reach anyone else you have to go thru alot of water. That is expensive for the operators.

mix_room: It's all about what we're used to.  What technique are used for the 100/7 and what others are available?

Here in Sweden ethernet (up to 1000/1000), DSL (up to 60/20 with VDSL2+), Cable (up to 100/10) and 3G/4G (for 3G up to 16/~3 and 4G I don't know) is available. Ofc depending on where you live. For some people nothing is available and for most only ADSL2+ is the highest possible. (I.e. choose between 24/1 or 20/3)

There have been various satelite and radio solutions aswell, but most of them have died out.

Read the other day that 14% of the connections today are ethernet. I was surprised, I thought it was less. This is my first apartment with ethernet, have used DSL and Cable in the past.


----------



## wonslung (Jul 3, 2010)

wow, the best we can get here is 50 Mb/s down 20 Mb/s up and thats $150 a month.


----------

