# Optimal Router Hardware



## Ancyker (Oct 19, 2019)

I couldn't decide if this should go in this section or the networking one but I decided since it's only about the hardware side it best belonged here. Also, I know this has been asked before, but I couldn't find any posts that were recent and specific to my needs.

So, I have been looking at getting FTTH and am looking at using FreeBSD as a router. The speed I'm looking at is 2gbit so a conventional home router won't handle it and enterprise routers are stupidly expensive compared to their stated performance. So, I decided hey, let's build a FreeBSD server to act as a router.

Here's the plan so far as far as hardware goes: Fiber into a Mellanox ConnectX-3 10GbE SFP+ card. Passive copper from another Mellanox ConnectX-3 10GbE SFP+ card going to a Ubiquiti Networks US-16-XG (10G 16-port managed switch). From there it will branch out into a Linksys EA9500 for wireless (in bridge mode) and a Ubiquiti BULLET-M2 (also in bridge mode) as well as switches, etc.

The plan is to be truly dedicated to this task, nothing else will be on this hardware. It would perform the same functions as a typical home router. This includes routing, standard firewall (closed ports/port forwarding), DHCP, DNS, QoS (to some extent), etc. No databases, web servers, media servers, etc.

So, that brings me to my question. What would the optimal CPU be for this task? I don't want to skimp on it, but I also don't want to go way overkill. I want to reduce latency and maximize PPS. What matters most here? Single-core speed? The number of cores? How much and how many are needed? Intel vs AMD? ECC sounds like a good idea, so if Intel a Xeon would be needed. That might make AMD a better choice if tons of CPU isn't needed while the higher end Intels tend to perform better. Speaking of RAM, how much RAM would be ideal? I've used FreeBSD for a long time, but I've never used it for this before. If you are going to recommend a barebone system, I'm going for a 1U/2U form factor. Power consumption is not a top concern, performance is more important.

Any advice about anything I've said is appreciated. If you see anything in my setup that seems wrong or has a better choice available feel free to let me know. Also, if another flavor of BSD would be better-suited feel free to state the case for it, though I'm more comfortable using FreeBSD as I have not used the others at all.

Thanks.


----------



## aht0 (Oct 19, 2019)

Have you thought about using OpenBSD? Their PF is under active development and I remember reading from somewhere that with Mellanox hardware you don't have to deal with "big lock" (you'll get proper SMP performance) on OpenBSD. Also their PF is supposedly better performing (some claim ~4x better) than version/fork FreeBSD has.
I realize this is FreeBSD forum but I have no bias against neither BSD and you asked for an opinion.


----------



## Phishfry (Oct 19, 2019)

Ancyker said:


> I want to reduce latency and maximize PPS. What matters most here? Single-core speed? The number of cores?


I went the Chelsio T540 route and stuffed 4 in a X9SRL. This is single socket LGA2011 using 16GB RAM and mirrored 16GB SSD.
It is extreme overkill. I wanted to ensure all 4 cards had x8 lanes of PCIe 3.0.
I am using an E5-2618LV2. I downgraded from 2650LV2 as no need for all those cores for a router/softswitch.
In my opinion a Core I3 quad core chip is plenty. Skylake 1151 chip like E3-1240LV5 is ideal drawing only 25W TDP.
Just remember that LGA1151 only has x16 lanes of PCIe3. So MATX is fine. For a single ConectX3 you could use ITX.
Use an industrial grade board for 24x7 ops. I prefer SuperMicro but also own a server grade Gigabyte Skylake MATX board.
Single core speed is only a factor when using something like PPPoE whereas PPP is a single threaded application.
There are ways around these limitations as well by using the mpd5 daemon.

No need for more than 8GB RAM as FreeBSD networking&routing has real low overhead. pf really doesn't need much umph either.
You start adding monitoring tools like Suricata or Snort and you might need to step up the RAM & cores some.
1U case with a 90 degree riser should work fine.
I used hotswap Emacs power for versatility. They are loud but dependable.
Because my network cards were full heigth I went with a 3U chassis I had in storage. Built a heat shroud for the fan to maximize airflow over the Chelsios. They run really hot.


----------



## Phishfry (Oct 19, 2019)

I really couldn't care if FreeBSD's pf is versions behind OpenBSD's version.
All it does is filter packets. No need for all the fancy features.
That is what is important to me. It is more likely another OS component that will be vulnerable.
Thus use the operating system you know the best as a bad setup is more likely than a pf vulnerability.
Do notice that OpenSSH is a frequent culprit on OpenBSD.





						Openbsd : Security vulnerabilities
					

Security vulnerabilities related to Openbsd : List of vulnerabilities 			related to any product of this vendor. Cvss scores, vulnerability details and links to full CVE details and references



					www.cvedetails.com
				




FreeBSD bugs are more varied in scope.





						Freebsd : Security vulnerabilities
					

Security vulnerabilities related to Freebsd : List of vulnerabilities 			related to any product of this vendor. Cvss scores, vulnerability details and links to full CVE details and references



					www.cvedetails.com
				



Unfortunately to undercut my argument FreeBSD did have a pf bug very recently.





						CVE-2019-5598 : In FreeBSD 11.3-PRERELEASE before r345378, 12.0-STABLE before r345377, 11.2-RELEASE before 11.2-RELEASE-p10, and 12.0-RE
					

CVE-2019-5598 : In FreeBSD 11.3-PRERELEASE before r345378, 12.0-STABLE before r345377, 11.2-RELEASE before 11.2-RELEASE-p10, and 12.0-RELEASE before 12.0-RELEASE-p4, a bug in pf does not check if the outer ICMP or ICMP6 packet has the same destination IP as the source IP of the inner protocol...



					www.cvedetails.com


----------



## Deleted member 30996 (Oct 20, 2019)

Phishfry said:


> I really couldn't care if FreeBSD is versions behind OpenBSD's version.
> All it does is filter packets. No need for all the fancy features.
> That is what is important to me.



I have a working OpenBSD installation on my IBM T43 but that's my Precious and never use it. I do use the same pf ruleset on FreeBSD and OpenBSD with the expection of changing one word in the egress rule on OpenBSD and both work fine. MIne is set to heavily block incoming traffic and I don't have any services running so rarely go beyond looking at the stats in the daily Security Mailings.


----------



## gpw928 (Oct 21, 2019)

Phishfry said:


> Do notice that OpenSSH is a frequent [vulnerability] culprit on OpenBSD.
> 
> 
> 
> ...


I think that  your reasons for choosing FreeBSD are sound, but might the reason that OpenBSD has lots of OpenSSH issues listed be because it is *the* development platform for OpenSSH?


----------



## toorski (Oct 21, 2019)

Take a look at this one:








						Vigor2960
					

End-of-Life Product




					www.draytek.com
				









						Amazon.com: DrayTek Vigor 2960 : Electronics
					

Buy DrayTek Vigor 2960: Routers - Amazon.com ✓ FREE DELIVERY possible on eligible purchases



					www.amazon.com
				




I have their older "Vigor 3200n" SOHO/Multi-WAN router that was given to me, many years ago, as a demo. But, I was never asked to return it. Although, they stopped updating firmware on this router, it still works OK, in my friend's small resort/ hotel/restaurant with 2x100mbps WAN connections and my tech/admin support   It has very nice and simple to navigate UI with tons of features.


----------



## tingo (Oct 22, 2019)

FWIW, I have a 1 Gbps internet connection (technically it is FTTH, but it ends in a Genexis Platinum 6840 gateway, so my  router / gateway / firewall only sees a Gigabit Ethernet interface). My gateway / router / firewall is a Shuttle XH61V, Celeron G540 cpu, 8 GB RAM, an SSD for storage, it has two gigabit ethernet interfaces. It runs FreeBSD, I use ipfw as firewall.
The machine never has any load;

```
root@kg-omni1# date;uptime
Tue Oct 22 20:25:04 CEST 2019
 8:25PM  up 62 days, 12:34, 1 user, load averages: 0.00, 0.00, 0.00
```
it looks like this all the time, no matter what kind of traffic I'm running through the internet connection.


----------



## toorski (Oct 22, 2019)

I have CTTW from a wooden pole with a shitty 100Mbps down and 50Mpbs up, because I live in a digital area shit hole. My ISP's IP/GW router's firmware is too old. My own LAN to GW-IP/WAN router is an old laptop that runs *BSD firewall connected to another old laptop, over dumb switch, with FreeBSD server that is also protected by another FW. I have minimal understanding of how the automagic of telecomm switches, TCP/IP routers, gateways, bridges and servers with their service ports, work. I just play, pray and hope that the chips and software running the monster called Internet, with its TCP/IP services, has enough AI to take care of itself and me – hehe

Even if I had FTTW and those slick IP/GW routers, none would help me and my dumb-ass TCP/IP show. But, I think that an IP/GW router with customized embedded OS kernel, TCP/IP-FW and related utilities is more efficient than poorly developed or configured router, on PC with a generic OS kernel.


----------



## usdmatt (Oct 23, 2019)

I used to use Draytek but have become a fan on Mikrotik for small office or home use now. Something like the well named RB4011iGS+5HacQ2HnD-IN would be a great router for a FTTH connection, isn't badly priced compared to "higher end" brands and it will happily route at more than 2Gb, even with 25+ firewall rules. They also have a cheaper desktop/rackmount version with a slightly less ridiculous name (RB4011iGS+RM) if you don't need built-in wifi.

Personally I tend to avoid homebrew routers. I couldn't build something to outperform the above Mikrotik for less money (unless I had a half decent machine lying around already), it would use more power and configuring/managing it, especially more advanced routing or IPsec configs, would be far more hassle.


----------



## Ancyker (Oct 23, 2019)

So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?


----------



## neel (Oct 24, 2019)

Probably.

I use an OPNsense-powered HP ProDesk 400 G4 as my "router" on 300 Mbps Verizon FTTH, connected directly to the Verizon ONT (GPON to Ethernet media converter). I know its overkill when compared to many low-power solutions, but I had trouble optimizing various "Mini PCs" (both HP and AliExpress/QCY models) so I decided to use a real desktop. And when >1 Gbps broadband comes, I can just swap the NICs.


----------



## toorski (Oct 24, 2019)

Ancyker said:


> So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?


Nowadays, any half-ass modern PC hardware and older decent platfrom with SSD, including laptops, can be used as a IP/GW router for a simple LAN, or as advanced as you want, if you know howto PF, IPFW and trim your FreeBSD kernel to do just what you need in your router. The problem with consumer level PC(s) with their CPU(s) and  generic kernels is that both waist tons of resource for things that you maybe or will do, with your CPU and OS. That's what developers and/or manufacturers of those slick and small footprint routers are charging for - custom hardware platfrom, ARM based CPU(s) and kernels to support all the fancy TCP/IP utilities ,with point and click UI, that you may or may not ever need 

ATM, I'm experimenting with FreeBSD to learn howto make my laptop a better router, than it's now, with things like DOS protection and few extra futures. I also want to run authoritative and caching DNS servers in the router.


----------



## neel (Oct 24, 2019)

toorski said:


> Nowadays, any half-ass modern PC hardware and older decent platfrom with SSD, including laptops, can be used as a IP/GW router for a simple LAN, or as advanced as you want, if you know howto PF, IPFW and trim your FreeBSD kernel to do just what you need in your router. The problem with consumer level PC(s) with their CPU(s) and  generic kernels is that both waist tons of resource for things that you maybe or will do, with your CPU and OS. That's what developers and/or manufacturers of those slick and small footprint routers are charging for - custom hardware platfrom, ARM based CPU(s) and kernels to support all the fancy TCP/IP utilities ,with point and click UI, that you may or may not ever need
> 
> ATM, I'm experimenting with FreeBSD to learn howto make my laptop a better router, than it's now, with things like DOS protection and few extra futures. I also want to run authoritative and caching DNS servers in the router.



However, you probably can't if you have a Gigabit FTTH connection which uses PPPoE. Some telcos like Bell Canada, CenturyLink (US), and NTT (Japan) are like this. I heard of people having trouble with PPPoE on Bell and CenturyLink Gigabit connections with pfSense.

I have Verizon FiOS FTTH and they use DHCP, and I never had Gigabit broadband, very less PPPoE on one. I have 300 Mbps, and if it weren't for my Tor relay, I would only have 100 Mbps (the most I do is ISO downloads or YouTube otherwise). But if US broadband was as cheap as Europe or Asia, maybe I'll go Gigabit, who knows?

My HP ProDesk handles 300 Mbps without PPPoE just fine, and can probably handle a Gigabit as well. Heck, I could do multi-Gigabit when it comes with just a NIC swap. It probably can also do PPPoE Gigabit, but I don't know.


----------



## gpw928 (Oct 24, 2019)

Ancyker said:


> So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?


Whether you go turnkey or DIY, a lot of the small form factor PCs and Mikrotik appliances you see on places like Amazon have CPUs, like the Celernon J1900, that don't support "Intel AES New Instructions (AES-NI)".  For a new investment in a firewall, AES-NI is a "must have" for future-proofing.


----------



## neel (Oct 25, 2019)

gpw928 said:


> Whether you go turnkey or DIY, a lot of the small form factor PCs and Mikrotik appliances you see on places like Amazon have CPUs, like the Celernon J1900, that don't support "Intel AES New Instructions (AES-NI)".  For a new investment in a firewall, AES-NI is a "must have" for future-proofing.


If you are just doing packet forwarding (like me), AES-NI isn't necessary. If you want to do VPN, it's a must-have.

Even if you are just doing packet forwarding, get something AES-NI compatible. Maybe you'll want VPN in the future, or if you ever resell your device your resale value is higher since the next owner can also do VPN.

I skip the Mini PCs and use a real desktop, tuning a bicycle is too hard so I'll just use a motorbike. But then, I have FTTH and run Tor relays on my connection, many of you may have fiber but will never run Tor and in that case a Mini PC works pretty well.

If you don't mind non-FreeBSD, a Ubiquiti box could also be a good buy. I never had one but heard great things about them.


----------



## toorski (Oct 25, 2019)

neel said:


> However, you probably can't if you have a Gigabit FTTH connection which uses PPPoE. Some telcos like Bell Canada, CenturyLink (US), and NTT (Japan) are like this. I heard of people having trouble with PPPoE on Bell and CenturyLink Gigabit connections with pfSense.



Sorry for  generalizing and characterizing my FreeBSD router. What I, maybe, meant was more like LAN/IP/FW router.

In my case, I operate on both Static LAN/IP's, and Static Public IPs from my ISP. So, my FreeBSD router handles my LAN or VLAN network traffic on one network interface and my WAN Public IP  on another (USB to Ethernet conversion cable) network interface which get routed through my IPS's LAN-GW/IP pass-through router with a very basic LAN port forwarding and security options.

That's why I consider my laptop with FreeBSD and two network interfaces an IP/GW router, and my other laptop with FreeBSD, connected over dumb switch to my FreeBSD router, a server. All other computers, that connect over my dumb switch to my FreeBSD router, just sit  there and wait to be updated, upgraded or hacked-up 

Edit:
Almost forgot,
I also have old wire/wireless router, for my wireless devices, connected to my ISP's router, which I also consider an IP/GW router - hehe


----------



## toorski (Oct 25, 2019)

The global switches connect to modems, modems connect to routers. Then, routers connect to more routers and switches. And, somewhere in-between are my IP/GW routers


----------



## Lars Skogstad (Oct 26, 2019)

Get a i3 with good ethernet ports, 16gb ram and you should be fine.  
Mine had an older i3 and rarely didn't see any cpu usage, hitting almost 1gbit from my 1gbit ISP. Think I was averaging on 987mbit or something.


----------



## LVLouisCyphre (Jan 3, 2020)

If it performs well as a server, it generally will perform better as a router.  You're only concerned up to ISO layer 4 (TCP/UDP ports) with the primary application of this box.  With servers you're concerned about performance all the way up to layer 7.  Firewalling, routing and VPN are not that big of a deal or CPU intensive in my experience.


----------



## neel (Jan 9, 2020)

I changed my firewall again, to a HP T730. I'm also moving, where my new ISP is Gigabit Wave G versus my old ISP, 300 Mbps Verizon FiOS. Interesting to see how the T730 will scale on a full Gigabit.

The T730 is a decent box, but has some issues with certain Intel NICs, some people had success with "genuine" (meaning non-counterfit) Intel T350 cards, I went with a Dell Broadcom 5720 unit. If you want a T730 and have the guts to go Broadcom, just do it.

I repurposed my previous firewall, a HP ProDesk 400 G4, as a desktop. I did a few upgrades to the CPU (Pentium->i7) and RAM (8GB->24GB) and it works pretty well. Not as powerful as my workplace's Dell Precision, but powerful enough for compiling Ports as a maintainer.


----------



## LVLouisCyphre (Jan 9, 2020)

Ancyker said:


> So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?


Yes. 

At the risk of carbon dating myself, I've had dedicated FreeBSD firewalls and routers for years.  Even a half duplex ISA NIC if you're running an old or slow broadband connection will do the job.  20 years ago, I had an old crap P54C-90 (socket 5) system, 32 MB RAM, i430VX chipset with an SMC 8013 16-bit ISA NIC (ed(4)) going to a cable modem connection at half duplex via a crossover cable.  I forget what internal NIC I used; probably an Intel 8461 Pro/100+ management; fxp(4).  The cable modem had a slower upload than download speed so half duplex wasn't a major issue.  The CPU was pretty much yawning waiting for something to do.  I was using ipfw(8) at the time before one of its rewrites.  This was before pf(4) existed.  Pf is probably what you should be using on your dedicated router.

An HP Proliant Microserver G7 could probably do the job as it has a PCIe 2.0 x16 slot.  I would get a system that has support the same PCIe version or newer as your 10GbE NIC to get the maximum bus speed of the NIC.  Packet filtering is not very CPU intensive especially if you have a dedicated box for it and a good NIC that does some of the L2 processing on the card. 

What will hammer a router CPU are (distributed) DoS attacks such as with the Blaster worm.  Construct your ACLs accordingly or it won't make a bit of difference how much bandwidth, CPU and memory you throw at your router.  In fact, higher bandwidth can cause the CPU to go into clock speed arrest.

I ended up pulling the overtime CERT duty to deworm the WAN.  It knocked out over 200 Cisco 2503 routers and a pair of 7513s.  The ICMP traffic was causing a WAN wide distributed DoS attack.  We kept the anti Blaster worm ACL to this day (or at least of as of when I retired on July 11, 2013). 

I could also tell you the horror story of a real librarian of genius (_*hums Budlight's __Real Men of Genius__ theme*_) took down a site's WAN connection (on the date payroll was due to be authorized) leeching via Napster by saturating their WAN connection. Bittorrent client users that do not throttle their bandwidth and go through a VPN are also known to do this. I had to hunt down one of those real men of genius as well. There I go again, carbon dating myself. 

I don't know anything about your switching hardware.  I'm a Cisco purist by trade.  Your Linksys is supported by DD-WRT which makes it good in my book.


----------



## Lars Skogstad (Jan 10, 2020)

I guess it depends if you're running a pure firewall only with NAT or with a lot of other feats. That can consume a bit more.


----------



## Phishfry (Jan 10, 2020)

I just bought a used Lanner NCA-1010B that was a Untangle branded box.
It is the smallest router in my inventory.





						NCA-1010
					

NCA-1010 is Lanner’s revolutionary ultra compact x86 networking system built with Intel® Bay Trail CPU (Atom™ E3815/E3825). The central processor comes with hardware-assisted security mechanisms including AES-NI, allowing only authorized software or data to run on NCA-1010. With built-in...




					www.lannerinc.com
				




Some of the shelf sized routers I have bought from ebay:
Checkpoint U-5
Jetway JCB375
Caswell CAD-205
Lanner FW-7535
Lanner FW-7525
Nexcom DNA-110
Sophos XG85
Sophos XG105
Sophos SG135
Astaro ASG110 rev4
PCEngines APU1,2,3


----------



## Crotalus (Jan 18, 2020)

Have you looked at PfSense for a router? It uses FreeBSD as the operating system. I put one together several years ago with an ITX motherboard the interface is through a web page for maintenance.


----------



## LVLouisCyphre (Jan 26, 2020)

Lars Skogstad said:


> I guess it depends if you're running a pure firewall only with NAT or with a lot of other feats. That can consume a bit more.


That's correct.  I have all of my FreeBSD (based) boxes set up as a standby firewall/router regardless of what their primary function is with carp(4), pf, et.al.  It's trivial to set up a standby firewall/router or have a server double as a standby firewall/router.  

You could call FreeBSD a RAIF/G/R; redundant array of independent (or inexpensive) of firewalls, gateways or routers.    

It's one of the most fundamental functions of FreeBSD that works exceptionally well.  The same thing can be done with OPN and pf Sense as they're both FreeBSD based.


----------



## neel (Apr 26, 2021)

michaelshah said:


> There was no way that FreeBSD could complete the task I assigned it! The maximum speed I achieved was 1 gigabyte, although I was just as intent on 2 gigabytes of speed as you were.


Having multi-Gigabit speeds means that you need either a >1 Gbps NIC, or need to bond multiple 1 Gbps NICs. Some cable modems allow for bonding multiple 1 Gbps NICs, but fiber is usually fiber to the SFP or less likely a NBASE-T/10GBASE-T Ethernet handoff.

In the US know Google Fiber (if you can get it) and Comcast are going with 2 and 1.2 Gbps respectively, but others like Verizon, AT&T, Spectrum, Cox are still at 1 Gbps max (if you can even get it). In Canada, Bell, Telus, and Shaw have 1.5 Gbps tiers, but others like Rogers, Cogeco, TekSavvy, etc. are still at 1 Gbps. In many European and Asian countries with more prevalent FTTH, there are multi-Gigabit FTTH speeds, but there are many countries in Europe/Asia with different ISPs. For instance, in Sweden or Japan 10 Gbps is widely available but in the UK or India, Gigabit FTTH is still a pipe dream for most.


----------



## malavon (Apr 26, 2021)

It seems you already have plenty of answers, but I'll add my experience to it. I've been using FreeBSD-based router/firewall voxes since I first had broadband, back in 2002. For a 100MBit connection I used a Pentium 2 300MHz, that also doubled as a NAS.
Nowadays I use an i3-4160 (Haswell era) on a Supermicro ITX board (server Intel NICs), which is also an ESXI box that runs a dozen VM's. CPU usage is next to nothing since I stopped running an ELK stack. 

So yes, pretty much any modern x64 hardware will do. I'd concentrate on the NICs you want, either onboard or discrete. Make sure your motherboard has enough PCIe lanes on all the ports you want to use. That's basically only important if they need more than 4 though.

btw. you have a few firewall options to do this. My previous box used pf, nowadays I use IPFW. Not sure if there's much difference in cpu load.


----------

