# DoS and DDoS attacks



## Remington (Jun 6, 2015)

I know DoS and DDoS attacks are usually targeted against large businesses and organizations with financial and political agendas but I do not see many reports of attacks against small and medium sized businesses.

I'm asking since I'm considering moving my server to a different data center with unlimited internet but they do not have anti-DDoS capability at the moment but I know its great to have that feature.  The question is it really necessary for small and medium business websites to have anti-DDoS feature?


----------



## gkontos (Jun 6, 2015)

Most don't utilize anti DoS features. You could become a victim by accident. Keep in mind that DoS and DDoS are very different with the second being far more difficult to defend against. If you are worried then my advice is to use a CDN service that provides such features.


----------



## Remington (Jun 6, 2015)

My servers have multiple public IP addresses as I host VPS so Cloudflare will not be much help in my case.  I just want to know how widespread the attacks are against small and medium websites.


----------



## gkontos (Jun 6, 2015)

Remington said:


> My servers have multiple public IP addresses as I host VPS so Cloudflare will not be much help in my case.  I just want to know how widespread the attacks are against small and medium websites.


I was referring to http/https services offered by small and medium websites. Cloudflare is not the only CDN that offers this type of protection. 

Like I said earlier, a DoS is very different from DDoS attack. For a small/medium website, eshop, I believe that the use of a WAF driven by a CDN is a necessity. At least in my experience most people that I deal with in this type of business have often experienced some type of attack. 

You may ask why, well hosting companies, especially big ones are a candy for this types of attacks. So, you might end up to be a collateral damage.


----------



## NewGuy (Jun 6, 2015)

I've run a number of small websites and rarely see any sort of serious DoS attack. There are attacks, but most of them are scripts trying to break in rather than take down the website. The risk is relatively low, in my opinion.

The question I think you need to look at is: How much does downtime cost the organization? Maybe it is worth running a second server in another location that can act as a failsafe. Maybe your organization will be fine if the site goes off-line for a day. I feel the question is as much a cost/management decision as a technical one.


----------



## Supermule (Jun 8, 2015)

pfsense and OPNsense is extremely vulnerable to DDoS attacks and can be brought down using sub 10mbit/s traffic patterns.

So its a real concern we see here and you can very easily get a script via Google and anyother search engine.

Isuue here is that anyone wanting to hurt a said site or business can do it. No IT skills needed.


----------



## kpa (Jun 8, 2015)

Supermule said:


> pfsense and OPNsense is extremely vulnerable to DDoS attacks and can be brought down using sub 10mbit/s traffic patterns.
> 
> So its a real concern we see here and you can very easily get a script via Google and anyother search engine.
> 
> Isuue here is that anyone wanting to hurt a said site or business can do it. No IT skills needed.



So? You can bring any firewall based on any OS with a single point of failure to its knees using the same method. What you're saying about pfSense or OPNsense isn't limited to just them.


----------



## Supermule (Jun 9, 2015)

It seems so. We have tested as well on Linksys, Cisco, Netgear and Zyxel and they are affected as well.

We had a test setup in Atlanta that took the Telco Cisco setup offline even before it hit his own setup. 

We didnt get much help on the pfsense forum, so that why I elevated it here.


----------



## gkontos (Jun 9, 2015)

Supermule, this thread is really irrelevant from pfsense. At least the OP concerns. You have opened another thread regarding your issue.


----------



## Remington (Jun 9, 2015)

Supermule said:


> pfsense and OPNsense is extremely vulnerable to DDoS attacks and can be brought down using sub 10mbit/s traffic patterns.



I already did DDoS attacks using 2 computers on my pfSense box with pings at 984Mbps with 4% CPU usage.  Of course everything is super slow but pfSense is still responding.  I don't know where you got your information, you got outdated or under-performing pfSense box.


----------



## Supermule (Jun 9, 2015)

You are more than welcome to provide me with an IP and a port to hit. Then I can show you how little traffic is needed to take pfsense offline.



Remington said:


> I already did DDoS attacks using 2 computers on my pfSense box with pings at 984Mbps with 4% CPU usage.  Of course everything is super slow but pfSense is still responding.  I don't know where you got your information, you got outdated or under-performing pfSense box.


----------



## Supermule (Jun 9, 2015)

I know. Thats why I elevated it here, but we need to set the baseline according to allready conducted tests.



gkontos said:


> Supermule, this thread is really irrelevant from pfsense. At least the OP concerns. You have opened another thread regarding your issue.


----------



## Remington (Jun 9, 2015)

Supermule said:


> You are more than welcome to provide me with an IP and a port to hit. Then I can show you how little traffic is needed to take pfsense offline.



That's not how it works. We'll need same pfSense version, configuration settings, firewall rules, hardware specs and NIC card.  My pfSense box uses Intel adapter instead of built-in NIC.  Cheap on-board NIC can cause the server to go down easily since it usually cannot handle high volume traffic.  I've been running different DDoS tests and pfSense is still up.  So something on your end is causing your pfSense to go down.  I don't see many people complaining about this on pfSense forum.  A lot of has to do with bad configuration settings or bad box.


----------



## Supermule (Jun 9, 2015)

Thanks for the kind reply.

A vanilla install of pfsense in Intel Dual port server NIC's can be taken down instantly.

There is no setting right now that addresses this issue. We have tested pfsense from 1.2.3 onwards to current.

2.2.2 definately is better then 1.2.3 but it goes offline instantly. Using vmxnet3 adapters makes it a lot worse than the em drivers.

If you want to see it, I can easily demonstrate it to you. It takes 3-5 mins and only thing I need is ping beeing replied to on WAN and a IP and port nr.

Then I can blow your pfsense out of the water using less than 10 mbit/s traffic.

We have done so a 1000 times during the last 3 mths testing like mad. Its NOT misconfigured.

*As long as you have a port forwarded, your vulnerable.*

But we dont need to discuss pfsense. Even ESF themselves are vulnerable but they wont listen or they dont have the knowledge to fix it.

Thats why I wanted to elevate discussing core issues in FreeBSD. And thats why we are here.

Currently trying to setup a bare metal bsd box running pf in Atlanta. Then we know for sure where the issues are.


----------



## Supermule (Jun 9, 2015)

And by the way, I run 2.2.2 AMD64. This is not limited to a specific release but affects all versions.

So it doesnt matter what you run or that we run the same version to test.



Remington said:


> That's not how it works. We'll need same pfSense version, configuration settings, firewall rules, hardware specs and NIC card.  My pfSense box uses Intel adapter instead of built-in NIC.  Cheap on-board NIC can cause the server to go down easily since it usually cannot handle high volume traffic.  I've been running different DDoS tests and pfSense is still up.  So something on your end is causing your pfSense to go down.  I don't see many people complaining about this on pfSense forum.  A lot of has to do with bad configuration settings or bad box.


----------



## Remington (Jun 9, 2015)

Okay.  I did further test by using NAT and port and you are correct that LAN speed is around 10 to 100 Kbps during DDoS attack.  I started DDoS test while uploading a large file to the server.  Everything became slow to 10 to 100 Kbps on LAN while DDoS was hitting at 1Gbps.  As soon as I killed the DDoS test the upload resumed at fast speed so pfSense is still working.  I think NAT software wasn't designed to handle high packets in fraction of a second.  So what is your point?  I mean DDoS does bring everything down to a crawl but it doesn't bring down pfSense and it would be the same for any routers.  I rather have pfSense take brute of abuses than servers behind it.

I think an expensive hardware router would be better suited to deal with that but that is entirely another subject.


----------



## Supermule (Jun 9, 2015)

Thats complete logic since there is no room on the wire for the file transfer.

Current issue is that it doesn't take a 1 Gbit DDoS to take PF offline. Notice that I say PF as in FreeBSD/PF. It can happen on a sub 10 Mbit/s DDoS and despite having a big pipe available, the packets is not routed until "something" happens behind the scenes that is not logged and transparent in either `top -HSP` or any other monitoring tool.

Issue persist on all kinds of interfaces and in different hypervisors and on bare metal. Currently running igb(4) drivers on ESXi and despite SMP, it doesn't get better. Thats why I elevated my issue here since I have an opportunity to learn a lot about FreeBSD and how to debug PF. Talked to devs at OPNSense and they are very friendly and wants to work with me on seeing what is the cause of this.

You are more than welcome to write me a PM with testing options or contact me on skype: kontaktnetsupply since its very easy to exchange info there.


----------



## gkontos (Jun 9, 2015)

I still don't understand how you have conducted those DDoS attacks. Let alone if they are indeed DDoS or just DoS. In any case, in order to understand the problem you will need to share the following information:

Target Host
Current Running firewall policy and configuration
Method of attack
Duration
Observed behaviour
Detailed logs of the device under attack 
All in detail of course.


----------



## Supermule (Jun 10, 2015)

Any server running FreeBSD/PF. Frontend FW's using PF and NAT'ing to windows servers.

None. A normal port 80 NAT rule. No specific options set other than using SYN Proxy states.

A SYN Script flooding the designated IP with spoofed traffic.

Could be anything you desire. It could last 2 weeks if you want. You can set it in the parameters of the script.

Packet loss and no routed legitimate traffic.

None since the systems normally crash. You need to use Wireshark or maybe Mikrotik router in between pf and internet to capture.

[_Mod: Stop using colors please_]


----------



## Remington (Jun 10, 2015)

Supermule said:


> Current issue is that it doesn't take a 1 Gbit DDoS to take PF offline. Notice that I say PF as in FreeBSD/PF. It can happen on a sub 10 Mbit/s DDoS and despite having a big pipe available, the packets is not routed until "something" happens behind the scenes that is not logged and transparent in either `top -HSP` or any other monitoring tool.



I was able to flood my server with tiny sized packet with high requests at around 50Mbps and that's low for 1Gbps bandpipe I got.  I tried to upload a file to my server and it was very very slow.  pfSense's CPU shot up to around 40% usage but didn't crash.  My pfSense box got 8 cores.

Well, I think its important to coordinate with data centers to block DDoS or DoS attack and most of them will work with their clients.  My data center gives me a report of DoS/DDoS attacks and I can ask them to permanently ban them so it won't affect my servers.  Cloudfare is good but its only for websites.  They don't help with SSH, Ping, FTP, or any other ports or using IP address to mount an attack on the server.

The things I like about pfSense are the routing capabilities, VPN, firewall, monitoring and statistic logs, etc. so I don't have to install bunch of same software on each server as I like to keep software in the host to a minimum as possible.  It makes it easy to switch to a backup server if one of the servers goes down or undergo upgrades.

Do you know if IPFW has the same problem as PF?


----------



## Supermule (Jun 10, 2015)

We haven't tested IPFW since it seems almost everybody is running PF.

They don't block this kind of traffic since it's sub 20 Mbit/s bandwidth. They can't separate the legitimate traffic from the attacking one.


----------



## junovitch@ (Jun 11, 2015)

Supermule said:


> None. A normal port 80 NAT rule. No specific options set other than using SYN Proxy states.
> 
> A SYN Script flooding the designated IP with spoofed traffic.
> [_Mod: Stop using colors please_]



pf.conf(5)


> SYN PROXY
> By default, pf(4) passes packets that are part of a tcp(4) handshake between the endpoints.  The synproxy state option can be used to cause pf(4) itself to complete the handshake with the active endpoint, perform a handshake with the passive endpoint, and then forward packets between the endpoints.
> 
> No packets are sent to the passive endpoint before the active endpoint has completed the handshake, hence so-called SYN floods with spoofed source addresses will not reach the passive endpoint, as the sender can't complete the handshake.



Generally synproxy isn't something that you would use as a standard configuration.  It would be applied to mitigate an ongoing attack.  What are you spoofing?  If it's the source IP then it would seem synproxy isn't doing what it is designed to do.  If it's something else then it seems you are using synproxy outside of the confines of what it's designed to do.  Is the issue any worse/better when you don't use synproxy?


----------



## kpa (Jun 11, 2015)

Synproxy by default is a bad idea because it changes the meaning of standard TCP three way handshake. If you have synproxy on it no longer means "the recipient endpoint has accepted the connection" but it turns into "the firewall in between (that you shouldn't know about) has let the connection trough". If the recipient of the connection turns down the connection that was accepted by the firewall you'll get a hung up connection that has to wait for time out to be cleared. This will obviously increase the load on the firewall because there will be more states waiting for timeout than what there would be normally.

synproxy slowdown


----------



## Supermule (Jun 11, 2015)

pfSense change states during attack.






First part is no enabled NAT. Traffic bounces of the FW. Then NAT is enabled and packet loss occurs. Overall traffic in mbit/s is not high, but PPS is.

Enabled adaptive FW timeouts set to 250.000 states. Throttling to aggressive at 500.000 despite 2MM max.

Going through different types of states until running stateless and the FW begins to route again.

But running stateless means you lose a lot of the advantage of a stateful inspection FW.



junovitch said:


> pf.conf(5)
> 
> 
> Generally synproxy isn't something that you would use as a standard configuration.  It would be applied to mitigate an ongoing attack.  What are you spoofing?  If it's the source IP then it would seem synproxy isn't doing what it is designed to do.  If it's something else then it seems you are using synproxy outside of the confines of what it's designed to do.  Is the issue any worse/better when you don't use synproxy?


----------



## juiced (Jun 11, 2015)

@*Supermule* Just a heads up - the stateless video you posted in the other thread wasn't stateless. It was using 60% of 2,000,000 tables at peak.

If you haven't already done so - substantially lower the states per host, state timeouts and limit connections per second.
-Set the actual timeouts lower -Some services can handle very low timeouts, some can't ->experiment with it.
-Set the adaptive timeout start point much lower.
-Limit simultaneous connections per client
-Limit states created per host
-Limit the number of "new" connections allowed per second - be cautious with this one.

The limiting options per host and per client won't always help if the attack is coming from a large number of actual sources - or is simulating random source's. So running lower settings for .first, .established and friends https://www.freebsd.org/cgi/man.cgi?query=pf.conf&sektion=5 +new connections per second can pick up the slack.

The above limiting rules may not work with all types or protocols. So making multiple rules might be needed. And look into using "Float" rules in pfsense + the quick pf option.

For faster NAT you can also try: net.inet.ip.fastforwarding=1
There's a few caveats https://wiki.freebsd.org/NetworkPerformanceTuning

-I apologize if I've posted anything you've already tested. But between both threads there's been hardly any information posted. I understand and respect your reasoning for not posting the script or exact attack type details. It makes sense, and frankly I'm glad it wasn't posted publicly.

However not posting things like the sysctl.conf, pf.conf, hardware specs as well as any settings you've already tried + those results, makes getting a helpful response nearly impossible. Videos of a gui don't really cut it. Especially if inaccurate. 

-And as I'm sure you're aware none of this will stop a massive multiple source attack.

@*Remington*
Sorry if I derailed your thread m8. Hopefully at least some of this is helpful.


----------



## Remington (Jun 11, 2015)

juiced said:


> Remington
> Sorry if I derailed your thread m8. Hopefully at least some of this is helpful.



No problem at all.  This is an interesting discussion as I would like to find solution to this as well optimize the configurations.


----------



## Supermule (Jun 12, 2015)

No worries.

This is whats in my loader.conf

```
autoboot_delay="3"
vm.kmem_size="435544320"
vm.kmem_size_max="535544320"
if_igb_load="YES"
kern.hz=1000
boot_serial="YES"
comconsole_speed="115200"
hw.usb.no_pf="1"
aio_load="YES"
cc_htcp_load="YES"
net.inet.tcp.hostcache.cachelimit="0"
hw.igb.txd="2048"
hw.igb.rxd="2048"
hw.igb.rx_process_limit="-1"
hw.igb.enable_aim="1"
hw.igb.num_queues="0"
hw.igb.enable_msix="1"
kern.ipc.nmbclusters="492680"
net.inet.tcp.syncache.hashsize="1024"
net.inet.tcp.tcbhashsize="65536"
net.isr.bindthreads="0"
net.isr.dispatch="direct"
net.isr.maxthreads="1"
```

It made very little impact on performance and resiliency. We have tried almost anything thats in https://calomel.org/freebsd_network_tuning.html

When the firewall gets NAT enabled and the packets needs to traverse pf then it bugs down.

Below tuning tips have been tried to no avail. It doesnt seem capable of sorting legit traffic from the DDoS one.



> Set the actual timeouts lower -Some services can handle very low timeouts, some can't ->experiment with it.
> -Set the adaptive timeout start point much lower.
> -Limit simultaneous connections per client
> -Limit states created per host
> -Limit the number of "new" connections allowed per second - be cautious with this one.


----------



## juiced (Jun 12, 2015)

Supermule said:


> Below tuning tips have been tried to no avail. It doesnt seem capable of sorting legit traffic from the DDoS one.


It doesn't need to sort legit from non.

If your system can't handle the number of states its dealing with "and it can't if setting it stateless solves it" then reduce the timeouts etc.


----------



## Remington (Jun 12, 2015)

What about 1:1 NAT without redirecting to different port or IP address?


----------



## Supermule (Jun 12, 2015)

THAT we havent tested 



Remington said:


> What about 1:1 NAT without redirecting to different port or IP address?


----------



## Supermule (Jun 12, 2015)

Run aggressive settings on states/timeouts and adaptive timeouts.

It handles better if they are set to none and the max limit on 2MM. 



juiced said:


> It doesn't need to sort legit from non.
> 
> If your system can't handle the number of states its dealing with "and it can't if setting it stateless solves it" then reduce the timeouts etc.


----------



## juiced (Jun 12, 2015)

Since you're completely missing the point I give up.

FWIW:
We smashed the hell out of FreeBSD locally bare metal -> bare metal on decent nic's. Once setup correctly It performed quite well and pretty much as expected.

-TGIF


----------



## Supermule (Jun 13, 2015)




----------

