# [FreeBSD + PF cbq + borrow] Dynamic shaping



## Denok (May 18, 2009)

Hi all,

Before now I've only used IPFW + Dummynet, but decided to use PF and ALTQ for dynamic shaping. But I can't make it use whole bandwidth when there is only one host on, so the link is free,  borrow parameter is designed to expand parent bandwidth when it is free, but not in my case 

current configuration gives host only 50% of bandwidth even when the link is free,
here is PF config:


```
ext_if="rl0" # 
int_if="rl1" # 
lan="192.168.10.0/24" # LAN
set loginterface $ext_if

table <user1_ips> { 192.168.10.2, 192.168.10.5 } #IP addresses of user1
table <user2_ips> { 192.168.10.3 } #IP of user2
scrub in all
# IN
altq on $int_if cbq bandwidth 100Mb queue { inet_in, default_in }
queue inet_in bandwidth 512Kb { user1_in, user2_in }
queue user1_in bandwidth 50% cbq(red, borrow)
queue user2_in bandwidth 50% cbq(red, borrow)
queue default_in bandwidth 99% cbq(default)

# OUT
altq on $ext_if cbq bandwidth 100Mb queue { inet_out, default_out }
queue inet_out bandwidth 256Kb { user1_out, user2_out }
queue user1_out bandwidth 50% cbq(red, borrow)
queue user2_out bandwidth 50% cbq(red, borrow)
queue default_out bandwidth 99% cbq(default)

# NAT
nat on $ext_if from $lan to !$lan -> $ext_if

# Queues
# IN
pass in on $int_if from <user1_ips> to !$lan queue user1_out no state
pass in on $int_if from <user2_ips> to !$lan queue user2_out no state
# OUT
pass out on $int_if from !$lan to <user1_ips> queue user1_in no state
pass out on $int_if from !$lan to <user2_ips> queue user2_in no state
```

Also, when I'll solve this problem, I'd like to make different shapes for local-IX traffic and overseas (World) traffic, what's the best way to do this with this config?

Thanks!


----------



## DutchDaemon (May 18, 2009)

Try setting the parent queues to borrow as well.


----------



## Denok (May 18, 2009)

Thanks for reply, 

can you, please, tell me more precise, i've changed the "top" parent queue (grandparent?) because queue "queue inet_out" don't have scheduler ( sched_options ) - btw, why?

So,


```
altq on $int_if cbq(red, borrow) bandwidth 100Mb queue { inet_in, default_in }
queue inet_in bandwidth 512Kb { user1_in, user2_in }
queue user1_in bandwidth 50% cbq(red, borrow)
queue user2_in bandwidth 50% cbq(red, borrow)
queue default_in bandwidth 99% cbq(default)
```

and nothing changed, still 50% of bandwith when it is supposed (and can) borrow all.


----------



## DutchDaemon (May 18, 2009)

queue inet_in is the parent queue in this case.


----------



## Denok (May 19, 2009)

Still no luck with cbq, meanwhile i've tried hfsc scheduler - shapes on incoming traffic work (dynamic too) but upload is not shaped at all (check config in attachment).

now I'll make one more attempt with cbq, with config that one dude gave me in reply to my post.

I feel like I'm the only one who is doing dynamic shapes with PF on FreeBSD! :q There are configs but none of them are 100% working.


----------



## Denok (May 19, 2009)

hfsc config


----------



## DutchDaemon (May 19, 2009)

No, you're not the only one, but it's working fine for me. On the other hand, I'm not trying to queue 100Mbit *and* 512Kbit on one and the same interface .. 

I have a 1Mbit upload, which is the standard/default queue. The sub queues are all set to borrow from it, and the standard default queue is set to borrow as well. 

This (extra parent sub queues) looks a bit contrived, but I'm prepared for a few extra child queues in the future, and I like to experiment with altq:


```
###################################
altq on $ext_if cbq bandwidth 1100Kb queue { std }

queue std       bandwidth 100%  priority 1      qlimit 1000     cbq(default borrow)     { acks_parent, dns_parent, ssh_parent, services_parent, http_parent, ping_parent }

queue acks_parent       bandwidth 20%   priority 7      qlimit 1000     cbq( borrow )   { acks }
queue acks      qlimit 1000     cbq(borrow)

queue ssh_parent        bandwidth 8%    priority 6      qlimit 1000     cbq( borrow ) { ssh }
queue ssh       qlimit 1000     cbq( borrow )

queue dns_parent        bandwidth 1%    priority 5                      cbq( borrow ) { dns }
queue dns       cbq( borrow )

queue services_parent   bandwidth 50%   priority 4      qlimit 1000     cbq( borrow ) { services }
queue services  qlimit 1000     cbq( borrow )

queue http_parent       bandwidth 20%   priority 3                      cbq( borrow ) { http }
queue http      cbq( borrow )

queue ping_parent       bandwidth 1%    priority 2                      cbq( borrow ) { ping }
queue ping      cbq( borrow )

#################################
```

I've assigned queues to all of my pass in / pass out rules. The only stuff that ends up in the default queue is p2p/bittorrent traffic, which gets pushed down by anything else (like ssh, http, dns, acks, etc.; all have higher priority). For example 'acks' has only 200Kbit assigned to it, but when I do some massive downloading (triggering loads of 'return acks'), it grows to 400-500Kbit without a problem. The only exception I've found is my nightly ftp upload of a backup tarball, which is in the 50% queue, but never gets more, even when there's bandwidth to borrow. It always flatlines at 512 Kbit for some reason (but it does push p2p out of the way).

Note: I just checked my pfstats and found that the ftp upload went to 604 Kbit. There was no p2p traffic at that time, so I have to assume that the receiving ftp server didn't have more bandwidth, or that my local disk simply can't push out more data than that 

That way I can fully utilise / max out my upload at any time, without stuff getting choked to death by p2p. Works fine for me.


----------



## s0xxx (May 19, 2009)

Please correct me if I'm wrong, but assigning 1100Kb to std queue, then specifying 100% for it with a borrow does not make much sense does it? It makes std (sorta) the root queue so it doesn't have anyone to borrow from. Or I am just missing something here?

```
std 
                                      |
                                      |
 acks_parent  dns_parent   ssh_parent  services_parent  http_parent  ping_parent
     |           |              |          |                 |            |
   acks         dns            ssh       services          http          ping
```


----------



## DutchDaemon (May 20, 2009)

It looks odd (and it certainly wasn't the first thing I tried to get borrowing to work), but it works fine. This construction turns std into a sort of default queue with the lowest priority, which means all the other queues (which all have higher priority) can borrow from it. The default queue itself has all the bandwidth already, so it doesn't need to borrow, but apparently _not_ setting a queue to borrow means that it doesn't borrow and *lend* ..

BTW, if you google around for cbq and borrow, you'll find a lot of confusion and irritation about borrowing not working, and I've seen that it didn't work on my own firewalls (queues stayed at their preset bandwidth, even when all other queues were set to borrow and had no traffic) until I added 'borrow' to just about everyhting. If cbq hasn't been rewritten in OpenBSD's reference implementation yet (they're a few versions ahead), it certainly should be. Unless it's down to FreeBSD porting issues, but I don't know that.


----------



## DutchDaemon (May 20, 2009)

By the way, this is the previous, simpler set-up, which simply didn't do what I wanted, though it looks much more logical ..


```
altq on $ext_if cbq bandwidth 1029Kb queue { std, acks, dns, ssh, services, http, ping, rest }
queue std bandwidth 8Kb cbq(default)
queue ssh bandwidth 128Kb priority 7 cbq(borrow)
queue acks bandwidth 128Kb priority 6 qlimit 100 cbq(borrow)
queue dns bandwidth 32Kb priority 5 qlimit 100 cbq(borrow)
queue services bandwidth 128Kb priority 4 cbq(borrow)
queue http bandwidth 64Kb priority 3 cbq(borrow)
queue ping bandwidth 16Kb priority 2 cbq(borrow)
queue rest bandwidth 525Kb priority 1 qlimit 200 cbq(borrow)
```


----------



## s0xxx (May 20, 2009)

Thanx for that, at least I know what "might" be wrong in case I run onto it. 

What about:

```
altq ... {std, other}
                std
                other (borrow)
                     ssh (borrow)
                     dns (borrow)
                     sservices (borrow)
                     ... (borrow)
```
Or even not using borrow at other queue, only on childs? I guess you already tried this one out, just curious.


----------



## DutchDaemon (May 20, 2009)

As far as I can tell from exeperimenting (profusely, I might add), a parent queue should always be set to borrow. It appears to take care of two things: borrowing from the root, lending to the children. So it sort of acts as a an inter-qeueue pipe.


----------



## syle (Aug 18, 2010)

http://forums.freebsd.org/showthread.php?t=4074

Can i see your rules, I'd like to see how you attached queue rules to the passin and passout.


----------



## DutchDaemon (Aug 18, 2010)

I'm not publishing my rules if you don't mind .. General principles can be found in other threads.

http://forums.freebsd.org/showthread.php?t=7730
http://forums.freebsd.org/showthread.php?t=11406

You basically want to put queue statements on each and every pass in/out rule.


----------



## syle (Aug 19, 2010)

Well my question had to do with why there are grandchildren.
In your queue rules, I do not understand why say there is ssh queue under ssh_parent. The point in creating ssh under ssh_parent I beleive would be to add 2 or
more rules in there where was has more priority over the other. The reason I asked for your ruleset was to try and understand why ssh would be used at all vs
ssh_parent for example.


----------



## syle (Aug 19, 2010)

On a sub note I should at least explain what I was trying to accomplish. At home I have a freebsd router, it has 3 ethernet cards, one for dsl, one for cable, one for internal network. The dsl and cable modem already have routers for them so it was just a simple assign 192.168.2.1 and 192.168.1.1 for gateways for each and nat the internal 192.168.0.0/24 network. Anyways where I have run into problems recently is with utorrent mostly because my upload bandwidth from ISP's here in canada is horrible.

For example:


```
altq on $ExtIf1 cbq bandwidth 424Kb queue { mts }
altq on $ExtIf2 cbq bandwidth 480Kb queue { shaw }
```

are dsl and cable max upload rates I tested in last week.

SO here was the problem, utorrent starts, my ping times started lagging me out. I coded a perl script to switch pf.conf rules and default route in the case one ISP went down and load balanced both connections with route-to if they are both up, but this relies heavily on ICMP working at all times and when utorrent was running my perl script was incorrently changing routes thinking a connection was down constantly.

I finally tracked down the problem to being utorrent max connections to high and connections per torrent, so I minimized it down to something like 50 and 14 and it helped alot. What was happening is when I'd start utorrent and run pftop, my connections would jump from 150 to like 600-700 or more basically saturating my upload links thus the problems with my pings.

Lowering thread counts on utorrent itself helped a great deal, but it was just a quick hack solution, I have other users in the house that won't be so forgiving or care to set those default threads lower. SO my idea was to use traffic shaping to set a default for tcp connections going out to only use 75% of the bandwidth on each connection so that my ssh, dns, http, ping connections would not be bog downed by this.

So here was my attempt based on your ruleset, but ignoring grandchildren rules you had:

```
altq on $ExtIf1 cbq bandwidth 424Kb queue { mts }
altq on $ExtIf2 cbq bandwidth 480Kb queue { shaw }

#the higher the priority, it will get served faster
queue mts bandwidth 100% priority 1 qlimit 1000 cbq(default borrow) { mts_priority1,mts_priority2 }
queue mts_priority1 bandwidth 75% priority 1 qlimit 1000 cbq(borrow)
queue mts_priority2 bandwidth 25% priority 6 qlimit 1000 cbq(borrow)

queue shaw bandwidth 100% priority 1 qlimit 1000 cbq(default borrow) { shaw_priority1,shaw_priority2 }
queue shaw_priority1 bandwidth 75% priority 1 qlimit 1000 cbq(borrow)                                  
queue shaw_priority2 bandwidth 25% priority 6 qlimit 1000 cbq(borrow)
```
I my ruleset for outgoing I thought now this would be pretty simple to do something like this for each connection:

```
#TCP
pass out on { $ExtIf1 } inet proto tcp from any to any flags S/SA modulate state queue (mts_priority1,mts_priority2) #tcp acks should have higher priority
pass out on { $ExtIf1 } inet proto tcp from any to port 53 flags S/SA modulate state queue mts_priority2             #dns just higher priority than defaul
pass out on { $ExtIf1 } inet proto tcp from any to port {80,443,8080} flags S/SA modulate state queue mts_priority2          #http and squid and ssl
pass out on { $ExtIf1 } inet proto tcp from any to port $ssh_ports flags S/SA modulate state queue mts_priority2 #ssh
pass out on { $ExtIf1 } inet proto tcp from any to port $im_ports flags S/SA modulate state queue mts_priority2 #IM
```

and so on for udp dns and whatever else i could find.

So I start up utorrent and it still bogs down my connection from all those threads utorrent creates for seeding. So I had a thought, maybe
altq only applies when a handshake is already done.

So I alter to:

```
pass out on { $ExtIf1 } inet proto tcp from any to any flags S/SA modulate state (max 200, source-track rule, max-src-states 20) queue (mts_priority1,mts_priority2)
pass out on { $ExtIf2 } inet proto tcp from any to any flags S/SA modulate state (max 200, source-track rule, max-src-states 20) queue (shaw_priority1,shaw_priority2)
```
and this seems to help greatly, at the problem that now I am limitting myself and altq seemed completely useless to this situation.
Basically this lets seeders have 20 on each connection and http dns etc have unlimited access.

So I guess my biggest question is : why did traffic shaping not help me with utorrent


----------



## DutchDaemon (Aug 19, 2010)

I've long since moved away from cbq scheduling because it simply didn't do what I needed it to. I've started using simple priq scheduling with better results.


```
###################################
altq on $ext_if priq bandwidth 764Kb queue { std, acks, toracks, ssh, dns, services, http, ping, torrent }

queue std       priority 0      qlimit 0        priq( default ecn )
queue acks      priority 15     qlimit 0
queue toracks   priority 14     qlimit 0
queue ssh       priority 13     qlimit 0        priq( ecn )
queue dns       priority 12     qlimit 0
queue services  priority 11     qlimit 0
queue http      priority 10     qlimit 0
queue ping      priority 9      qlimit 0
queue torrent   priority 1      qlimit 0
#################################
```

I also use ack queuing on *all* inbound and outbound tcp, icmp and udp rules (with torrent-related acks in their own lower category to keep them below 'real acks'). So *all* of my pass rules have a queue(one, two) format.


----------

