# PF; traffic accounting



## _martin (Jan 22, 2012)

Hi, 

Due to some issues with my ISP I've decided to do a traffic accounting on my server. Here's what I have trouble with: 

Server is multihomed, I need to do a traffic accounting per each IP and then, in some cases, I need to do a per protocol statistics. 

Basic skeleton is as follows: 


```
ext_if="em0"

IP-PUB-1="192.0.2.1"
IP-PUB-2="192.0.2.2"

# some normalization rules

# some nat/rdr rules 

# here I just want to account the traffic and let it be decided later what to do with it
pass in on $ext_if from any to $IP-PUB-1 label ip-pub-1-in
pass in on $ext_if from any to $IP-PUB-2 label ip-pub-2-in 

block in all

#  --< additional rules here >--

# per proto per IP 
pass in quick proto tcp from <ssh_trusted> to $IP-PUB-1 port 22 keep state label pub1-ssh-in
```

I understand that rules evaluated through ip-pub-1-in/ip-pub-2-in will be processed later down trough other filtering rules (unless quick keyword specified), but I thought that traffic passed trough it would be accounted. But it's not. I see evaluations, but all 0's afterward:

`# pfctl -vsl`

```
ip-pub-1-in 635 0 0 0 0 0 0 0
```

Obviously, I'm missing something. :/


----------



## SirDice (Jan 23, 2012)

Use something like net/pfflowd and an application like net-mgmt/nfsen.


----------



## _martin (Jan 23, 2012)

SirDice said:
			
		

> Use something like net/pfflowd and an application like net-mgmt/nfsen.



Hmm .. it seems that pfflowd requires 


```
options	pfsync
```

in kernel -- something I don't have in my currently running kernel (reboot of the server might not be possible right away).

Not sure if I get it correctly but it seems it tracks pakets/traffic based on rules it (PF) tracks. That's something I've problem with in the first place.


----------



## _martin (Jan 24, 2012)

OK, so I went trough OpenBSD docs and found a way that suits me best. I made this configuration in LAB first, please do comment if you think it ain't the way to go.

First, my new /etc/pf.conf (LAB config):


```
ext_if="em0"

IP_PUB_1="172.31.1.114"
IP_PUB_2="172.31.1.214"

anchor "pub1_in" from any to $IP_PUB_1 {
        pass in quick proto tcp from any to $IP_PUB_1 port 22 keep state label pub1_ssh_in
        pass in quick proto tcp from any to $IP_PUB_1 port 80 keep state label pub1_web_in
}

anchor "pub2_in" from any to $IP_PUB_2 {
        pass in quick proto tcp from any to $IP_PUB_2 port 22 keep state label pub2_ssh_in
        pass in quick proto tcp from any to $IP_PUB_2 port 80 keep state label pub2_web_in
}

block in all
pass out all
```

Check the overall traffic on the server:

`# pfctl -vsr`

```
anchor "pub1_in" inet from any to 172.31.1.114
  [ Evaluations: 33        Packets: 937       Bytes: 95148       States: 2     ]
  [ Inserted: uid 0 pid 453 ]
anchor "pub2_in" inet from any to 172.31.1.214
  [ Evaluations: 27        Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 453 ]
block drop in all
  [ Evaluations: 27        Packets: 25        Bytes: 1252        States: 0     ]
  [ Inserted: uid 0 pid 453 ]
pass out all flags S/SA keep state
  [ Evaluations: 27        Packets: 4         Bytes: 364         States: 0     ]
  [ Inserted: uid 0 pid 453 ]
```

Check the per host traffic:

`# pfctl -vsl -a pub1_in`

```
pub1_ssh_in 21 984 100418 564 50242 420 50176
pub1_web_in 19 19 1010 10 570 9 440
```

So this does basically what I need.


----------



## SirDice (Jan 24, 2012)

It's been a while since I played with pfflowd, details are a bit blurry. But I can't remember I had to do something 'special' with my rule-set to get it to collect the correct data.

Netflow is more or less a standard way to get traffic flow information.


----------



## _martin (Jan 24, 2012)

Those tools seem good and all, but I don't need (or at least not necessarily) any graphs - I just need to see those numbers from shell to compare when something happens. 
Well those rules help to 'catch' what's needed - on-fly processing of each and every packet would be (I guess) just too expensive.

But I celebrated too early. Though these rules help me when IP address is activated on the interface (webs, jails), it doesn't work when I want to catch VirtualBox traffic.

To elaborate: one egress interface - em0 - is bridged with vboxnet0. I just need to count traffic that comes in/out from this IP address. I can see this traffic with tcpdump on em0, I thought it is possible to count it with PF too.


----------



## SirDice (Jan 24, 2012)

Try setting one of those 'counting rules' on the vboxnet0 interface. It's just an interface like all the others.


----------



## _martin (Jan 24, 2012)

SirDice said:
			
		

> Try setting one of those 'counting rules' on the vboxnet0 interface. It's just an interface like all the others.



Sorry, I forgot to mention -  did that already. I've tried both variants, either:


```
anchor "VM1-in" on $vbx_if from any to $IP_PUB_1
```

or


```
anchor "VM1-in" on $ext_if from any to $IP_PUB_1
```

and tried one without specifying interface too. Unfortunately I don't see any traffic passing this rule.


----------



## _martin (Jan 24, 2012)

Well I did couple more tests and found out that PF doesn't filter any traffic to IPs which are assigned to VirtualBox. Strange - PF has no problem filtering on bridged interfaces.

It seems that vboxnetflt kernel module might have something to do with this.


----------

