# ip_bridge with ipfw still seems bugged



## enhanced (Mar 18, 2011)

I have been trying to make this work for a while now and am having unexpected results, as follows:

FreeBSD 8.1 Release i386 setup with 3 nics.. em0 is numbered for management, em1 and em2 are physically between hostA and hostB and are members of bridge0, all unnumbered to allow for transparent bridging.

Example scenarios:

basic test

* ipfw has an allow any any rule in it:​* send icmp between hostA and hostB (I'm just gonna call these A and B) and everything looks good​
divert test (note divert and bridge are built into the kernel)

* add an ipfw divert rule, or series of for counting and testing purposes​o divert 8000 ip4 from any to any via bridge0  (this was tried with all, ip and ip4 and on em1 and em2 also)​* do not have a process yet listening on *:8000 to do something with the packets that are sent to it​* start icmp from A to B​* icmp gets through and the divert counters do not increment?​* start a simple perl script that takes the diverted packets(code below) and re-injects them​* icmp is still getting through but not hitting the perl process and divert counters still not incrementing​* down em2, icmp does not flow (perl process still running)​* bring up em2 -arp and now the perl process shows that it's receiving / transmitting the icmp packets, divert counters stop incrementing​* notable latency increase on the icmp roundtrip​* kill the perl process that the packets are flowing through, icmp continues to flow through the interfaces, divert still increments, packet latency decreases?​
I have tried playing with loads of sysctl knobs to see if that would help, different flavors of divert rules etc...  any help would be greatly appreciated.

The ultimate goal here is to have snort run inline transparently on FreeBSD.

** begin perl snippet**

```
#!/usr/bin/perl -w
use Net::Divert;

select STDERR; $| = 1;

my $divobj = Net::Divert->new('localhost',8000);

printf(STDERR "open new divobj\n");

$divobj->getPackets(\&alterPacket);

sub alterPacket { my($packet,$fwtag) = @_;
        printf(STDERR "i");
        $divobj->putPacket($packet,$fwtag);
        printf(STDERR "o");
}
```


----------



## SirDice (Mar 18, 2011)

You don't need to divert anything for snort to pick up the traffic. The traffic doesn't go _through_ snort, it just listens to it. Simply bind snort to the bridge interface and you're good to go.


----------



## enhanced (Mar 18, 2011)

And that would work fine, the purpose here is to have snort inline.. passively snort works fine as you have described, but this is for snort to be transparently inline using divert sockets (the only way on BSD that it currently should work).  Thus, allowing for drop rules to be functional.


----------



## SirDice (Mar 21, 2011)

To be honest I've never seen much use for an IPS. The problem with it is that it's prone to false positives and false negatives. False positives will drop legitimate traffic and false negatives don't detect malicious content. So you basically end up with a false sense of security.


----------



## enhanced (Mar 21, 2011)

SirDice said:
			
		

> To be honest I've never seen much use for an IPS. The problem with it is that it's prone to false positives and false negatives. False positives will drop legitimate traffic and false negatives don't detect malicious content. So you basically end up with a false sense of security.



The purpose of this thread is not to spark a discussion on the efficacy of IPS but rather troubleshoot the unexpected results that are occurring.

Having said that, in response to your above statement, IPS, as any technology, is only as good as the individual(s) that are using it.  An IPS that is blindly put inline without any intelligent ruleset consideration, event tuning and analysis performed and regular review is certainly going to perform as you had indicated.  If, however, those actions are conducted on a regular and recurring basis, an IPS can be highly effective.


----------



## osman (Mar 21, 2011)

Seems IPFW is not enabled for bridge mode.

Just do 


```
sysctl -a|grep ipfw
```


```
sysctl -a|grep bridge
```

You will find the sysctl variable.

Regards
usman


----------



## enhanced (Mar 22, 2011)

I do not believe this to be a sysctl knob issue, as noted I have played with a number of knobs and the current value of the one that you have mentioned is: net.link.bridge.ipfw: 1

I don't think that you can even load a divert rule into ipfw without that knob being on..


----------



## osman (Mar 22, 2011)

When you do
[cmd=]ipfw show[/cmd]
Do you see counter increment for any rule?


----------



## enhanced (Mar 22, 2011)

Yes, but in an, what I could call "unpredictable" way.  As noted in the original post, perhaps not clearly, let's take a simple example

I have an if_bridge and a divert rule set to divert all ip4 traffic, the next rule is to allow all ip4 traffic.. if I do not have a process running to catch the diverted traffic then the divert rule never increments but the ip4 allow all rule does (the counters).  

If I start a process that will handle the diverted traffic from the first rule, traffic still continues to flow and the divert rule never increments.  Next I shutdown one of the physical interfaces that are part of the if_bridge and traffic stops flowing (the process to handle the diverted traffic is still running), once I bring the downed interface back up, the divert ipfw rule starts to increment and traffic starts to flow, now I kill the process that should be handling the diverted packets and the traffic continues to flow but the divert rule counter stops incrementing and the allow all rule continues incrementing..


----------



## osman (Mar 22, 2011)

Very strange. I just remember once I had to tweak some sysctl values to enable IPFW to work in bridge mode but I guess it was case of dummeynet. IPFW divert is maybe not supported well in if_bridge.

Have a look at if_bridge(4).

There are some sysctl values listed under PACKET FILTERING section, they might do some trick for you.

Also have a look at this http://lists.freebsd.org/pipermail/freebsd-net/2008-March/017222.html

Regards
usman


----------



## enhanced (Mar 22, 2011)

Maybe a little progress, packets get in but not out hah

ipfw rules:

```
02001 divert 8000 ip4 from any to any
06500 allow ip4 from any to any
```

same perl code as above, and this code has been tested using divert on a single numbered interface, well, and I can still get to the box on the management em0 interface.. looks like I am not able to re-inject the packet to the correct interface of the bridge after it's diverted?  An example is it looks like the packet hits the first member interface, gets diverted then gets re-injected at the exact same rule where it is then again diverted..

I say this because if I send a single ICMP packet over the wire that packet gets duplicated over and over and over (evidenced by tcpdump on the interface, see TCPDUMP output below.  Note that there is no active ICMP traffic running at the time of the capture, but the ipfw counters continue to increment and the tcpdump traffic it is fairly evident what is happening...

Sysctl Values:

```
net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=1
```

TCPDUMP output:

```
# tcpdump -i em1 -c 100 | grep ICMP
tcpdump: WARNING: em1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
16:18:12.870177 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.870183 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.870199 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.870236 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.870825 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.870831 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.870845 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.870881 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.871398 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.871405 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.871421 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.871457 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.872010 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.872017 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.872034 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.872070 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.872603 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.872610 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.872625 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.872662 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.873185 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.873196 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.873210 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.873246 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.873796 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.873807 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.873821 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.873857 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
16:18:12.874406 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 3, length 64
16:18:12.874413 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 2, length 64
16:18:12.874429 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 1, length 64
16:18:12.874465 IP zeus > 192.168.2.38: ICMP echo request, id 2625, seq 4, length 64
100 packets captured
```


----------



## osman (Mar 22, 2011)

Wow, very interesting.
Now there can be two reasons for this behavior.

1. Script not knows how to hand over duplicate packet back to IPFW, which snort knows very well because you compile snort with IPFW support, Then duplicate packet is passed after bypassing that divert rule.

2. If this still happens with snort (which I don't think will happen), then maybe this is because every packet is hitting IPFW in every direction. Packet should hit either inbound or outbound direction of interface. As your current rule is 
	
	



```
divert 8000 ip4 from any to any
```


Regards
usman


----------



## enhanced (Mar 22, 2011)

Yes, it does it with snort also.. that's actually what I am testing with.. I provided the perl as a simple example... FYI I am @snort.org


----------



## enhanced (Mar 22, 2011)

I also tried to add directionality, see the counters below.. note that this is only seconds after initiating a single ICMP packet and only an ssh session is open to the actual numbered interface (where we will also see counters on the allow rule incrementing) and only issuing small commands...



```
02001  827692 184993272 divert 8000 ip4 from any to any in via em1
06500 1656090 370082806 allow ip4 from any to any
```


----------



## osman (Mar 23, 2011)

Disable this one net.link.bridge.pfil_bridge. And let this one stay enabled net.link.bridge.pfil_member.

if_bridge(4) syas 





> When ipfw is enabled, pfil_bridge and pfil_member will be disabled so that IPFW is not run twice; these can be re-enabled if desired.



Not sure if it's causing loop but just a try.

As per snort inline document, snort should be able to hand over packet to next IPFW rule.

Above the divert rule, also try to add a rule to skip packets coming from loop back interface direction.

Regards
usman


----------



## DutchDaemon (Mar 23, 2011)

Osman, please don't use [color] tags, use  tags: proper formatting:  http://forums.freebsd.org/showthread.php?t=8816. Also check out other tags, like http://forums.freebsd.org/misc.php?do=bbcode. Saves us both a lot of work ..() manual page" href="https://www.freebsd.org/cgi/man.cgi?query=: http://forums.freebsd.org/misc.php?do=bbcode. Saves us both a lot of work ..&sektion=&manpath=freebsd-release-ports">: http://forums.freebsd.org/misc.php?do=bbcode. Saves us both a lot of work ..()


----------



## osman (Mar 23, 2011)

Sorry Boss, really very sorry that you had to correct almost my every post. Checking the instructions. 

Thanks a lot.

Regards


----------



## enhanced (Mar 23, 2011)

osman said:
			
		

> Disable this one net.link.bridge.pfil_bridge. And let this one stay enabled net.link.bridge.pfil_member.
> 
> if_bridge(4) syas
> 
> ...



My complete ruleset looks like:

```
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
02001 divert 8000 ip4 from any to any in via em1
06500 allow ip4 from any to any
06501 allow ip from any to any
06554 deny ip from any to any
```

Thus the loopback should be allowed, yes?


----------



## osman (Mar 23, 2011)

Rule set seems OK. According to Snort Inline document for routing mode it should be able to work fine.


What happens now when Snort is not running and divert rule is enabled? Still traffic loop?


Regards
usman


----------



## enhanced (Mar 24, 2011)

The packets don't appear to replay over and over and over...

Ruleset with counts after traffic from below tcpdump:

```
# ipfw -a list
00100   0     0 allow ip from any to any via lo0
00200   0     0 deny ip from any to 127.0.0.0/8
00300   0     0 deny ip from 127.0.0.0/8 to any
02001  12  3936 divert 8000 ip4 from any to any via em1
06500 832 98490 allow ip4 from any to any
06501   6   276 allow ip from any to any
65535  85  7343 deny ip from any to any
```

65535 deny ip from any to any

Note that the dhcp request is coming from the em2 side of bridge0 and that the divert rule is on em1.. just for testing purposes.. since nothing is re-injecting the packets, we don't even see them hit the actual bridge due to the divert on em1?

*tcpdump* output:

`# tcpdump -i bridge0 -c 10`

```
tcpdump: WARNING: bridge0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), capture size 96 bytes
17:06:11.387384 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:06:14.388945 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:06:21.397758 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:06:31.406085 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:06:44.413847 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:07:04.420455 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:07:11.429250 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
```


----------



## enhanced (Mar 24, 2011)

Or I mean we don't see them make it out of the bridge because em1 is diverting all packets.. here is another test but defining directionality...

Here we can clearly see the packet making it through for the request.. and the response from the DHCP server.. the divert takes it and it goes to nowhere because *snort* is not listening.. next post will be the same but with *snort* listening...

*tcpdump* output:

```
# tcpdump -i bridge0 -c 10
tcpdump: WARNING: bridge0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), capture size 96 bytes
17:13:44.083409 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:13:44.083889 ARP, Request who-has 192.168.2.41 tell 192.168.2.1, length 46
17:13:44.568592 IP 192.168.2.1.bootps > 192.168.2.41.bootpc: BOOTP/DHCP, Reply, length 300
17:13:52.091217 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:13:52.091648 IP 192.168.2.1.bootps > 192.168.2.41.bootpc: BOOTP/DHCP, Reply, length 300
17:14:12.097849 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:14:12.098267 IP 192.168.2.1.bootps > 192.168.2.41.bootpc: BOOTP/DHCP, Reply, length 300
17:14:24.085836 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
17:14:24.086182 IP 192.168.2.1.bootps > 192.168.2.41.bootpc: BOOTP/DHCP, Reply, length 300
17:14:25.105588 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:0c:29:17:a0:53 (oui Unknown), length 300
```

ipfw rules:

```
# ipfw -a list
00100   0     0 allow ip from any to any via lo0
00200   0     0 deny ip from any to 127.0.0.0/8
00300   0     0 deny ip from 127.0.0.0/8 to any
02001  10  3280 divert 8000 ip4 from any to any in via em1
06500 420 57010 allow ip4 from any to any
06501   6   276 allow ip from any to any
65535  85  7343 deny ip from any to any
```


----------



## enhanced (Mar 24, 2011)

Removing the divert and allowing the host to get an IP, then pinging that IP from the em1 side (with *snort* running) still shows the packet infinite loop.  But removing *snort* or any process that catches the diverted packets on port 8000 does not display the infinite loop condition.


----------



## osman (Mar 24, 2011)

Yes, packets should be discarded because snort is not listening. This means IPFW is not causing infinite loop. Seems snort is injecting duplicate packet in wrong way resulting in loop.

I guess now some Snort developer might be able to help you. I have seen people saying on mailing list that its not possible because of no divert support in IPFW when in bridge mode, but IPFW seems to be diverting fine.


Regards
usman


----------



## enhanced (Mar 24, 2011)

Right, hence the purpose of the perl code (above) that takes the packet and re-injects as it was intended, but the perl exhibits the same problem


----------



## chunt (May 5, 2011)

*Any word back from the developers*

enhanced,

Did you find anything out on this?  I am in the middle of trying to pull this together myself.


----------

