# Bridge networking and ipfw question



## Nyakov (Feb 6, 2019)

I want to run some vnets and vms on my machine so I set up bridge like this:

```
#re0 - is physical interface
cloned_interfaces="bridge0"
ifconfig_bridge0="ether xxx addm re0 SYNCDHCP"
ifconfig_re0="up"
```

Then I configured ipfw to pass some traffic through bridge0 and block everything else. Last rule is:
`$cmd 00999 deny log all from any to any`

And then I get this messages in log:

```
Feb  5 19:33:28 nyan-nas kernel: ipfw: 999 Deny UDP 192.168.1.11:23445 192.168.1.255:23445 in via re0
Feb  5 19:33:28 nyan-nas kernel: ipfw: 499 Deny UDP 192.168.1.11:23445 192.168.1.255:23445 in via bridge0
Feb  5 19:33:28 nyan-nas kernel: ipfw: 999 Deny UDP 192.168.1.11:23445 192.168.1.255:23445 in via re0
```

The question is. Why is re0 do not receive any normal packets(for the view of ipfw) but receive broadcast packets?
Is it normal\desired behavior?
Because I want for re0 to be totally transparent, like ordinary switch port.


----------



## SirDice (Feb 6, 2019)

Nyakov said:


> Why is re0 do not receive any normal packets(for the view of ipfw) but receive broadcast packets?


As far as I know the packets are passed directly to the bridge before the interface gets to process them, so the interface never really "sees" them. The broadcasts are sent out each member of the bridge (that's the nature of broadcasts) and are, in essence, fed back to the interface. 

I ran into this when I was playing with STP and bridges.


----------



## nslay (Feb 6, 2019)

Last time I had this issue, I discovered bridge's members were being filtered by the firewall by default. 

You might try adding this to /etc/sysctl.conf
net.link.bridge.pfil_member=0

I'm more confused that more isn't being blocked by ipfw, because pf was blocking wlan and lan clients from each other entirely because it had no rules defined for the member interfaces!


----------



## D-FENS (Feb 6, 2019)

What I have normally done is, I add the physical interface to the bridge, then I set the IP address on the bridge itself. Then I set the firewall rules on the bridge.
I think SirDice's remark explains why this works.

Alternatively, I think one could set the IP address on the interface itself or on a tap/tun device, but I guess in this case the firewall should pass everything on the bridge interface and not  block anything. The filtering should be done then in the bridged interfaces themselves. However, I have not tried this.


----------



## Nyakov (Feb 6, 2019)

nslay said:


> You might try adding this to /etc/sysctl.conf
> net.link.bridge.pfil_member=0



I have this sysctl state:

```
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0
```
When pfil_local_phys pfil_member and pfil_bridge set to 1 by default - make no difference for me at all. I still get broadcast packets on re0. And firewall is still working the same with no respect to this options.



roccobaroccoSC said:


> What I have normally done is, I add the physical interface to the bridge, then I set the IP address on the bridge itself. Then I set the firewall rules on the bridge.


This is exactly what I am doing. What confuses me is that ipfw gets broadcast packets from re0.



SirDice said:


> As far as I know the packets are passed directly to the bridge before the interface gets to process them, so the interface never really "sees" them. The broadcasts are sent out each member of the bridge (that's the nature of broadcasts) and are, in essence, fed back to the interface.
> 
> I ran into this when I was playing with STP and bridges.


Ok, now I begin to understand whats going on here.
Then I have some new questions.

Is there a way to tune this?
If there is not, then, is it designed behavior, or a bug? It just looks bizarre to return packet to origination interface. And in scenario of high broadcast traffic from all members of the bridge there will be unnecessary load.


----------



## SirDice (Feb 6, 2019)

Nyakov said:


> Is there a way to tune this?


Not that I'm aware of. At least for the 'standard' bridge(4) interface. You can't control where it hooks in. You might be able to change the behavior if you use netgraph(4) and ng_bridge(4) to build your bridges by hand as that would allow you to place it exactly where you need it. But I'm not too familiar with netgraph(4). I know what it is and what you could do with it, not how to use it. It's still on my very, very long list of things to learn. 



Nyakov said:


> If there is not, then, is it designed behavior, or a bug?


By design as far as I know.


----------



## D-FENS (Feb 6, 2019)

Nyakov said:


> This is exactly what I am doing. What confuses me is that ipfw gets broadcast packets from re0.



You wrote you were trying to set up some VNETs and VMs running. If your goal ist to use the host as a gateway for the VMs, you might consider the setup I prefer using:

I leave the host's network interface alone.
I create for each jail an epair and put one side inside the jail. The other side remains outside the jail.
I bridge the outside epairs' parts into a bridge (but not the physical interface).
I do routing on the main host with IPFW and the kernel NAT feature.
If you use VMs, then you can use for example TAP or TUN devices for this.

The only thing is, your VMs will not have direct access to the LAN. The traffic needs to be routed. For me it's actually a good thing.

If you DO want to bridge them to the LAN, how about this setup:

you bridge your re0 (or igb0, ...) with the TUN/TAP/EPAIR interfaces of the VMs into bridge0.
you leave bridge0 without an IP address.
you set the IP address on the re0 itself.
In the firewall config allow everything for bridge0 (it's just a pass-through level).
set your rules on re0 and the VM's interfaces as required.
Would this work?


----------



## Nyakov (Feb 6, 2019)

roccobaroccoSC said:


> you leave bridge0 without an IP address.
> you set the IP address on the re0 itself.



Is it a valid setup? Because handbook specifically says to set host IP on bridge interface.


----------



## D-FENS (Feb 7, 2019)

I don't know if it's valid. I have just tried it (with firewall down) and it works bothways:

bridge0 including em0, IP address set on em0
bridge0 including em0, IP address set on bridge0.
What does bridging do? It copies all packets frames from one interface to all bridged interfaces. So if your IP is set on the card, the bridge will copy whatever you put into it to all the other bridged interfaces.
If your IP is set on the bridge, you put the packets frames into the bridge interfaces and again - they go into all the bridged cards. The end effect is the same.

Now, you can take this setup and allow allow all ip on the "bridge0". Then you can go and tune your rules for each of the separate interfaces not worrying about the bridge.


----------



## D-FENS (Feb 16, 2019)

Nyakov said:


> Is it a valid setup? Because handbook specifically says to set host IP on bridge interface.


did it work?


----------



## Nyakov (Feb 20, 2019)

roccobaroccoSC said:


> did it work?


I used setup from handbook and just blocked unnecessary multicast in ipfw.
Cannot afford to experiment.

I still don't think physical interface should get any ip packets in to system if it does not configured to have ip stack.


----------

