# PF rules won’t load on a XEN instance



## Veeny (Sep 30, 2021)

Folks,

I have created a new virtual FreeBSD machine on one of my hosting provider's servers, and upgraded it to 12.2-RELEASE-p7

I want to enforce very basic pf rules:


```
block in from any

pass inet proto tcp from any port {http, https, ssh}

# ANCHORS FOR BLACKLISTD
anchor "blacklistd" all {
        anchor "22" all
}
```

But when I try to disable/enable pf, I get the following message:


```
> sudo service pf stop
pfctl: DIOCGETSTATUS: Permission denied
> sudo service pf start
Enabling pfpfctl: DIOCADDRULENV: Operation not supported by device
pfctl: DIOCGETSTATUS: Permission denied
pfctl: pf already enabled
.
```

The network interface is xn0:


```
> dmesg | grep xn0
xn0: xbd0: <Virtual Network Interface>20480MB <Virtual Block Device> at device/vbd/51712 at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 00:16:3e:1b:69:43
xn0: backend features: feature-sg feature-gso-tcp4
xn0: 2 link states coalesced
xn0: link state changed to UP
```

I'm not sure what happens here. Apparently, the hypervisor doesn't authorize the underlying OS any access to the peripheral (promiscuous mode)?
I have, however, other instances of FreeBSD running on the same infrastructure, and pf has always worked like a charm.

Does anyone have an idea what’s going on?

Thanks a bunch,
Vincent


----------



## SirDice (Sep 30, 2021)

Veeny said:


> (promiscuous mode)


PF doesn't put the interface in promiscuous mode. So it's something else that's causing it.


----------



## Kristof Provost (Sep 30, 2021)

> Enabling pfpfctl: DIOCADDRULENV: Operation not supported by device

That's very strange. DIOCADDRULENV is a new ioctl(). A 12.2 userspace should not know to call it. Did you somehow update your userspace without updating the kernel?


----------



## Veeny (Sep 30, 2021)

SirDice said:


> PF doesn't put the interface in promiscuous mode. So it's something else that's causing it.


Well, makes sense, since pf works in pass-through mode.

I tried to google the error messages, but to no avail. DIOCADDRULENV is obviously some sort of ioctl, but why it fails is obscure.
Using the -i option doesn't make it work any better:

`> sudo pfctl -f /etc/pf.conf -i /dev/xn0
pfctl: DIOCADDRULENV: Operation not supported by device`


----------



## Veeny (Sep 30, 2021)

Kristof Provost said:


> > Enabling pfpfctl: DIOCADDRULENV: Operation not supported by device
> 
> That's very strange. DIOCADDRULENV is a new ioctl(). A 12.2 userspace should not know to call it. Did you somehow update your userspace without updating the kernel?


Ah no, I used the standard `freebsd-update -r 12.2` to upgrade the system from 12.1. Nothing fancy, pinkie promise!


----------



## Kristof Provost (Sep 30, 2021)

Veeny said:


> Ah no, I used the standard `freebsd-update -r 12.2` to upgrade the system from 12.1. Nothing fancy, pinkie promise!


Check again, because that's basically the only way this could be happening.


----------



## Veeny (Sep 30, 2021)

Kristof Provost said:


> Check again, because that's basically the only way this could be happening.




```
> uname -K -U
1202000 1202508
> sudo freebsd-update fetch
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.2-RELEASE-p10.
```


----------



## Kristof Provost (Sep 30, 2021)

I promise you, a 12.2 pctl does not call DIOCADDRULENV. It does not know it, as you can confirm by looking at the releng/12.2 branch. Somehow you wound up with a more recent pfctl on your system.


----------



## Veeny (Sep 30, 2021)

```
> strings /sbin/pfctl | grep CADD
DIOCADDADDR
DIOCADDRULENV
DIOCADDALTQ
```
I thought there was a way to figure out from what version of the code my pfctl had been compiled from, but no dice.


Kristof Provost said:


> I promise you, a 12.2 pctl does not call DIOCADDRULENV. It does not know it, as you can confirm by looking at the releng/12.2 branch. Somehow you wound up with a more recent pfctl on your system.



Oh, I don't call your word into question! I'll try and fetch pfctl directly from the ftp server and replace mine. I'll keep you posted.


----------



## SirDice (Sep 30, 2021)

Veeny said:


> but no dice.


I had nothing to do with it! 

Looking through the logs releng/13.0 doesn't have it either, stable/13 does though. 






						src - FreeBSD source tree
					






					cgit.freebsd.org
				






Veeny said:


> Oh, I don't call your word into question!


Look at the author/committer  



Veeny said:


> I'll try and fetch pfctl directly from the ftp server and replace mine. I'll keep you posted.


Is it possible somebody accidentally checked out -CURRENT in /usr/src and then built and installed sbin/pfctl.c?


----------



## Veeny (Sep 30, 2021)

SirDice said:


> I had nothing to do with it!


Lol. I know, I'm not going to drag you into this 



SirDice said:


> Looking through the logs releng/13.0 doesn't have it either, stable/13 does though.
> 
> 
> 
> ...



I don't think so, really. I mean, the first time I created this server, I upgraded to 13.0-RELEASE, but it failed ignominiously (it couldn't remount root, there were hash errors everywhere). So I deleted the server, and recreated it with a fresh 12.1 install. Is it possible I was given the same disk, simply overwritten? Might have some utilities survived from the crash/deletion/recreation? I wonder.



Veeny said:


> Oh, I don't call your word into question! I'll try and fetch pfctl directly from the ftp server and replace mine. I'll keep you posted.


With pfctl from the ftp server, it works fine. Thanks for the hint!


----------



## SirDice (Sep 30, 2021)

Veeny said:


> . I mean, the first time I created this server, I upgraded to 13.0-RELEASE, but it failed ignominiously (it couldn't remount root, there were hash errors everywhere).


That could happen. But looking through the logs 13.0-RELEASE doesn't have that change either. So something else went awry. 



Veeny said:


> So I deleted the server, and recreated it with a fresh 12.1 install. Is it possible I was given the same disk, simply overwritten? Might have some utilities survived from the crash/deletion/recreation? I wonder.


I doubt that. Even if you installed it on top of an existing install those files would have been overwritten. 


If that pfctl(8) was modified on the system there's no telling what else may have been inadvertently updated though. It might be a good idea to run `freebsd-update IDS`.


----------



## Veeny (Sep 30, 2021)

SirDice said:


> It might be a good idea to run `freebsd-update IDS`.


Almost every binary file turns out to be wrong.
I'm not sure how the system runs, but it does. PHP works, and so does NGINX. ZFS seems to work too, so…
I'm not sure what I shall do. For the time being, I'm going to left it as is. If anything goes wrong again, I will reinstall the system.

Thanks for your help!


----------



## SirDice (Sep 30, 2021)

Veeny said:


> Almost every binary file turns out to be wrong.


Yeah, I was afraid of that. 

That sort of confirms my idea of what may have happened. I suspect at some point -CURRENT was checked out in /usr/src and world was built and installed from that. It's quite easy to get the _wrong_ version of the source code if you're not familiar with the process.



Veeny said:


> If anything goes wrong again, I will reinstall the system.


Expect the unexpected to happen. And plan ahead.


----------



## Veeny (Sep 30, 2021)

SirDice said:


> Yeah, I was afraid of that.
> 
> That sort of confirms my idea of what may have happened. I suspect at some point -CURRENT was checked out in /usr/src and world was built and installed from that. It's quite easy to get the _wrong_ version of the source code if you're not familiar with the process.
> 
> ...


But I never updated from source! I used `freebsd-upgrade -r 12.2`. The update was purely made from binaries.

`> cd /usr/src
> ls -la
total 8
drwxr-xr-x   2 root  wheel  512 Sep 23 07:47 .
drwxr-xr-x  15 root  wheel  512 Sep 29 09:55 ..`


----------

