# RPI3 B some issues with jails



## DiscmanDaemon (Apr 30, 2018)

Hello all,
I have been experimenting with running a project that works great on AMD64/X86 on an RPI3 B, and have had some hiccups with jails, and I am wondering if anyone has experienced anything similar, and has any idea as to the cause.

I've noticed the standard method of NATting jails on a loopback to the LAN does not seem to work (as covered here). Interestingly, when I tried enabling raw sockets, I can ping, and traceroutes with IMCP, but anything with tcp seems to go nowhere. I do have a dns set in /etc/resolv.conf

Interestingly, if I directly link a jail to the external interface "ue0|some.in-network.ip.address" everything works as expected, so I'm sure something must be up with the NATting.

Here is my /etc/pf.conf

```
ext_if="ue0"
jail_if="lo1"
IP_PUB="192.168.1.233"
NET_JAIL="10.0.0.0/24"
scrub in all
# nat all jail traffic
nat pass on $ext_if from $NET_JAIL to any -> $IP_PUB

# passing everything for test
pass out
pass in
```

Output from ifconfig

```
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
        ether b8:27:eb:d3:80:33
        hwaddr b8:27:eb:d3:80:33
        inet 192.168.1.235 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.0.0.1 netmask 0xffffffff
        inet 10.0.0.10 netmask 0xffffffff
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

If anyone has any suggestions for debugging or remedying this issue, it would be greatly appreciated! it should be noted this exact same setup works great on X86/AMD64 architecture boxes


----------



## SirDice (May 1, 2018)

The IP addresses don't line up. In your pf.conf you're using 192.168.1.233 but the interface has address 192.168.1.235.

To make this more dynamic use this instead:

```
nat pass on $ext_if from $NET_JAIL to any -> ($ext_if)
```


----------



## Kristof Provost (May 1, 2018)

It might be a good idea to tcpdump / wireshark and see if you can work out what's wrong with TCP. I've seen reports of issues with pf and TCP checksums in combination with certain network cards, but I've never been able to get my hands on affected hardware.


----------



## DiscmanDaemon (May 2, 2018)

SirDice said:


> The IP addresses don't line up. In your pf.conf you're using 192.168.1.233 but the interface has address 192.168.1.235.


My bad on the cut-and-paste. I have this same issue on 2 seperate RPIs and they have 233 and 235 as their addresses. 

I am beginning to wonder if this is caused by a mismatch between the world in the jails and the kernel on the pi as they are different versions. I am going to try building world specifically for the version of FreeBSD that is on the RPI and use that as the basejail and see if we get a different result as I have exhausted ideas on PF (I did take a look with wireshark and found nothing useful-wireshark outside the jail showed no sign of the packets and wireshark inside the jail just showed the packets not leaving the jail)

If this is indeed the case, this thread should be moved back to embedded since it would have nothing to do with the PF.


----------



## SirDice (May 2, 2018)

DiscmanDaemon said:


> I am beginning to wonder if this is caused by a mismatch between the world in the jails and the kernel on the pi as they are different versions.


That shouldn't be much of a problem as long as the host system's version is higher than then the jail's version. It's perfectly acceptable to have a 11.1-RELEASE host with a 10.3-RELEASE jail. The other way around however (jail is newer than host) is going to cause problems.


----------

