# Fixing network access in Jail



## ambs (Mar 17, 2018)

I am mostly a Linux user, and my knowledge on FreeBSD is quite limited (learning now, as I need to use some FreeBSD servers). On some of them, when I create a jail, it doesn't have networking access, even if I configure it correctly (well, I would say it is correct, but if it doesn't work, something is missing).

Follows details. Not sure if I post everything needed. Let me know if I can help debugging the issue.

Jail has IP 10.20.0.13. It is correctly defined in the network interface:


```
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether xxx
        hwaddr xxx
        inet 10.20.0.1 netmask 0xffffffff broadcast 10.20.0.1
        inet 10.20.0.4 netmask 0xffffffff broadcast 10.20.0.4
        inet 10.20.0.13 netmask 0xffffffff broadcast 10.20.0.13
```

Note that some of the other configured jails have networking, so the interface should be working correctly.

My ezjail-admin config is:


```
export jail_pdfs2_hostname="pdfs2"
export jail_pdfs2_ip="bce0|10.20.0.13"
export jail_pdfs2_rootdir="/usr/jails/pdfs2"
export jail_pdfs2_exec_start="/bin/sh /etc/rc"
export jail_pdfs2_exec_stop=""
export jail_pdfs2_mount_enable="YES"
export jail_pdfs2_devfs_enable="YES"
export jail_pdfs2_devfs_ruleset="devfsrules_jail"
export jail_pdfs2_procfs_enable="YES"
export jail_pdfs2_fdescfs_enable="YES"
export jail_pdfs2_image=""
export jail_pdfs2_imagetype=""
export jail_pdfs2_attachparams=""
export jail_pdfs2_attachblocking=""
export jail_pdfs2_forceblocking=""
export jail_pdfs2_zfs_datasets=""
export jail_pdfs2_cpuset=""
export jail_pdfs2_fib=""
export jail_pdfs2_parentzfs=""
export jail_pdfs2_parameters=""
export jail_pdfs2_post_start_script=""
export jail_pdfs2_retention_policy=""
```

I have [FONT=Courier New]/etc/resolv.conf[/FONT] in the jail configured with the same IP as other working jails.

The firewall (pf) is allowing all outgoing traffic...

[FONT=Courier New]netstat -r[/FONT] result is similar for both working/non-working jails

No idea where else I should look. Thanks


----------



## pkc (Mar 17, 2018)

It may be best to show the output of `ping`, for example pinging other addresses that are assigned on your machine


----------



## ambs (Mar 17, 2018)

I can't ping because I do not have raw sockets enabled. But I am able to contact other jails using curl, for example.
But if I try to contact an external website using the domain, I get a 'can't resolve host'.
If I try to do the same by IP, the connection hangs, up to timeout.
Thanks!


----------



## pkc (Mar 17, 2018)

What are your other network interfaces besides `bce0`? And in particular is NAT being performed on some interface on this machine? In any case I would `tcpdump` the interfaces that you expect traffic from this jail to traverse on its way to the internet. Then observe what happens to the traffic.

It would probably help you that you enable raw sockets temporarily to assist in debugging.

It may also be good to provide some of the output of `netstat -r`. I am not an expert.


----------



## ambs (Mar 17, 2018)

It does not help not being the person responsible for the machine configuration. I just joined recently, and need to understand how things are configured.
As far as I could understand, googling around, NAT is not enabled (I can't find anything similar to what is described here: https://www.freebsd.org/doc/handbook/firewalls-ipfw.html).
From outside, HTTP is being redirected using squid.

Other interfaces: bce1, igb0, igb1, pflog0 and two loop devices (lo0 and lo1). I tried to configure one jail following ezjail-admin documentation, creating a loop device (lo1) but the jail ended up without external access. bce1 is also configured, but with an IP in other group (10.0.0.X).

My experience with  tcpdump is not much. Tried this:

```
@Host:
sudo tcpdump  -i bce0 host XX.XX.XX.XX   # public IP of other machine, other continent
@Jail:
telnet XX.XX.XX.XX  22
```
and tcp dump shows nothing...


----------



## ambs (Mar 21, 2018)

Got some time to continue in my fight with this jail. I enabled icmp.

From the jail, I can ping the host:

```
[root@pdfs2 /]# ping 10.20.0.1
PING 10.20.0.1 (10.20.0.1): 56 data bytes
64 bytes from 10.20.0.1: icmp_seq=0 ttl=64 time=0.039 ms
```

Also, traceroute works, as expected

```
[root@pdfs2 /]# traceroute 10.20.0.1 
traceroute to 10.20.0.1 (10.20.0.1), 64 hops max, 40 byte packets
 1  10.20.0.1 (10.20.0.1)  0.046 ms  0.028 ms  0.019 ms
```

But I can't ping or traceroute anything outside:

```
[root@pdfs2 /]# ping  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
[root@pdfs2 /]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  *^C
```

My netstat routing info inside the jail:

```
[root@pdfs2 /]# netstat -r
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
pdfs2.research.an link#1             UHS         lo0
```

note that the destination name is the hostname of the jail, defined in `/etc/hosts`

pf, in the host, has skip for this network interface.

by the way, netstat routing info in the host:


```
[ambs@research ~]$ netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.0.1           UGS        bce1
10.0.0.0/24        link#2             U          bce1
research           link#2             UHS         lo0
...
```

and I still couldn't find any difference in the config of the two jails, the existing one that has net access, and the one I created, which doesn't.


----------



## tokred (Mar 22, 2018)

You posted ifconfig of bce0 but only routes regarding bce1 - the whole picture would make things easier. 

I am not very experienced with jails myself, but I would guess that you are missing NAT of your jail IP via the machine's IP. Without this, your jails sends out packets coming from 10.20.0.13, so you would need to tell your gateway that your host is responsible for 10.20.0.13 (or /24) in order to route answer traffic back to you.

Best regards


----------



## gkontos (Mar 22, 2018)

Your jails seem to bound in bce0. However, your gateway appears to be bce1. It would help if you could post the full output of `#ifconfig`.

Most likely you are missing a NAT rule. Something like that should work:


```
nat pass on bce1 from 10.20.0.0/24 to any -> bce1
```


----------



## ambs (Mar 22, 2018)

Sorry for not posting the complete ifconfig.


```
[ambs@research ~]$ ifconfig -a
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:32:61:12
        hwaddr 78:2b:cb:32:61:12
        inet XXX.XX.XXX.XXX netmask 0xffffff00 broadcast XXX.XX.XXX.XXX
        inet 10.20.0.1 netmask 0xffffffff broadcast 10.20.0.1
        inet 10.20.0.4 netmask 0xffffffff broadcast 10.20.0.4
        inet 10.20.0.3 netmask 0xffffffff broadcast 10.20.0.3
        inet 10.20.0.5 netmask 0xffffffff broadcast 10.20.0.5
        inet 10.20.0.6 netmask 0xffffffff broadcast 10.20.0.6
        inet 10.20.0.10 netmask 0xffffffff broadcast 10.20.0.10
        inet 10.20.0.11 netmask 0xffffffff broadcast 10.20.0.11
        inet 10.20.0.12 netmask 0xffffffff broadcast 10.20.0.12
        inet 10.20.0.14 netmask 0xffffffff broadcast 10.20.0.14
        inet 10.20.0.16 netmask 0xffffffff broadcast 10.20.0.16
        inet 10.20.0.13 netmask 0xffffffff broadcast 10.20.0.13
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:32:61:13
        hwaddr 78:2b:cb:32:61:13
        inet 10.0.0.241 netmask 0xffffff00 broadcast 10.0.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:9b:ea:fc
        hwaddr 00:1b:21:9b:ea:fc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:9b:ea:fd
        hwaddr 00:1b:21:9b:ea:fd
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.1.1 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        groups: lo
```


----------



## ambs (Mar 22, 2018)

gkontos said:


> Your jails seem to bound in bce0. However, your gateway appears to be bce1. It would help if you could post the full output of `#ifconfig`.
> 
> Most likely you are missing a NAT rule. Something like that should work:
> 
> ...



Added this to pf, but no joy  Thanks


----------



## pkc (Mar 23, 2018)

Sorry, I might be able to help if it were `ipfw`. I would still say you would want to do something like:



```
# tcpdump -i bce0 host x.x.x.x and icmp
```

and


```
# tcpdump -i bce1 host x.x.x.x and icmp
```

where `x.x.x.x` is the IP of the jail in question. Then see where the traffic goes. Try pinging a few different addresses, as you already have done, and observe the differences (when does it stop working?). For example, if your default gateway is through `bce1`, you will expect to see the outgoing pings for `8.8.8.8` going through it on tcpdump.

Keep in mind you can always provide the `-S` option to `ping` to explicitly provide the source address you intend (ie the x.x.x.x above) if ever in doubt.


----------



## ShelLuser (Mar 23, 2018)

What does `# pfctl -s nat` show you?


----------



## SirDice (Mar 30, 2018)

Your jails are bound to bce0 but the network is connected to bce1. There's no need for NAT but you do need to enable routing in order to route traffic between the interfaces. The rest of your network also needs to know the route (back) to 10.20.0.0/24.


----------



## gkontos (Mar 30, 2018)

Correct, assuming that 10.0.0.1/32 the gateway, maybe his Internet router, is configured to perform NAT for 10.20.0.0/24, then all he needs is a static route there.

I think that for his FreeBSD box, the only thing he needs is a `gateway_enable="YES"`.


----------

