# How to connect a FreeBSD jail to the internet? (Amazon EC2)



## phil5 (Feb 23, 2019)

I am running a FreeBSD EC2 instance with a single jail.

In a regular homebrew setup, I would create a jail in "shared IP" (iocage) mode, and configure my router to forward various ports to the jail. In EC2, I am not sure whether to share the public IP or assign a new one.

I created a new Network Interface and Elastic IP, but could not get the jail to ping or telnet anything, not even the host. Which makes me suspect that the jail's internal config is wrong. The host could however ping the jail (which still does not mean that the Elastic IP is the right approach...)

Any ideas? (I posted the above in the AWS forums.)  


In addition, I wonder if the jail approach is necessary in EC2? The instance's sole purpose is a webserver for Wordpress.

In my present setup (which I am attempting to migrate to AWS), the host is useful for running various cron jobs which implement my backup solution.

Any suggestions on both issues is greatly appreciated.


----------



## phil5 (Feb 23, 2019)

I let go of the added Network Interface/Elastic IP approach and added a secondary IP (same subnet) to my instance. 

With a vanilla iocage jail I can now resolve addresses but get no response. I think I need to configure NAT on the host to enable port forwarding?

I still would like to have separate public IPs for the host and jail, however.


----------



## alexseitsinger (Feb 23, 2019)

With sysutils/ezjail all that's needed is:

/etc/rc.conf
ifconfig_<nic>_alias="<address>/<subnet>"
(Making sure that the IP is in the subnet provided by the router.)

cp /etc/resolv.conf /usr/jails/<jail_name>/etc/resolv.conf
(Copying the official resolv.conf to the jail)

/usr/local/etc/ezjail/<jail_name>
export jail_<jail_name>_parameters="allow.raw_sockets=1"
(Allow things like ping)

And then I'm able to use the internet from the jail.


----------



## alexseitsinger (Feb 23, 2019)

Since youre using a virtual server, I think you will need to add a non-routable address to the jail, and then NAT the traffic from it using PF.


----------



## phil5 (Feb 24, 2019)

Thanks. I finally got it working, sort of. My pf rule was wrong. 

I tested 3 ways:

With a standard cloned loopback interface with pf rule (not particular to EC2/AWS)
By assigning a Secondary private IPv4 IP to my instance's network interface and substituting that IP into the pf rule
Same as (2.) but with Elastic IP associated to the secondary IP

*However*, I still cannot connect my jail to the internet thru a second network interface (xn1):

`# ifconfig`

```
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9001
        options=503<RXCSUM,TXCSUM,TSO4,LRO>
        ether 12:ae:db:a2:0c:68
        inet6 fe80::10ae:dbff:fea2:c68%xn0 prefixlen 64 scopeid 0x2
        inet 172.31.91.191 netmask 0xfffff000 broadcast 172.31.95.255
        media: Ethernet manual
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
xn1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9001
        options=503<RXCSUM,TXCSUM,TSO4,LRO>
        ether 12:d2:14:4a:a7:58
        inet6 fe80::10d2:14ff:fe4a:a758%xn1 prefixlen 64 scopeid 0x3
        inet 172.31.81.75 netmask 0xffffffff broadcast 172.31.81.75
        media: Ethernet manual
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
```

I tried the following rule (does not work):

`# cat /etc/pf.conf`

```
ext_if="xn1"
jail_net="172.31.81.75/32"

nat pass on $ext_if from $jail_net to any -> $ext_if

pass out
pass in
```

Which is based on my original rule for the loopback (127.1.0.0/24 on lo1, not shown in ifconfig above):

`# cat /etc/pf.conf`

```
ext_if="xn0"
jail_net="127.1.0.0/24"

nat pass on $ext_if from $jail_net to any -> $ext_if

pass out
pass in
```


----------



## alexseitsinger (Feb 24, 2019)

> nat pass on $ext_if from $jail_net to any -> $ext_if



I might be wrong about this, but I believe the syntax is:

```
nat on $ext_if from $jail_net to any -> ($ext_if)
```


----------



## phil5 (Feb 24, 2019)

alexseitsinger said:


> I might be wrong about this, but I believe the syntax is:
> 
> ```
> nat on $ext_if from $jail_net to any -> ($ext_if)
> ```


I'm not sure either, but from the handbook:


> The parentheses surrounding the last part of the nat rule ($ext_if) is included when the IP address of the external interface is dynamically assigned.



All IPs are static in this case (I believe).

Although, I suspect the rule is wrong since NAT shouldn't be necessary-- the interface (xn1) is associated to a public IP.


----------



## phil5 (Feb 24, 2019)

Same problem: Multiple public network interfaces not accessable on Amazon EC2/VPC

TL;DR: Issue stems from both network interfaces sharing the same subnet.

Workaround (linked at the bottom): Get Two Public IPs on an Amazon EC2 Instance for Free

This workaround is the same as no. 3 mentioned in post #5 of this thread.

Not really a solution in this case, since the goal is to set different security groups for each IP, hence the second network interface.

Ref:
Elastic IP Addresses https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html#working-with-multiple-ipv4
Elastic Network Interfaces https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html


----------



## alexseitsinger (Feb 25, 2019)

It sounds like the server might have it's outbound connections restricted to a single specific interface/NIC/IP and all other connection attempts are dropped by default. I've known other PaaS providers to do something like this. Maybe contact their support team to confirm this to be sure.


----------



## Hei (Oct 8, 2020)

So I ran into the same issue and I think I found a solution. Not sure if phil5 has solved this. I still leave my solution here just in case anyone runs into the same issue in the future.

I had an ec2 instance with 2 network interfaces. One (xn0) of them was associated with a public IP while the other one (xn1) wasn't. I created a jail with sysutils/ezjail which used xn1 (and also lo1), and set up the NAT rule just like phil5 did. But the jail was not able to connect to the internet through xn0. I then monitored the traffic using sysutils/pftop and figured the source address of the packets was 127.0.1.1, which obviously did not match the NAT rule.

I then went to /usr/local/etc/ezjail/<jail_name> and swapped the order of the IPs to see if it does anything.

i.e. Changed from 
	
	



```
export jail_test_ip="lo1|127.0.1.1,xn1|xxx.xxx.xxx.xxx"
```
 to 
	
	



```
export jail_test_ip="xn1|xxx.xxx.xxx.xxx,lo1|127.0.1.1"
```

The jail is now able to access the internet, though I still do not understand why the order of the IPs matters. I was always putting lo1 before any other interfaces when I was using jails. Not sure why things worked differently on ec2. Would really appreciate if someone can give some insight on that.


----------

