# Cannot connect to internet from within the jail



## fnoyanisi (Feb 28, 2018)

Hi there,

I have an Amazon AMI running FreeBSD 11.1-RELEASE. I have only one public IP address and I want to setup several jails to host different domains (each jail to accomodate one domain).

On each jail, I have www/nginx running as the web server. The host machine also runs a www/nginx daemon which handles http requests and acts as the reverse-proxy managing all the internal traffic redirections based on the http address.

My domain is registered to my public IP address, so when I type my address into my browser's address bar, I can see my web page hosted on the jail (so nginx reverese proxy on the host AMI works). However, I do not have internet access from within the jail.

Previously, I had pf(4) without a revere proxy, so it was working fine for a single jail. How can I connect my jails to internet by keeping www/nginx as my reverse proxy?

Thanks

PS : before deploying this configuration on my AMI, I tried this on a FreeBSD VM (being the host), and created two jails in it) and it worked fine.

Below is my rc.conf

```
ec2_configinit_enable=YES
ec2_fetchkey_enable=YES
ec2_ephemeralswap_enable=YES
ec2_loghostkey_enable=YES
firstboot_freebsd_update_enable=YES
firstboot_pkgs_enable=YES
growfs_enable="YES"
ifconfig_DEFAULT="SYNCDHCP accept_rtadv"
sshd_enable="YES"
firstboot_pkgs_list="awscli"
ipv6_activate_all_interfaces="YES"
dhclient_program="/usr/local/sbin/dual-dhclient"

####################################
# FNI
####################################

# Disable sendmail
sendmail_enable="NO"
sendmail_msp_queue_enable="NO"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"

# jails
cloned_interfaces="lo1 lo2"
ezjail_enable="YES"

# nginx as reverse proxy
nginx_enable="YES"
```

and here is ifconfig(8) output

```
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
          options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
         inet6 ::1 prefixlen 128 
         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
         inet 127.0.0.1 netmask 0xff000000
         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
         groups: lo 
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
         options=503<RXCSUM,TXCSUM,TSO4,LRO>
         ether 02:b6:74:a8:15:86
         hwaddr 02:b6:74:a8:15:86
         inet6 fe80::b6:74ff:fea8:1586%xn0 prefixlen 64 scopeid 0x2 
         inet 172.31.2.11 netmask 0xfffff000 broadcast 172.31.15.255 
         inet 192.168.1.51 netmask 0xffffffff broadcast 192.168.1.51 
         inet 192.168.1.52 netmask 0xffffffff broadcast 192.168.1.52 
         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
         media: Ethernet manual
         status: active
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
         options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
         inet 127.0.1.1 netmask 0xffffffff 
         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
         groups: lo 
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
         options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
         inet 127.0.1.2 netmask 0xffffffff 
         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
         groups: lo
```

and below is nginx.conf on the host machine

```
worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /usr/local/www/default;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }
    }


  server { 
  server_name mydomain.com;

  location / {
    # app1 reverse proxy follow
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_pass http://192.168.1.51:8080;
  }
}
```


----------



## khuman (Feb 28, 2018)

Can you give us your pf.conf and nginx.conf for jail's nginx?


----------



## SirDice (Feb 28, 2018)

Note that the nginx reverse proxy is for _incoming_ (from outside to internal) traffic. If your jails need to access the outside world you'll need to set up NAT for them or also configure nginx to proxy _outgoing_ traffic. But note that the nginx proxy is limited, you can't use it for SMTP for example. So it's going to depend on what kind of access the jails require.

And you don't need to create a new lo interface for each jail, you can attach multiple jails to lo1.


```
dice@armitage:~ % ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.21.1 netmask 0xffffff00
        inet 192.168.21.3 netmask 0xffffff00
        inet 192.168.21.5 netmask 0xffffff00
        inet 192.168.21.10 netmask 0xffffff00
        inet 192.168.21.4 netmask 0xffffff00
        inet 192.168.21.20 netmask 0xffffff00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```


```
root@armitage:~ # ezjail-admin list
STA JID  IP              Hostname                       Root Directory
--- ---- --------------- ------------------------------ ------------------------
ZR  1    192.168.21.3/24 webtrees.sirdice.nl            /jails/webtrees
ZR  2    192.168.21.5/24 phabricator                    /jails/phabricator
ZR  3    192.168.21.10/24 mysql.sirdice.nl               /jails/mysql
ZR  4    192.168.21.4/24 mail.sirdice.nl                /jails/mail
ZR  5    192.168.21.20/24 gamez.sirdice.nl               /jails/gamez
```

And the essential parts from pf.conf:

```
ext_if="vtnet0"
int_if="lo1"
int_net="192.168.21.0/24"

# Outgoing traffic from jails to the internet:
nat on $ext_if from $int_net to any -> ($ext_if:0)
```

On the host I use net/haproxy as a reverse proxy so none of my jailed websites are directly accessible. The basic idea is the same as for using nginx as a reverse proxy.


----------



## dch (Feb 28, 2018)

+1 for haproxy


----------



## fnoyanisi (Feb 28, 2018)

khuman , the nginx.conf used by the jail is below

```
user  www;
worker_processes  1;
error_log /var/log/nginx/error.log info;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log /var/log/nginx/access.log;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       8080;
        server_name  mydomain.com;
        root /usr/local/www/wordpress;
        index index.php index.html index.htm;

        location / {
            try_files $uri $uri/ /index.php?q=$uri&$args;
        }

        error_page      500 502 503 504  /50x.html;
        location = /50x.html {
            root /usr/local/www/nginx-dist;
        }

        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/var/run/php-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $request_filename;
                include fastcgi_params;
        }
    }
}
```

I do not have pf(4) running at all and I only have http access to my jails from outside world.
SirDice , is there another pf.conf line for the incoming traffic from internet to jails? I most probably will be looking to use SMTP on one of the jails n the future.

I am not religious about www/nginx, I can give a go to net/haproxy.

Thanks


----------



## SirDice (Feb 28, 2018)

fnoyanisi said:


> @SirDice , is there another line pf.conf for the incoming traffic from internet to jails?


There's only a rule to allow HTTP(S) traffic to HAProxy. There are no redirects (not for the web jails at least).


----------



## vejnovic (Feb 28, 2018)

fnoyanisi said:


> My domain is registered to my public IP address, so when I type my address into my browser's address bar, I can see my web page hosted on the jail (so nginx reverese proxy on the host AMI works). However, I do not have internet access from within the jail.


I have similar problem. You could look `/etc/resolv.conf` in your jail and may add name server.


----------



## fnoyanisi (Mar 1, 2018)

vejnovic said:


> I have similar problem. You could look `/etc/resolv.conf` in your jail and may add name server.


Thanks! I have a nameserver configured in fact.


----------



## fnoyanisi (Mar 2, 2018)

Thanks, it works now!

In the outgoing case, if pf is not running, how do the http response packets originated from the jail are handled?

In the case of pf, I would assume the packet is captured by pf after it is created and passed to pf, and NAT takes care of the rest.


----------

