# Network unreachable inside jail



## matthewp (Mar 10, 2015)

I have a jail which can't connect to the internet for some reason. Another jail is fine. Their config seems identical to me.

```
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
127.0.1.2          link#3             UH          lo1
192.168.1.51       link#1             UHS         lo0
```
Additionally this seems weird:

```
# service routing stop
route: writing to routing socket: Operation not permitted
route: writing to routing socket: Operation not permitted
route: writing to routing socket: Operation not permitted
route: writing to routing socket: Operation not permitted
ifconfig: socket(family 28,SOCK_DGRAM: Protocol not supported
```


----------



## sidetone (Mar 10, 2015)

The jail's config ip has to match the alias ip in the base system. This is set in the alias line in the base system's rc.conf.

Inside the jail check that you have /etc/resolv.conf and /etc/hosts matching what you have in the other jail and basesystem.

Check the config file /usr/local/etc/jails/jailnamefile with the one in your other jail: this is where much of the jail's options are set.

According to the error message above see:
jail(8) jail.conf settings

```
Jail Parameters
allow.*
  allow.raw_sockets
  The jail root is allowed to create raw sockets.  Setting
  this parameter allows utilities like ping(8) and
  traceroute(8) to operate inside the jail.  If this is
  set, the source IP addresses are enforced to comply with
  the IP address bound to the jail, regardless of whether
  or not the IP_HDRINCL flag has been set on the socket.
  Since raw sockets can be used to configure and interact
  with various network subsystems, extra caution should be
  used where privileged access to jails is given out to
  untrusted parties.
```


----------



## perkypork (Mar 10, 2015)

You shouldn't enable allow.raw_sockets, it creates security issues. Just use `host google.com` to test networking inside the jail.


----------



## asteriskRoss (Mar 10, 2015)

matthewp: When you say your jail cannot connect to the Internet, what have you tried?  What did you expect and what did you get?  Have you configured name resolution in your jail (/etc/resolv.conf)?



matthewp said:


> Additionally this seems weird.
> 
> ```
> # service routing stop
> ```


Were you running that command inside the jail?  Unless you are using a vnet jail (you will know if you are, otherwise you aren't), routing is managed on the host system.


----------



## matthewp (Mar 10, 2015)

asteriskRoss said:


> matthewp: When you say your jail cannot connect to the Internet, what have you tried?  What did you expect and what did you get?  Have you configured name resolution in your jail (/etc/resolv.conf)?



I'm trying to pkg install stuff.  I do have /etc/resolv.conf that makes the other jail (and the host).




> Were you running that command inside the jail?  Unless you are using a vnet jail (you will know if you are, otherwise you aren't), routing is managed on the host system.



I ran it inside the jail but I don't know if it should work or not, it's just something I tried, let's ignore that one for now.


----------



## matthewp (Mar 10, 2015)

Thanks everyone, I'm testing with `host google.com`.  It does work, strangely. But nothing else seems to work. When I try to pkg install I get:

```
# pkg install git
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/freebsd:10:x86:64/latest/Latest/pkg.txz: Network is unreachable
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
```


----------



## matthewp (Mar 10, 2015)

Here is my host's rc.conf:

```
hostname="webserver"
# ifconfig_em0="inet 192.168.1.150 netmask 255.255.255.0"
# ifconfig_em0_ipv6="inet6 accept_rtadv"
ifconfig_re0="inet 192.168.1.152 netmask 255.255.255.0"
ifconfig_re0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
# ezjail_enable="YES"
jail_enable="YES"
cloned_interfaces="${cloned_interfaces} lo1"
```


----------



## matthewp (Mar 10, 2015)

sidetone said:


> The jail's config ip has to match the alias ip in the base system. This is set in the alias line in the base system's rc.conf.



I don't have an alias stuff in my host rc.conf, is there docs on this?



> Inside the jail check that you have /etc/resolv.conf and /etc/hosts matching what you have in the other jail and basesystem.



I did check those, they are the same.



> Check the config file /usr/local/etc/jails/jailnamefile with the one in your other jail: this is where much of the jail's options are set.



I don't have a /usr/local/etc/jails folder at all. I initially setup my jails with ezjail but am not using that to manage them now.  I am using an /etc/jail.conf file.


----------



## asteriskRoss (Mar 10, 2015)

As far as documentation goes, there is the chapter on jails in the FreeBSD handbook, and the man pages for jail(8), jail.conf(5) and rc.conf(5).

Your migration from sysutils/ezjail to managing those jails outside the ezjail framework is a possible source of issues.  ezjail sets things up in a particular way, using nullfs(5) to share the guts of what is needed for a jail between all jails and sticking the unique bits on top.  It also manages much of the configuration for you, like assigning the appropriate aliases to the desired network interfaces.  I suspect it is this configuration that ezjail was handling for you that you are now missing.  Can you run `ifconfig` in both the host and the jail and post the outputs?

If you understood exactly how it all worked when you changed to using /etc/jail.conf then it could work, but in my view would be a strange choice.  Section 15.5 of the FreeBSD handbook describes how to set something up by hand that is a bit like the ezjail configuration and might be a better place to start if that is what you want.

sysutils/ezjail is excellent and if you're just starting out with jails I would really recommend using it.


----------



## wblock@ (Mar 10, 2015)

It would be helpful to see the jail.conf entries for both the working and non-working jails.


----------



## matthewp (Mar 12, 2015)

Ok, I'll try using ezjail. I originally migrated away from it because it hasn't been updated in over a year and a half, but maybe I should use it anyways.


----------



## matthewp (Mar 12, 2015)

Ok, deleted and recreated the bad jail with ezjail and see the same issue.  

So now I find out that the problem is the host!  I'm not able to update pkg there.  However one of my jails is serving a website so I know there is some network.  Here's what happens when I try to install curl from the host.


```
root@webserver:~ # pkg install curl
Updating FreeBSD repository catalogue...
pkg: http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest/meta.txz: Network is unreachable
pkg: repository FreeBSD has no meta file, using default settings
pkg: http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest/packagesite.txz: Network is unreachable
pkg: Unable to update repository FreeBSD
All repositories are up-to-date.
The following 2 packages will be affected (of 0 checked):

New packages to be INSTALLED:
    curl: 7.41.0
    ca_root_nss: 3.17.4_1

The process will require 5 MiB more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y
pkg: http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest/All/curl-7.41.0.txz: Network is unreachable
```


----------



## sidetone (Mar 12, 2015)

The problem is in /etc/resolv.conf or /etc/hosts. Also, check that your gateway is listed in /etc/rc.conf. If it's wireless and not this, it could be wpa_supplicant's configuration, which I doubt.

Ping different things, from your gateway to outside the gateway with both the numerical ip and web address.


----------



## matthewp (Mar 12, 2015)

Sorry for the ignorance, my gateway is my router, correct?


----------



## matthewp (Mar 12, 2015)

Ah, I didn't have a default gateway. Adding that seems to have fixed everything.... knock on wood. Thanks everyone for being patient and helping me.


----------

