# Internet access from jails



## rinalds (Mar 15, 2009)

I'm using jails to isolate Apache virtual hosts, which works great. The only problem is that internet access *from* the jails does not work well. For example:



> jail# telnet freebsd.org 80
> *...spends about 10 seconds to resolve the name...*
> Trying 69.147.83.40...
> *...spends about a minute here...*
> telnet: connect to address 69.147.83.40: Operation timed out



However, the strange thing is that sometimes it manages to connect after some 5-10 seconds. Connection to server's public IP address works without problems.

If I try to run this through truss, I get this:



> ....
> __sysctl(0x7fffffffe7c0,0x4,0x80152e0f0,0x7fffffffe868,0x0,0x0) = 0 (0x0)
> socket(PF_INET6,SOCK_DGRAM,17)			 ERR#43 'Protocol not supported'
> socket(PF_INET,SOCK_DGRAM,17)			 = 3 (0x3)
> ...



Jail's /etc/rc.conf:


> hostname="support.livecart.com"
> network_interfaces=""



Host's /etc/rc.conf:


> defaultrouter="67.220.195.137"
> hostname="localhost.localdomain"
> ifconfig_fxp0="inet 67.220.195.138  netmask 255.255.255.248"
> ifconfig_fxp0_alias0="inet 67.220.195.139 netmask 255.255.255.255"
> ...



/etc/pf.conf:


> nat on fxp0 from lo1:network to any -> (fxp0)



/etc/resolv.conf (same for jail and host):


> nameserver 206.251.73.9
> nameserver 4.2.2.1



sysctl -a | grep jail


> security.jail.jailed: 0
> security.jail.mount_allowed: 0
> security.jail.chflags_allowed: 0
> security.jail.allow_raw_sockets: 1
> ...



Any suggestions would be appreciated.


----------



## anomie (Mar 17, 2009)

rinalds said:
			
		

> Any suggestions would be appreciated.



Run tcpdump(8) in a separate terminal (from inside the jail) to try to determine where it is lagging. Is it taking a long time to send the SYN packet? Is it sending the SYN right away but waiting a long time to receive the SYN-ACK? Or something else? 

`# tcpdump port 80`

... and then: 

`#  nc -zvw 1 [url]http://www.freebsd.org[/url] 80`

Post the results here in code tags, and try to explain at which stages in the output the lagging occurred. 

(P.S. telnet is just fine, but I strongly prefer nc or nmap for this sort of testing, which I why I put it in my example.)


----------



## SirDice (Mar 17, 2009)

Can you explain why you're using lo1 and NAT? Why not bind the jail directly to the external IP?


----------



## rinalds (Mar 17, 2009)

Thanks for your help  Strangely, I'm getting mixed results.

Attempt 1:



> jail# nc -zvw 1 http://www.freebsd.org 80
> nc: connect to http://www.freebsd.org port 80 (tcp) failed: Operation timed out





> host# tcpdump port 80 | grep freebsd
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on fxp0, link-type EN10MB (Ethernet), capture size 96 bytes
> 14:50:32.192577 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 85318051 0>
> ...



Attempt 2:


> jail# nc -zvw 1 http://www.freebsd.org 80
> Connection to http://www.freebsd.org 80 port [tcp/http] succeeded!





> host# tcpdump port 80 | grep freebsd
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on fxp0, link-type EN10MB (Ethernet), capture size 96 bytes
> 15:22:29.129832 IP 67-220-195-139.hosted.static.webnx.com.61577 > http://www.freebsd.org.http: S 1423380358:1423380358(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 87208838 0>
> ...



Sometimes it connects, but usually it doesn't. I noticed that the successful connect logged address resolved from the public IP (67-220-195-139.hosted.static.webnx.com), while the failed attempt showed 192.168.0.1. No configuration was changed in between the attempts. I tried adding 192.168.0.1 to /etc/hosts, but it didn't help.

Interestingly, when I try to ping the resolved address, I get this:


> host# ping 67-220-195-139.hosted.static.webnx.com
> ping: cannot resolve 67-220-195-139.hosted.static.webnx.com: Unknown host





> Can you explain why you're using lo1 and NAT? Why not bind the jail directly to the external IP?



There are a lot more virtual hosts than I have public IPs available. I'm using nginx to proxy the http requests to Apache instances on local IPs.


----------



## SirDice (Mar 18, 2009)

rinalds said:
			
		

> There are a lot more virtual hosts than I have public IPs available. I'm using nginx to proxy the http requests to Apache instances on local IPs.


Sounds reasonable :e

I have been looking at your configuration and I can't find anything relating to 192.168.0.1. It's not set in rc.conf?!?!?

What does *ifconfig* say inside the running jail? 

Try adding jail_*_ip and jail_*_interface to /etc/rc.conf to make sure the jail's bound to the correct interface and ip address.


----------



## rinalds (Mar 18, 2009)

*Solved!*

Ahh! 192.168.0.1 was indeed added to rc.conf as another alias for fxp0... I left it out from my initial post stupidly thinking it was not relevant. Removed it and everything seems to work finally :h

Many thanks again for helping to figure this out!


----------

