# [RESOLVED] Zabbix and jails redirection



## fred974 (Feb 26, 2014)

/------------------------------ My host ------------------------------\
---{ internet } { 82.36.74.191-ExtFixIP } --- [ 192.168.0.1-local gateway ] --- [ 192.168.0.198-FreeBSD Host ] --- jails --- [ 192.168.0.115 ] webjail
.........................................................................................................................................|---[ 192.168.0.120 ] DBjail
.........................................................................................................................................|--- [ 192.168.0.125 ] mailjail

Hello everyone,
I have 3 jails in my FreeBSd jail...

I just finish to install zabbix-server and zabbix-agent  on my Host.
At the moment I haven't yet installed the agent on the jails and its a nightmare of its own..
Could someone help me to make sure tcp port 10050 is open.
so far i have the following in my pf.conf file:

```
ext_if="em0"
zabbix={ 10050 }
pass in on $ext_if proto tcp from any to ($ext_if) port $tcp_services flags S/SA keep state
```
Do I need to do any redirection? I the above rule enough?


----------



## _martin (Feb 26, 2014)

*Re: Zabbix and jails redirection*

If you have a look on my howto here: http://forums.freebsd.org/viewtopic.php?f=39&t=30063  you'll see how to set it up. When it comes to opening a port only (from outside), you should have the following in the filtering section:


```
pass in quick proto tcp to 82.36.74.191 port 10050
```

Your pf.conf seems to be either not complete or is missing some parts you actually have there.
But you didn't share your intentions. I don't know where the service using 10050 is running actually.


----------



## fred974 (Feb 26, 2014)

*Re: Zabbix and jails redirection*

Sorry ...
/etc/rc/conf

```
### Network configuration
ifconfig_bge0="inet 192.168.0.198 netmask 255.255.255.0"                                 
defaultrouter="192.168.0.1"                                                                                                                     
### internal network configuration Jail interface aliases                         
ifconfig_bge0_alias0="inet 192.168.0.115 netmask 255.255.255.0" # webjai
ifconfig_bge0_alias1="inet 192.168.0.120 netmask 255.255.255.0" # sqljail
ifconfig_bge0_alias2="inet 192.168.0.125 netmask 255.255.255.0" # mailjail
```
/etc/pf.conf

```
### jail IPs
webjail   ="192.168.0.115"
sqljail   ="192.168.0.120"
mailjail ="192.168.0.125"

### jail Ports
webports ="{ http,https }"
sqlports ="{ 3306 }"
mailports ="{ 25 }"
zabbix= "{ 10050 }"

### nat all jail traffis
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if)

### Translation
rdr on $ext_if proto tcp from any to $ext_if port $webports -> $webjail
rdr on $ext_if proto tcp from any to $ext_if port $sqlports -> $sqljail
rdr on $ext_if proto tcp from any to $ext_if port $mailports -> $mailjail
pass in on $ext_if proto tcp from any to ($ext_if) port $zabbix flags S/SA keep state
... More rule bellow
```
The zabbix-server and zabbix-agent are running on the host (192.168.0.198) and zabbix listen on port 10050 .
Now..this is were I lack of knowledge as I'm new to zabbix.
I don't know if the zabbix port is open externally or internally


----------



## _martin (Feb 26, 2014)

*Re: Zabbix and jails redirection*

I don't use zabbix either, but this not the issue right now. 

I can't figure out your topology drawing. Can you elaborate on that ? I don't know what "EstFixIP" is, I don't know what's the difference between "my host" and "FreeBSD host". I'd need more details on that. 
Looking at the pf.conf you pasted NAT setup does not make much sense - you are NATing to the same network you have on your (assuming to be) an egress interface (assuming that pf.conf is from "FreeBSD host"). 

Right now it seems that "FreeBSD host" is on the local network and have jail IPs assigned to its outgoing interface (bge0). In this setup no NAT is needed.

Question is: where from are you trying to reach that zabbix port?


----------



## fred974 (Feb 27, 2014)

*Re: Zabbix and jails redirection*

Hi @matoatlantis,

Looking at the info I have given, I must admit, its a mess...

82.36.74.191 - ExtFixIP ==> is my my public fix IP address
192.168.0.198 - FreeBSD_Host ==> is the my physical machine with FreeBSD 10 ZFS Operative system. All my jails reside inside the FreeBSD_Host
webjail IP ==>  192.168.0.115
DBjail IP ==> 192.168.0.120
mailjail IP ==> 192.168.0.125

My Server has a single Ethernet card - bge0 The jail network is an alias onbge0 - lo1 is not used here as per @SirDice advise

pf.conf is on FreeBSD_Host and redirect traffic to all jails and secure the FreeBSD_Host itself.

now taking your previous comment on-board, I change the pf.conf. bellow is my full file (still learning pf):

```
### macro name for external interface.
ext_if = "bge0"

### jail IPs
webjail   ="192.168.0.115"
sqljail   ="192.168.0.120"
mailjail ="192.168.0.125"

### jail Ports
webports ="{ http,https }"
sqlports ="{ 1913 }"
mailports ="{ 25 }"
bfports = "{ 22 }" # SSH brute force prevention
zabbixport = "{ 10050 }"

table <fail2ban> counters persist file "/var/db/pf/bf.table"
table <ssh_abuse> counters persist file "/var/db/pf/sshabuse.table"

scrub in on $ext_if all fragment reassemble

### nat all jail traffis
nat on $ext_if from $ext_if:network to any -> ($ext_if)

### Translation
rdr on $ext_if proto tcp from any to $ext_if port $webports -> $webjail
rdr on $ext_if proto tcp from any to $ext_if port $sqlports -> $sqljail
rdr on $ext_if proto tcp from any to $ext_if port $mailports -> $mailjail

### set a default deny everything policy.
block all

### exercise antispoofing on the external interface, add the local loopback interface as an exception
set skip on lo0
antispoof for $ext_if inet

### block anything coming from sources that we have no back routes for.
block in from no-route to any

### block packets that fail a reverse path check.
block in from urpf-failed to any

### drop broadcast requests quietly.
block in quick on $ext_if from any to 255.255.255.255

### block packets claiming to come from reserved internal address blocks
block in log quick on $ext_if from { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 } to any

### block probes that can possibly determine our operating system 
block in quick on $ext_if proto tcp flags FUP/WEUAPRSF
block in quick on $ext_if proto tcp flags WEUAPRSF/WEUAPRSF
block in quick on $ext_if proto tcp flags SRAFU/WEUAPRSF
block in quick on $ext_if proto tcp flags /WEUAPRSF
block in quick on $ext_if proto tcp flags SR/SR
block in quick on $ext_if proto tcp flags SF/SF

### keep state on any outbound tcp, udp or icmp traffic. modulate the isn of
### outgoing packets.
pass out on $ext_if proto { tcp, udp, icmp } from any to any modulate state

### open ports for OpenNTPD time service 
# Ipv4 Open outgoing port TCP 123 (NTP)
pass out on $ext_if proto tcp to any port ntp
# Ipv4 Open outgoing port UDP 123 (NTP)
pass out on $ext_if proto udp to any port ntp

### allow inbound ssh traffic with synproxy handshaking.
pass in on $ext_if proto tcp from any to any port ssh flags S/SA synproxy state

### allow inbound www traffic with synproxy handshaking.
pass in on $ext_if proto tcp from any to any port www flags S/SA synproxy state

### open zabbix traffic with synproxy handshaking.
pass in on $ext_if proto tcp from any to ($ext_if) port $zabbixport flags S/SA keep state

###  ruleset that prevents excessive abuse by hosts
block in quick from <ssh_abuse>
block in quick on $ext_if from <fail2ban> to any

### send all ssh excessive abuse connection to blackhole table
pass in on $ext_if proto tcp to any port ssh flags S/SA keep state (max-src-conn 10, max-src-conn-rate 3/5, overload <ssh_abuse> flush)
```


----------



## SirDice (Feb 27, 2014)

*Re: Zabbix and jails redirection*

Remove all the NAT and redirections, you don't need them. Other hosts can connect to the 192.168.0.0/24 addresses on the interface directly.


----------



## fred974 (Feb 27, 2014)

*Re: Zabbix and jails redirection*



			
				SirDice said:
			
		

> Remove all the NAT and redirections, you don't need them. Other hosts can connect to the 192.168.0.0/24 addresses on the interface directly.


As sugested i removed the NAT and redirections and the web is no longer accessible..
tried to add

```
pass in on $ext_if from any to $webjail port $webport
pass in on $ext_if from any to $sqljail port $sqlport
pass in on $ext_if from any to $mailjail port $mailport
```
But with no success


----------



## SirDice (Feb 27, 2014)

*Re: Zabbix and jails redirection*

This rule:

```
block in log quick on $ext_if from { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 } to any
```
Also blocks all your incoming packets from 192.168.0.0/24 (192.168.0.0/16 includes 192.168.0.0/24). And because it uses the quick keyword no other rules will be evaluated.


----------



## fred974 (Feb 27, 2014)

*Re: Zabbix and jails redirection*



			
				SirDice said:
			
		

> This rule:
> 
> ```
> block in log quick on $ext_if from { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 } to any
> ...



So should I remove that line all together?
will that work?

```
table <blocked_nets> { 10.0.0.0/8, \
                       172.16.0.0/12 \
                       192.168.0.0/16, \
					   255.255.255.255/32 }
					   
block in log on $ext_if from <blocked_nets> to any
```


----------



## _martin (Feb 27, 2014)

*Re: Zabbix and jails redirection*

Ok, so let's see if I got it right. Your default gateway is 192.168.0.1 which is some sort of home router (NAT box). It is also a box where your public IP is assigned to. This box is not FreeBSD and it's not part of your problem you are trying to solve. 

You are trying to reach a web page within your network (192.168.0.0/24) hosted on FreeBSD box (192.168.0.198) inside of a jail (192.168.0.115). Zabbix is installed on the FreeBSD host. Correct ?

If so, your rc.conf seems ok. I see you have more services running on this host, but I'll focus on the zabbix (10050), web (80,443) and and SSH (22) only. Other services are to be configured the same way. 
Any other traffic to the server will be denied. For the demonstration purposes I'll allow tcp/udp on 10050 port. 

/etc/pf.conf:

```
ext_if="bge0"

IP_FREEBSD_HOST="192.168.0.198"
IP_WEB="192.168.0.115"

PORT_WEB="{80,443}"
PORT_SSH="{22}"
PORT_ZABBIX="{10050}"

# [options]
set skip on lo0

# [normalizaiton]
scrub in all

# [translation]

# [filtering]
pass out all
block in all

pass in quick proto tcp from any to $IP_FREEBSD_HOST port $PORT_SSH
pass in quick proto tcp from any to $IP_WEB port $PORT_WEB
pass in quick proto {tcp,udp} from any to $IP_FREEBSD_HOST port $PORT_ZABBIX
```


----------



## fred974 (Feb 27, 2014)

*Re: Zabbix and jails redirection*



			
				matoatlantis said:
			
		

> Ok, so let's see if I got it right. Your default gateway is 192.168.0.1 which is some sort of home router (NAT box). It is also a box where your public IP is assigned to. This box is not FreeBSD and it's not part of your problem you are trying to solve.
> 
> You are trying to reach a web page within your network (192.168.0.0/24) hosted on FreeBSD box (192.168.0.198) inside of a jail (192.168.0.115). Zabbix is installed on the FreeBSD host. Correct ?


Bingo! you are 100% correct 
I'll start with the example you provided and build on that.
I'll let you know how I get on.

thank you
Fred


----------



## fred974 (Feb 27, 2014)

*Re: Zabbix and jails redirection*

Hi @matoatlantis,
Sorry to be a pain..but I have done a copy and past of the example you gave me and restarted the server.
Once the rules was loaded, I still had ssh connection but no access to my website.
Are you guys sure I don't need the redirect rule?


----------



## _martin (Feb 27, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> Hi @matoatlantis,
> Sorry to be a pain..but I have done a copy and past of the example you gave me and restarted the server.
> Once the rules was loaded, I still had ssh connection but no access to my website.
> Are you guys sure I don't need the redirect rule?


No pain. I did test those rules and they are working. You don't need to redirect, you have those IP addresses assigned directly on outgoing interface; IPs are locally accessible. 
Where did you test it from ? I'm assuming you are accessing the web page from a computer within your local network.

You don't need to reboot, just do a 

```
pfctl -f /etc/pf.conf
```
to apply new PF config. 

To avoid issues with webserver configuration itself do a test of your setup using e.g. `nc`. In jail, make sure web server is down (that means nothing has opened a socket 192.168.0.115:80). Then launch the following command: 

```
webjail# nc -l -80
```
And try to telnet to it from a computer within your network: 


```
telnet 192.168.0.115 80
```
You can also check the traffic on your BSD host: 


```
tcpdump -n -f port 80
pfctl -ss
```

*EDIT:* I also assume you are using my pf.conf and you didn't just merge it with your current config. As @SirDice said below, your pf.conf is way too complicated.


----------



## SirDice (Feb 28, 2014)

*Re: Zabbix and jails redirection*

I've only seen parts of it but your pf.conf seems overly complicated. I'd just move it to the side and start with a new ruleset. Build it up slowly and each time you add something test it. Don't put all those odd tests in there to prevent scans and such. A scrub takes care of most of the crappy packets. Don't worry about those OS fingerprints. Hardly anybody uses scans like that because it's usually easier to find the info in other ways.


----------



## fred974 (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				matoatlantis said:
			
		

> fred974 said:
> 
> 
> 
> ...


Yes I am starting from scratch using the example you provided.

/etc/pf.conf:

```
ext_if="bge0"

IP_FREEBSD_HOST="192.168.0.198"
IP_WEB="192.168.0.115"

PORT_WEB="{80,443}"
PORT_SSH="{22}"
PORT_ZABBIX="{10050}"

# [options]
set skip on lo0

# [normalizaiton]
scrub in all

# [translation]

# [filtering]
pass out all
block in all

pass in quick proto tcp from any to $IP_FREEBSD_HOST port $PORT_SSH
pass in quick proto tcp from any to $IP_WEB port $PORT_WEB
pass in quick proto {tcp,udp} from any to $IP_FREEBSD_HOST port $PORT_ZABBIX
```
`root@FREEBSD_HOST:/ # jexec 1`
`root@webjail:/ # nc -l 80`

```
nc: Address already in use
```
`root@webjail:/ # service nginx stop`

```
Stopping nginx.
Waiting for PIDS: 1047.
```
`root@webjail:/ # nc -l 80`

```
no response ... so port 80 is not listened on
```
`root@FREEBSD_LAPTOP_ON LAN:/ # ping 192.168.0.115`

```
no response
```
`root@FREEBSD_LAPTOP_ON LAN:/ # telnet 192.168.0.115`

```
no response
```
`root@FREEBSD_LAPTOP_ON LAN:/ # telnet 192.168.0.115 80`

```
no response
```
`root@FREEBSD_HOST:/ # telnet 192.168.0.115 80`

```
Trying 192.168.0.115...
telnet: connect to address 192.168.0.115: Operation timed out
telnet: Unable to connect to remote host
```
`root@FREEBSD_HOST:~ # telnet 192.168.0.115 80`

```
Trying 192.168.0.115...
Connected to 192.168.0.115.
Escape character is '^]'.
Connection closed by foreign host.
```
`root@FREEBSD_HOST:~ # tcpdump -n -f port 80`

```
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
capability mode sandbox enabled
```
`root@FREEBSD_HOST:~ # pfctl -ss`

```
No ALTQ support in kernel
ALTQ related functions disabled
all udp 192.168.0.198:20609 -> 109.169.89.48:123       MULTIPLE:MULTIPLE
all tcp 192.168.0.198:22 <- 213.146.159.254:2709       ESTABLISHED:ESTABLISHED
```
`root@webjail:/ # whois google.com`

```
Whois Server Version 2.0

Domain names in the .com and .net domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

GOOGLE.COM.ZZZZZZZZZZZZZZZZZZZZZZZZZZ.HAVENDATA.COM
GOOGLE.COM.ZZZZZZZZZZZZZ.GET.ONE.MILLION.DOLLARS.AT.WWW.UNIMUNDI.COM
GOOGLE.COM.ZZZZZ.GET.LAID.AT.WWW.SWINGINGCOMMUNITY.COM
GOOGLE.COM.ZOMBIED.AND.HACKED.BY.WWW.WEB-HACK.COM
GOOGLE.COM.ZNAET.PRODOMEN.COM ....
```
`root@FREEBSD_HOST:~ # vi /etc/jails.conf`

```
webjail {
   path = /local/jails/webjail;
   mount.devfs;
   devfs_ruleset = 4;
   host.hostname = webjail.mydomain.com;
   ip4.addr = 192.168.0.115;
   exec.start = "/bin/sh /etc/rc";
   exec.stop = "/bin/sh /etc/rc.shutdown";
}

sqljail {
   path = /local/jails/sqljail;
   mount.devfs;
   devfs_ruleset = 4;
   host.hostname = sqljail.mydomain.com;
   ip4.addr = 192.168.0.120;
   exec.start = "/bin/sh /etc/rc";
   exec.stop = "/bin/sh /etc/rc.shutdown";
}

mailjail {
   path = /local/jails/mailjail;
   mount.devfs;
   devfs_ruleset = 4;
   host.hostname = mailjail.mydomain.com;
   ip4.addr = 192.168.0.125;
   exec.start = "/bin/sh /etc/rc";
   exec.stop = "/bin/sh /etc/rc.shutdown";
}
```
`root@webjail:/ # vi /etc/rc.conf`

```
# Miscellaneous Configuration
hostname="webjail.mydomain.com"
#network_interfaces="lo1"
kern_securelevel_enable="YES"
kern_securelevel="1"
rpcbind_enable="NO"
cron_flags="$cron_flags -J 15"
syslogd_flags="-ss"
sendmail_enable="NONE"
clear_tmp_enable="YES"
## Mail Config
postfix_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
# SSHD Configuration
sshd_enable="YES"
nginx_enable="YES"
php_fpm_enable="YES"
```
/etc/sysctl.conf

```
kern.ipc.maxsockbuf=2097152
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.recvbuf_max=2097152
net.inet.tcp.cc.algorithm=htcp
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet.tcp.hostcache.expire=900
kern.ipc.somaxconn=1024
net.inet.tcp.mssdflt=1460
net.inet.tcp.nolocaltimewait=1
net.inet.tcp.experimental.initcwnd10=1
net.inet.tcp.rfc1323=1
net.inet.tcp.rfc3390=1
net.inet.tcp.sendspace=262144
net.inet.tcp.syncache.rexmtlimit=1
net.inet.tcp.syncookies=0
net.inet.ip.check_interface=1
net.inet.ip.portrange.randomized=1
net.inet.ip.process_options=0
net.inet.ip.random_id=1
net.inet.ip.redirect=0
net.inet.ip.accept_sourceroute=0
net.inet.ip.sourceroute=0
net.inet.icmp.bmcastecho=0
net.inet.icmp.maskfake=0
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.icmp.icmplim_output=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.ecn.enable=0
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.icmp_may_rst=0
net.inet.tcp.msl=5000
net.inet.tcp.path_mtu_discovery=0
net.inet.tcp.rfc3042=0
net.inet.tcp.sack.enable=1
net.inet.udp.blackhole=1
net.inet.tcp.blackhole=2
security.bsd.see_other_uids=0
net.local.stream.sendspace=164240  # (default 8192)
net.local.stream.recvspace=164240  # (default 8192)
###### jails #########
security.jail.allow_raw_sockets=1       # (default 0)
security.jail.enforce_statfs=2          # (default 2)
security.jail.set_hostname_allowed=0    # (default 1)
security.jail.socket_unixiproute_only=1 # (default 1)
security.jail.sysvipc_allowed=0         # (default 0)
security.jail.chflags_allowed=0         # (default 0)
#security.jail.mount_zfs_allowed: 1
#security.jail.mount_allowed: 1
```
Guys,

You cannot imagine how much I do appreciate your help here.. Thank you very much for assisting me.
I have no clue as to why This is hapening


----------



## _martin (Feb 28, 2014)

*Re: Zabbix and jails redirection*

Ping was disabled in the config I made. You can allow it by the following pf.conf modification: 


```
ICMP_TYPES="{echoreq,unreach}"

# in filtering
pass inet proto icmp all icmp-type $ICMP_TYPES keep state
```
Can you also share the output of `ifconfig` and `netstat -nr` command ? Just to be sure, also `pfctl -snat`, `pfctl -sr` and `arp -an` after the rules are applied.

Strange thing is that you have not even received a packet to 192.168.0.115 when you tried to reach it from FREEBSD_LAPTOP_ON LAN. This suggests that the problem occurs before PF filtering.


----------



## SirDice (Feb 28, 2014)

*Re: Zabbix and jails redirection*

The standard tool to use is tcpdump(1). With it you can see the packets arriving on the interface. Even if the host itself is completely firewalled the packets should still arrive. If nothing gets to the server the issue is somewhere else on the network.


----------



## fred974 (Feb 28, 2014)

*Re: Zabbix and jails redirection*

Ok, I restarted the laptop and I now get the following result (Nginx was running at the time)
`root@FREEBSD_LAPTOP_ON LAN:/root # telnet 192.168.0.115 80`

```
Trying 192.168.0.115...
Connected to 192.168.0.115.
Escape character is '^]'.
Connection closed by foreign host.
```
`root@FREEBSD_HOST:~ # ifconfig`

```
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:6a:6a:8c
        inet 192.168.0.198 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::d685:64ff:fe6a:6a8c%bge0 prefixlen 64 scopeid 0x1
        inet 192.168.0.115 netmask 0xffffff00 broadcast 192.168.0.255
        inet 192.168.0.120 netmask 0xffffff00 broadcast 192.168.0.255
        inet 192.168.0.125 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
```
`root@FREEBSD_HOST:~ # netstat -nr`

```
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS         0      775   bge0
127.0.0.1          link#2             UH          0      578    lo0
192.168.0.0/24     link#1             U           0       21   bge0
192.168.0.115      link#1             UHS         0        0    lo0
192.168.0.120      link#1             UHS         0    10454    lo0
192.168.0.125      link#1             UHS         0        0    lo0
192.168.0.198      link#1             UHS         0     8175    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%bge0/64                    link#1                        U          bge0
fe80::d685:64ff:fe6a:6a8c%bge0    link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff01::%bge0/32                    fe80::d685:64ff:fe6a:6a8c%bge0 U          bge0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%bge0/32                    fe80::d685:64ff:fe6a:6a8c%bge0 U          bge0
ff02::%lo0/32                     ::1                           U           lo0
```
`root@FREEBSD_HOST:~ # pfctl -snat`

```
No ALTQ support in kernel
ALTQ related functions disabled
```
`root@FREEBSD_HOST:~ # pfctl -sr[cmd]
[code]
No ALTQ support in kernel
ALTQ related functions disabled
scrub in all fragment reassemble
pass out all flags S/SA keep state
block drop in all
pass in quick inet proto tcp from any to 192.168.0.115 port = http flags S/SA keep state
pass in quick inet proto tcp from any to 192.168.0.115 port = https flags S/SA keep state
pass in quick inet proto tcp from any to 192.168.0.198 port = ssh flags S/SA keep state
pass in quick inet proto tcp from any to 192.168.0.198 port = 10050 flags S/SA keep state
pass in quick inet proto udp from any to 192.168.0.198 port = 10050 keep state
[/code]
[cmd]root@FREEBSD_HOST:~ # arp -an`

```
? (192.168.0.198) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
? (192.168.0.1) at 9c:d3:6d:d3:17:1b on bge0 expires in 928 seconds [ethernet]
? (192.168.0.120) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
? (192.168.0.125) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
? (192.168.0.115) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
```
Any idea why using my original pf.conf file, the web worked?


----------



## _martin (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> Ok, I restarted the laptop and I now get the following result (Nginx was running at the time)
> `root@FREEBSD_LAPTOP_ON LAN:/root # telnet 192.168.0.115 80`
> 
> ```
> ...



But hey, now it's working! So problem was with your notebook maybe ?

Configuration (in config files) and output from the commands is what one expects -- that is all ok.


----------



## fred974 (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				matoatlantis said:
			
		

> Configuration (in config files) and output from the commands is what one expects -- that is all ok.


Ho well, I'll have to carry on looking . Thank you very much for all the time you spent on helping me 
One last thing,when doing `lynx [url=http://192.168.0.115]http://192.168.0.115[/url]` from FREEBSD_LAPTOP_ON LAN I can see the webpages using the pf rules you provided.
Is that an indication of anything?


----------



## _martin (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> matoatlantis said:
> 
> 
> 
> ...


Well, that's the indication that it is working. That's what I said actually above. First, when you pasted the output of `tcpdump`, I said that this suggests that the problem is somewhere before PF - meaning it has to be somewhere else (as later @SirDice said). It doesn't matter if you use telnet or lynx, result is the same. 

As it is working after you rebooted the notebook I'd say you had some problems with the notebook itself. 
So the question now - is it now working as you expected ?


----------



## fred974 (Feb 28, 2014)

*Re: Zabbix and jails redirection*

From the notebook, I can see the pages but only to `lynx [url=http://IP]http://IP[/url]`, not using `lynx [url=http://domain.com]http://domain.com[/url]`
I assume the problem is not with DNS as when using the old rules, I can access the domain webpage's


----------



## SirDice (Feb 28, 2014)

*Re: Zabbix and jails redirection*

Resolving happens on the client, not on the server you're connecting to. TCP/IP doesn't understand names, only IP addresses. So the name must be resolved to an IP address before the client can connect to it.


----------



## _martin (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> From the notebook, I can see the pages but only to `lynx [url=http://IP]http://IP[/url]`, not using `lynx [url=http://domain.com]http://domain.com[/url]`
> I assume the problem is not with DNS as when using the old rules, I can access the domain webpage's



Ok, but that is not the PF problem any more . Goal was to set the firewall and allow the traffic for the respected jails. If you can reach the service in jail, all is ok.

Make sure jails have the proper method of resolving set too (hosts, resolv.conf, nsswitch.conf .. ). Also check the configuration of the web server itself.
Do tests that jail can resolve its hostname properly.


----------



## fred974 (Feb 28, 2014)

*Re: Zabbix and jails redirection*



> Ok, but that is not the PF problem any more . Goal was to set the firewall and allow the traffic for the respected jails. If you can reach the service in jail, all is ok.
> 
> Make sure jails have the proper method of resolving set too (hosts, resolv.conf, nsswitch.conf .. ). Also check the configuration of the web server itself.
> Do tests that jail can resolve its hostname properly.


No it cannot resolved by hostname. Actually none of my boxes do...
I might need to set a dnsserver to achieve that.
Am I right there?


----------



## _martin (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> I might need to set a dnsserver to achieve that.
> Am I right there?


Yes and no. Depends what are you trying to achieve. Local names can be resolved locally. One option would be to make one "global" /etc/hosts file and redistribute it among your servers (and jails) in your subnet. Make sure /etc/nsswitch.conf has policy set that names will be first resolved using files.

/etc/hosts example:


```
192.168.0.1  gw gw.example.com
192.168.0.115 www www.example.com example.com
```
Also make sure the proper search pattern is set in /etc/resolv.conf:


```
search example.com
```

Another one is to set the DNS on your router, update the entries there and set the nameserver (on clients) to this ip (192.168.0.1) in your /etc/resolv.conf. DNS on your router would have forwarding set to the DNS routers provided by your ISP. Actually the best way would be to combine #1 and #2 option.

Third one would be to set the DNS server on your FreeBSD, create zones and forward to the ISP DNS servers. This option is, IMHO, not really a good idea in your scenario.


----------



## SirDice (Feb 28, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> > Ok, but that is not the PF problem any more . Goal was to set the firewall and allow the traffic for the respected jails. If you can reach the service in jail, all is ok.
> >
> > Make sure jails have the proper method of resolving set too (hosts, resolv.conf, nsswitch.conf .. ). Also check the configuration of the web server itself.
> > Do tests that jail can resolve its hostname properly.
> ...


Yes, I'm guessing the host on 192.168.0.1 takes care of the internet connection? Does it perhaps also run a (caching) DNS server? Most SOHO internet routers do.


----------



## fred974 (Mar 3, 2014)

*Re: Zabbix and jails redirection*

Hi @SirDice and @matoatlantis,

I have been going over all my config files and cannot find any issue with them..
I have added the following line to the pf.conf you provided and the web is now working

```
rdr on $ext_if proto tcp from any to $ext_if port $PORT_WEB -> $IP_WEB
```
pf.conf

```
ext_if="bge0"

IP_FREEBSD_HOST="192.168.0.198"
IP_WEB="192.168.0.115"
ICMP_TYPES="{echoreq,unreach}"

ICMP_TYPES="{echoreq,unreach}"
PORT_WEB="{80,443}"
PORT_SSH="{22}"
PORT_ZABBIX="{10050}"

# [options]
set skip on lo0

# [normalizaiton]
scrub in all

rdr on $ext_if proto tcp from any to $ext_if port $PORT_WEB -> $IP_WEB

# [translation]

# [filtering]
pass out all
block in all

pass in quick proto tcp from any to $IP_FREEBSD_HOST port $PORT_SSH
pass in quick proto tcp from any to $IP_WEB port $PORT_WEB
pass in quick proto {tcp,udp} from any to $IP_FREEBSD_HOST port $PORT_ZABBIX
pass inet proto icmp all icmp-type $ICMP_TYPES keep state
```
According to you guys, redirect is not nessesary as my jails and host are sharing the sane bge0 interface directly.

So do you have any idea what files is messing this pf setting?


----------



## _martin (Mar 3, 2014)

*Re: Zabbix and jails redirection*

I really don't know. It has to work without rdr, as you have those IPs activated on interface facing LAN. 
Maybe a test would be in order. 

Just so we understand each other I'm assuming your topology is the following: 


```
+----------+
              |  router  |
              +----^-----+
                   | 192.168.0.1
                   |                                FreeBSD host
              +----+-----+                       +---------------+
              |  switch  +----------------->bge0+| 192.168.0.198 |
              +----------+                       | 192.168.0.115 | (wwwjail)
                +                                +---------------+
                |  192.168.0.x
            +-----------+
            | notebook  |
            +-----------+
```
Apply the PF config I pasted (i.e. remove the rdr rule from the last PF config pasted), load it and make sure there are no rdr rules active (`pfctl -snat` has to show nothing). 

Start the jails, start the webserver.  Make sure you can reach it from these locations: 


```
(wwwjail)# telnet 192.168.0.115 80
(freebsdhost)# telnet 192.168.0.115 80
```

These two tests have to be successful.  make `arp -an` from freebsdhost and notebook.

Start `tcpdump` on freebsdhost using following command (catch all except port 22 traffic): 


```
tcpdump -n -f port not 22
```
Now ping the IP of the wwwjail from notebook. Do the `arp -an` on the notebook again (looking for the MAC address of the wwwjail in notebook's arp cache).
Now try to reach wwwjail from notebook:


```
telnet 192.168.0.115 80
```
And let's see your output from all the commands I mentioned. 

Right now it seems like notebook had a problem reaching your wwwjail IP even when it was assigned on local network (`tcpdump` you previously posted didn't catch the traffic to www IP). You are redirecting from any to $ext_if, so you are even redirecting the packet to itself (from www IP to www IP). Which is not correct.


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*

HI @matoatlantis
The topology you drawn is correct. Just to clarify tough, the router also act as the switch... 

```
+----------+
              |  router  |
              +----^-----+
                   | 192.168.0.1
                   |                                FreeBSD host
              +----+-----+                       +---------------+
              |  switch  +----------------->bge0+| 192.168.0.198 |
              +----------+                       | 192.168.0.115 | (wwwjail)
                +                                +---------------+
                |  192.168.0.150
            +-----------+
            | notebook  |
            +-----------+
```
Let's the testing begin...
pf.conf

```
ext_if="bge0"

IP_FREEBSD_HOST="192.168.0.198"
IP_WEB="192.168.0.115"
ICMP_TYPES="{echoreq,unreach}"

ICMP_TYPES="{echoreq,unreach}"
PORT_WEB="{80,443}"
PORT_SSH="{22}"
PORT_ZABBIX="{10050}"

# [options]
set skip on lo0

# [normalizaiton]
scrub in all

# [translation]

# [filtering]
pass out all
block in all

pass in quick proto tcp from any to $IP_FREEBSD_HOST port $PORT_SSH
pass in quick proto tcp from any to $IP_WEB port $PORT_WEB
pass in quick proto {tcp,udp} from any to $IP_FREEBSD_HOST port $PORT_ZABBIX
pass inet proto icmp all icmp-type $ICMP_TYPES keep state
```
`root@FREEBSD_HOST:~ # pfctl -snat`

```
No ALTQ support in kernel
ALTQ related functions disabled
```
`root@FREEBSD_HOST:~ # telnet 192.168.0.115 80`

```
Trying 192.168.0.115...
Connected to 192.168.0.115.
Escape character is '^]'.
Connection closed by foreign host.
```
`root@webjail:/ # telnet 192.168.0.115 80`

```
Trying 192.168.0.115...
Connected to webjail.
Escape character is '^]'.
Connection closed by foreign host.
```
`root@FREEBSD_HOST:~ #  arp -an`

```
? (192.168.0.198) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
? (192.168.0.1) at 9c:d3:6d:d3:17:1b on bge0 expires in 946 seconds [ethernet]
? (192.168.0.120) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
? (192.168.0.125) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
? (192.168.0.115) at d4:85:64:6a:6a:8c on bge0 permanent [ethernet]
```
`root@FREEBSD_LAPTOP_ON LAN:/root #  arp -an`

```
? (192.168.0.1) at 9c:d3:6d:d3:17:1b on alc0 expires in 844 seconds [ethernet]
? (192.168.0.3) at 60:67:20:9a:33:fc on alc0 expires in 1173 seconds [ethernet]
? (192.168.0.150) at 00:23:5a:76:de:78 on alc0 permanent [ethernet]
```
`root@FREEBSD_LAPTOP_ON LAN:/root # ping 192.168.0.115`

```
PING 192.168.0.115 (192.168.0.115): 56 data bytes
64 bytes from 192.168.0.115: icmp_seq=0 ttl=64 time=0.306 ms
64 bytes from 192.168.0.115: icmp_seq=1 ttl=64 time=0.387 ms
64 bytes from 192.168.0.115: icmp_seq=2 ttl=64 time=0.268 ms
64 bytes from 192.168.0.115: icmp_seq=3 ttl=64 time=0.297 ms
64 bytes from 192.168.0.115: icmp_seq=4 ttl=64 time=0.326 ms
64 bytes from 192.168.0.115: icmp_seq=5 ttl=64 time=0.371 ms
^C
--- 192.168.0.115 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.268/0.326/0.387/0.042 ms
```
`root@FREEBSD_LAPTOP_ON LAN:/root #  arp -an`

```
? (192.168.0.1) at 9c:d3:6d:d3:17:1b on alc0 expires in 564 seconds [ethernet]
? (192.168.0.3) at 60:67:20:9a:33:fc on alc0 expires in 1190 seconds [ethernet]
? (192.168.0.115) at d4:85:64:6a:6a:8c on alc0 expires in 1101 seconds [ethernet]
? (192.168.0.150) at 00:23:5a:76:de:78 on alc0 permanent [ethernet]
```
`root@FREEBSD_HOST:~ # tcpdump -n -f port not 22`

```
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
capability mode sandbox enabled
13:39:20.260813 IP 192.168.0.3.17500 > 192.168.0.255.17500: UDP, length 114
13:39:21.187564 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 359
13:39:21.188749 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 304
13:39:21.189650 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 295
13:39:21.190396 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 369
13:39:23.765429 IP 192.168.0.198.44091 > 194.168.8.100.53: 57586+% [1au] A? 2.europe.pool.ntp.org. (50)
13:39:23.797009 IP 194.168.8.100.53 > 192.168.0.198.44091: 57586 4/0/1 A 193.225.121.161, A 62.113.243.206, A 217.147.208.1, A 176.31.45.66 (114)
13:39:23.797999 IP 192.168.0.198.57757 > 194.168.4.100.53: 41804+% [1au] DNSKEY? . (28)
13:39:23.805294 IP 194.168.4.100.53 > 192.168.0.198.57757: 41804 2/0/1 DNSKEY, DNSKEY (450)
13:39:23.806199 IP 192.168.0.198.30759 > 194.168.8.100.53: 25337+% [1au] DNSKEY? . (28)
13:39:23.821294 IP 194.168.8.100.53 > 192.168.0.198.30759: 25337 2/0/1 DNSKEY, DNSKEY (450)
13:39:23.822198 IP 192.168.0.198.56285 > 194.168.4.100.53: 43215+% [1au] DNSKEY? . (28)
13:39:23.829494 IP 194.168.4.100.53 > 192.168.0.198.56285: 43215 2/0/1 DNSKEY, DNSKEY (450)
13:39:23.830406 IP 192.168.0.198.14919 > 194.168.4.100.53: 26056+% [1au] DNSKEY? . (28)
13:39:23.837416 IP 194.168.4.100.53 > 192.168.0.198.14919: 26056 2/0/1 DNSKEY, DNSKEY (450)
13:39:23.838303 IP 192.168.0.198.58802 > 194.168.8.100.53: 58653+% [1au] DNSKEY? . (28)
13:39:23.852498 IP 194.168.8.100.53 > 192.168.0.198.58802: 58653 2/0/1 DNSKEY, DNSKEY (450)
13:39:23.853986 IP 192.168.0.198.22605 > 194.168.4.100.53: 11799+% [1au] A? 2.europe.pool.ntp.org.home.lan. (68)
13:39:23.861581 IP 194.168.4.100.53 > 192.168.0.198.22605: 11799 NXDomain 0/1/1 (143)
13:39:23.862383 IP 192.168.0.198.55826 > 194.168.8.100.53: 22188+% [1au] AAAA? 2.europe.pool.ntp.org.home.lan. (68)
13:39:23.876445 IP 194.168.8.100.53 > 192.168.0.198.55826: 22188 NXDomain 0/1/1 (143)
13:39:23.877493 IP 192.168.0.198.36223 > 194.168.4.100.53: 47657+% [1au] AAAA? 3.europe.pool.ntp.org. (50)
13:39:23.885253 IP 194.168.4.100.53 > 192.168.0.198.36223: 47657 0/1/1 (114)
13:39:23.886076 IP 192.168.0.198.48651 > 194.168.8.100.53: 56547+% [1au] A? 3.europe.pool.ntp.org.home.lan. (68)
13:39:23.900571 IP 194.168.8.100.53 > 192.168.0.198.48651: 56547 NXDomain 0/1/1 (143)
13:39:23.901356 IP 192.168.0.198.50832 > 194.168.8.100.53: 45112+% [1au] AAAA? 3.europe.pool.ntp.org.home.lan. (68)
13:39:23.916464 IP 194.168.8.100.53 > 192.168.0.198.50832: 45112 NXDomain 0/1/1 (143)
13:39:29.893348 ARP, Request who-has 192.168.0.115 tell 192.168.0.150, length 46
13:39:29.893370 ARP, Reply 192.168.0.115 is-at d4:85:64:6a:6a:8c, length 28
13:39:29.893563 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 0, length 64
13:39:29.893604 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 0, length 64
13:39:30.894839 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 1, length 64
13:39:30.894871 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 1, length 64
13:39:31.895878 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 2, length 64
13:39:31.895911 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 2, length 64
13:39:32.896854 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 3, length 64
13:39:32.896886 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 3, length 64
13:39:33.897829 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 4, length 64
13:39:33.897861 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 4, length 64
13:39:34.898899 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 5, length 64
13:39:34.898931 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 5, length 64
13:39:35.899850 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 6, length 64
13:39:35.899884 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 6, length 64
13:39:36.900821 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 7, length 64
13:39:36.900856 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 7, length 64
13:39:37.901801 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 8, length 64
13:39:37.901833 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 8, length 64
13:39:38.902819 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 9, length 64
13:39:38.902849 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 9, length 64
13:39:38.921028 IP 192.168.0.198.21598 > 194.168.8.100.53: 19507+% [1au] A? 0.uk.pool.ntp.org.home.lan. (64)
13:39:38.936573 IP 194.168.8.100.53 > 192.168.0.198.21598: 19507 NXDomain 0/1/1 (139)
13:39:38.937366 IP 192.168.0.198.24730 > 194.168.4.100.53: 52772+% [1au] AAAA? 0.uk.pool.ntp.org.home.lan. (64)
13:39:38.945380 IP 194.168.4.100.53 > 192.168.0.198.24730: 52772 NXDomain 0/1/1 (139)
13:39:38.946539 IP 192.168.0.198.51011 > 194.168.8.100.53: 51145+% [1au] AAAA? 1.uk.pool.ntp.org. (46)
13:39:38.960735 IP 194.168.8.100.53 > 192.168.0.198.51011: 51145 0/1/1 (110)
13:39:38.961540 IP 192.168.0.198.11344 > 194.168.8.100.53: 19284+% [1au] A? 1.uk.pool.ntp.org.home.lan. (64)
13:39:38.976657 IP 194.168.8.100.53 > 192.168.0.198.11344: 19284 NXDomain 0/1/1 (139)
13:39:38.977412 IP 192.168.0.198.55517 > 194.168.8.100.53: 53758+% [1au] AAAA? 1.uk.pool.ntp.org.home.lan. (64)
13:39:38.992808 IP 194.168.8.100.53 > 192.168.0.198.55517: 53758 NXDomain 0/1/1 (139)
13:39:38.994650 IP 192.168.0.198.23194 > 194.168.4.100.53: 57877+% [1au] A? 3.uk.pool.ntp.org. (46)
13:39:39.001464 IP 194.168.4.100.53 > 192.168.0.198.23194: 57877 4/0/1 A 77.98.80.102, A 213.130.44.252, A 91.212.90.20, A 87.117.247.137 (110)
13:39:39.004268 IP 192.168.0.198.37700 > 194.168.8.100.53: 6350+% [1au] A? 1.europe.pool.ntp.org. (50)
13:39:39.057972 IP 192.168.0.198.13869 > 194.168.8.100.53: 11772+% [1au] A? 1.europe.pool.ntp.org. (50)
13:39:39.059568 IP 194.168.8.100.53 > 192.168.0.198.37700: 6350 4/0/1 A 88.198.40.104, A 188.226.179.193, A 5.9.56.53, A 193.55.167.1 (114)
13:39:39.072618 IP 194.168.8.100.53 > 192.168.0.198.13869: 11772 4/0/1 A 188.226.179.193, A 5.9.56.53, A 193.55.167.1, A 88.198.40.104 (114)
13:39:39.903872 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 10, length 64
13:39:39.903904 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 10, length 64
13:39:40.904853 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 11, length 64
13:39:40.904887 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 11, length 64
13:39:41.905879 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 12, length 64
13:39:41.905912 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 12, length 64
13:39:42.906861 IP 192.168.0.150 > 192.168.0.115: ICMP echo request, id 58205, seq 13, length 64
13:39:42.906894 IP 192.168.0.115 > 192.168.0.150: ICMP echo reply, id 58205, seq 13, length 64
13:39:50.188057 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 359
13:39:50.189106 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 304
13:39:50.190007 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 295
13:39:50.190725 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 369
13:39:50.496848 IP 192.168.0.3.17500 > 192.168.0.255.17500: UDP, length 114
13:39:53.581609 IP 192.168.0.3 > 224.0.0.2: igmp leave 224.0.0.253
13:39:53.623607 ARP, Request who-has 192.168.0.1 tell 192.168.0.3, length 46
13:39:53.657237 IP 192.168.0.3.50684 > 239.255.255.250.3702: UDP, length 996
13:39:53.658886 IP6 fe80::c873:fe93:a24b:6483.50685 > ff02::c.3702: UDP, length 996
13:39:53.665960 ARP, Request who-has 192.168.0.1 tell 192.168.0.3, length 46
13:39:53.688707 ARP, Request who-has 192.168.0.1 tell 192.168.0.3, length 46
13:39:53.761204 IP6 fe80::c873:fe93:a24b:6483.50685 > ff02::c.3702: UDP, length 996
13:39:53.901547 IP 192.168.0.3.50684 > 239.255.255.250.3702: UDP, length 996
13:39:54.079789 IP 192.168.0.198.47714 > 194.168.4.100.53: 51982+% [1au] AAAA? 0.uk.pool.ntp.org. (46)
13:39:54.087999 IP 194.168.4.100.53 > 192.168.0.198.47714: 51982 0/1/1 (110)
13:39:54.089256 IP 192.168.0.198.58307 > 194.168.4.100.53: 22304+% [1au] A? 1.uk.pool.ntp.org. (46)
13:39:54.097901 IP 194.168.4.100.53 > 192.168.0.198.58307: 22304 4/0/1 A 87.117.251.3, A 176.58.109.199, A 82.219.4.31, A 217.114.59.66 (110)
13:39:54.099494 IP 192.168.0.198.49607 > 194.168.4.100.53: 53788+% [1au] A? 2.uk.pool.ntp.org. (46)
13:39:54.130958 IP 194.168.4.100.53 > 192.168.0.198.49607: 53788 4/0/1 A 5.2.16.114, A 109.74.206.120, A 87.124.126.49, A 194.1.151.226 (110)
13:39:54.131609 IP 192.168.0.198.36546 > 194.168.4.100.53: 33912+% [1au] AAAA? 2.uk.pool.ntp.org. (46)
13:39:54.137103 IP 192.168.0.3.57109 > 239.255.255.250.3702: UDP, length 656
13:39:54.138570 IP6 fe80::c873:fe93:a24b:6483.57110 > ff02::c.3702: UDP, length 656
13:39:54.139623 IP 194.168.4.100.53 > 192.168.0.198.36546: 33912 4/0/1 AAAA 2a01:8900::aaaa, AAAA 2a01:8900:0:1::aaaa, AAAA 2a01:7e00::f03c:91ff:fe93:e774, AAAA 2a00:fd80:0:7::5 (158)
13:39:54.140448 IP 192.168.0.198.17872 > 194.168.4.100.53: 20906+% [1au] A? 2.uk.pool.ntp.org.home.lan. (64)
13:39:54.147713 IP 194.168.4.100.53 > 192.168.0.198.17872: 20906 NXDomain 0/1/1 (139)
13:39:54.148489 IP 192.168.0.198.24188 > 194.168.8.100.53: 33546+% [1au] AAAA? 2.uk.pool.ntp.org.home.lan. (64)
13:39:54.167201 IP 194.168.8.100.53 > 192.168.0.198.24188: 33546 NXDomain 0/1/1 (139)
13:39:54.168349 IP 192.168.0.198.38136 > 194.168.4.100.53: 34736+% [1au] AAAA? 3.uk.pool.ntp.org. (46)
13:39:54.175613 IP 194.168.4.100.53 > 192.168.0.198.38136: 34736 0/1/1 (110)
13:39:54.176408 IP 192.168.0.198.9761 > 194.168.4.100.53: 38787+% [1au] A? 3.uk.pool.ntp.org.home.lan. (64)
13:39:54.183522 IP 194.168.4.100.53 > 192.168.0.198.9761: 38787 NXDomain 0/1/1 (139)
13:39:54.184303 IP 192.168.0.198.29981 > 194.168.8.100.53: 45633+% [1au] AAAA? 3.uk.pool.ntp.org.home.lan. (64)
13:39:54.198950 IP 194.168.8.100.53 > 192.168.0.198.29981: 45633 NXDomain 0/1/1 (139)
13:39:54.199935 IP 192.168.0.198.39810 > 194.168.4.100.53: 37172+% [1au] A? 0.europe.pool.ntp.org. (50)
13:39:54.253975 IP 192.168.0.198.52091 > 194.168.4.100.53: 45393+% [1au] A? 0.europe.pool.ntp.org. (50)
13:39:54.264235 IP 194.168.4.100.53 > 192.168.0.198.39810: 37172 4/0/1 A 195.56.189.66, A 81.0.124.200, A 193.225.14.181, A 78.47.226.8 (114)
13:39:54.264375 IP 194.168.4.100.53 > 192.168.0.198.52091: 45393 4/0/1 A 81.0.124.200, A 193.225.14.181, A 78.47.226.8, A 195.56.189.66 (114)
13:39:54.304452 IP 192.168.0.198.9687 > 194.168.8.100.53: 34766+% [1au] AAAA? 1.europe.pool.ntp.org. (50)
13:39:54.307211 IP 192.168.0.3.57109 > 239.255.255.250.3702: UDP, length 656
13:39:54.319228 IP 194.168.8.100.53 > 192.168.0.198.9687: 34766 0/1/1 (114)
13:39:54.320030 IP 192.168.0.198.50838 > 194.168.8.100.53: 37197+% [1au] A? 1.europe.pool.ntp.org.home.lan. (68)
13:39:54.330923 IP6 fe80::c873:fe93:a24b:6483.57110 > ff02::c.3702: UDP, length 656
13:39:54.335543 IP 194.168.8.100.53 > 192.168.0.198.50838: 37197 NXDomain 0/1/1 (143)
13:39:54.336335 IP 192.168.0.198.48961 > 194.168.8.100.53: 47384+% [1au] AAAA? 1.europe.pool.ntp.org.home.lan. (68)
13:39:54.350844 IP 194.168.8.100.53 > 192.168.0.198.48961: 47384 NXDomain 0/1/1 (143)
13:39:54.352706 IP 192.168.0.198.52650 > 194.168.8.100.53: 48083+% [1au] A? 3.europe.pool.ntp.org. (50)
13:39:54.394619 IP 194.168.8.100.53 > 192.168.0.198.52650: 48083 4/0/1 A 78.46.107.140, A 193.227.197.2, A 46.165.212.204, A 85.21.78.91 (114)
13:40:09.451416 IP 192.168.0.198.43993 > 194.168.8.100.53: 57418+% [1au] A? 0.uk.pool.ntp.org. (46)
13:40:09.465589 IP 194.168.8.100.53 > 192.168.0.198.43993: 57418 4/0/1 A 80.84.51.90, A 178.79.155.116, A 109.169.89.48, A 178.79.160.57 (110)
13:40:09.471228 IP 192.168.0.198.9938 > 194.168.4.100.53: 6198+% [1au] AAAA? 2.europe.pool.ntp.org. (50)
13:40:09.499639 IP 194.168.4.100.53 > 192.168.0.198.9938: 6198 4/0/1 AAAA 2001:738:0:851:225:90ff:fe06:7594, AAAA 2a00:fd80:0:7::5, AAAA 2001:1b28:1:5::132, AAAA 2001:470:1f14:1e0c::2 (162)
13:40:19.628267 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 359
13:40:19.629456 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 304
13:40:19.630207 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 295
13:40:19.631117 IP 192.168.0.1.1900 > 239.255.255.250.1900: UDP, length 369
13:40:20.709213 IP 192.168.0.3.17500 > 192.168.0.255.17500: UDP, length 114
^C
271 packets captured
410 packets received by filter
0 packets dropped by kernel
```
`oot@FREEBSD_LAPTOP_ON LAN:/root # lynx [url=http://192.168.0.115]http://192.168.0.115[/url]`

```
Our website is  Coming Soon

   But you can get in touch by following us
   [fb.png] [twitter.png] [rss.png] [linkedin.png]

   And be the first to know when we're ready
   ____________________ Submit
```
`Root@FREEBSD_LAPTOP_ON LAN:/root # lynx [url=http://mydomain.com]http://mydomain.com[/url]`
I get the following on the screen for about 5 minutes

```
Making HTTP connection to mydomain.com
```
 and then

```
Looking up mydomain.com
Making HTTP connection to mydomain.com
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://mydomain.com/
```
`root@FREEBSD_LAPTOP_ON LAN:/root # telnet 192.168.0.115 80`

```
Trying 192.168.0.115...
Connected to 192.168.0.115.
Escape character is '^]'.
Connection closed by foreign host.
```

0 error or warning or anything useful in the nginx log files.

This make absolutely no sense.
Could it be something I added in my /etc/sysctl.conf?

```
kern.ipc.maxsockbuf=2097152
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.recvbuf_max=2097152
net.inet.tcp.cc.algorithm=htcp
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet.tcp.hostcache.expire=900
kern.ipc.somaxconn=1024
net.inet.tcp.mssdflt=1460
net.inet.tcp.nolocaltimewait=1
net.inet.tcp.experimental.initcwnd10=1
net.inet.tcp.rfc1323=1
net.inet.tcp.rfc3390=1
net.inet.tcp.sendspace=262144
net.inet.tcp.syncache.rexmtlimit=1
net.inet.tcp.syncookies=0
net.inet.ip.check_interface=1
net.inet.ip.portrange.randomized=1
net.inet.ip.process_options=0
net.inet.ip.random_id=1
net.inet.ip.redirect=0
net.inet.ip.accept_sourceroute=0
net.inet.ip.sourceroute=0
net.inet.icmp.bmcastecho=0
net.inet.icmp.maskfake=0
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.icmp.icmplim_output=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.ecn.enable=0
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.icmp_may_rst=0
net.inet.tcp.msl=5000
net.inet.tcp.path_mtu_discovery=0
net.inet.tcp.rfc3042=0
net.inet.tcp.sack.enable=1
net.inet.udp.blackhole=1
net.inet.tcp.blackhole=2
security.bsd.see_other_uids=0
net.local.stream.sendspace=164240  # (default 8192)
net.local.stream.recvspace=164240  # (default 8192)
###### jails #########
security.jail.allow_raw_sockets=1       # (default 0)
security.jail.enforce_statfs=2          # (default 2)
security.jail.set_hostname_allowed=0    # (default 1)
security.jail.socket_unixiproute_only=1 # (default 1)
security.jail.sysvipc_allowed=0         # (default 0)
security.jail.chflags_allowed=0         # (default 0)
#security.jail.mount_zfs_allowed: 1
#security.jail.mount_allowed: 1
```
Thank you for helping


----------



## SirDice (Mar 4, 2014)

*Re: Zabbix and jails redirection*

Does mydomain.com actually resolve? And if so, does it resolve to an internal or external address?


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				SirDice said:
			
		

> Does mydomain.com actually resolve? And if so, does it resolve to an internal or external address?


 `root@FREEBSD_LAPTOP_ON LAN:/root # ping mydomain.com`

```
Pinging mydomain.com [82.36.74.191] with 32 bytes of data:
Reply from 82.36.74.191: bytes=32 time=12ms TTL=64
Reply from 82.36.74.191: bytes=32 time=2ms TTL=64
Reply from 82.36.74.191: bytes=32 time=3ms TTL=64
Reply from 82.36.74.191: bytes=32 time=3ms TTL=64

Ping statistics for  82.36.74.191:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 12ms, Average = 5ms
```
`root@FREEBSD_HOST:~ # ping mydomain.com`

```
ping: cannot resolve mydomain.com: Host name lookup failure
```

My windows machine can also resolve mydomain.com


----------



## kpa (Mar 4, 2014)

*Re: Zabbix and jails redirection*

The mydomain.com has to be resolvable trough DNS. Looks like you only have the name in the /etc/hosts file of the laptop? Or is the server using a resolver that does not yet know about the name?


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				kpa said:
			
		

> The mydomain.com has to be resolvable trough DNS. Looks like you only have the name in the /etc/hosts file of the laptop? Or is the server using a resolver that does not yet know about the name?


No Not at all
`FREEBSD_LAPTOP_ON LAN:/root # vi etc/hosts`

```
::1                     localhost localhost.my.domain
127.0.0.1               localhost localhost.my.domain
192.168.0.150           beasty beasty.home.lan
```
For the web I use the dns server from my registrar [123reg]
I can ping mydomain.com from inside and outside of my LAN,FreeBSD as well as windows machine..
Maybe I am missing your point . sorry


----------



## kpa (Mar 4, 2014)

*Re: Zabbix and jails redirection*

What do you have in /etc/resolv.conf on FREEBSD_HOST?


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				kpa said:
			
		

> What do you have in /etc/resolv.conf on FREEBSD_HOST?


Well, before I installed unbound dns I had 
`root@FREEBSD_HOST:~ # vi /etc/resolv.conf.20140228.222202`

```
# Generated by resolvconf
nameserver 194.168.4.100
nameserver 194.168.8.100
```
since the installation of unbound, I have a new file
`root@FREEBSD_HOST:~ # vi /etc/resolv.conf`

```
# Generated by resolvconf
# nameserver 194.168.4.100
# nameserver 194.168.8.100
search etsitservices.lan
nameserver 127.0.0.1
options edns0
```


----------



## kpa (Mar 4, 2014)

*Re: Zabbix and jails redirection*

Does this run on the FREEBSD_HOST return the correct address for mydomain.com?

`drill mydomain.com @127.0.0.1`


----------



## _martin (Mar 4, 2014)

*Re: Zabbix and jails redirection*

Oh, ok, right. I got confused there a bit. 
The setup without NAT is OK and it should be kept as is. We know that you can reach a wwwjail within your network without problem (by means of IP connectivity).

On notebook you are resolving mydomain.com to 82.x.x.x ! That means you are trying to reach your router, not the wwwjail.

Do you really need to have a DNS daemon running on freebsdhost ? First try to work it out without it, then move to the DNS configuration itself (note that this has not much to do with the original post though).

You could run a DNS service on the router (usually some sort of dnsmasq daemon even up by default). This DNS (on router) can set your ISP's DNS as forwarders (those virgin media DNS - 194.168.x.x). Assuming you have some sort of home router, this is done automatically when DHCP is set on outgoing interface (without default DNS settings that is).
If you go with this option all your LAN nodes/computers should set primary DNS as 192.168.0.1. 

You could do a simple test. Edit /etc/hosts on freebsdhost notebook wwwjail and add the following line: 


```
192.168.0.115   mydomain.com mydomain
```

And try to reach/resolve the wwwjail IP from your notebook again.

Now if you want to reach your web from outside, you do need to have a NAT/rdr set on router. NAT you already have. Then only to set the rdr of port 80 to 192.168.0.115 on router. 

There _is a question how come it was working with rdr rule. That I need to think more about.


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				kpa said:
			
		

> Does this run on the FREEBSD_HOST return the correct address for mydomain.com?
> 
> `drill mydomain.com @127.0.0.1`


`root@FREEBSD_HOST:~ # drill mydomain.com @127.0.0.1`

```
;; ->>HEADER<<- opcode: QUERY, rcode: SERVFAIL, id: 32291
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; mydomain.com.   IN      A

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 46 msec
;; SERVER: 127.0.0.1
;; WHEN: Tue Mar  4 20:41:09 2014
;; MSG SIZE  rcvd: 35
```
I have no idea what this output mean


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*

Hi @matoatlantis
Ok I don't need a dns but I tought this could be the reason my i could not access the www.
To clarify, the mydomain.com entry on the 123reg dns has got the following entries

```
DNS entry 	Type 	Priority 	TTL 	Destination/Target
               @		   A 			                  82.36.74.191		 	         
                www		   A 			                  82.36.74.191
```
On my virgin router I have a forwarding rule that redirect port 80 & 443 to 192.168.0.198. From there pf does the rest.
With the curent pf.conf file, I have no access to the website unless I add the rdr rule to the pf.conf
Should I uninstall unbound and restore the original /etc/resolve.conf file?


----------



## _martin (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> On my virgin router I have a forwarding rule that redirect port 80 & 443 to 192.168.0.198.


Bingo! We got it .. this is the problem. Set it to 192.168.0.115. Your www IP is the local IP on the net. And that would explain why you could access the web from outside if you used the rdr rule.

*EDIT:* When it comes to the DNS and stuff. You really don't need it here. You have no benefit of it. Many routers (here talking about this home ones) do have a way of doing name service for you. Those would be recognizable by your LAN. Once your name service is running, set the 192.168.0.1 as a primary DNS on all LAN hosts (those who use static IP config that is).


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				matoatlantis said:
			
		

> fred974 said:
> 
> 
> 
> > Bingo! We got it .. this is the problem. Set it to 192.168.0.115. Your www IP is the local IP on the net. And that would explain why you could access the web from outside if you used the rdr rule.


Bingo indeed, I have access to the website. It make sense now that I now what the problem is...
@matoatlantis I'll have to buy you a beer :beergrin  if you are near me {UK,Leeds}
All the command you provided me is now in my list of stuff to know and learn

...Now my final question in this post... is it more secure to redirect my router to 19.168.0.198 and then use pf to redirect to wwwjail or to let the router to redirect to wwwjail directly?


----------



## _martin (Mar 4, 2014)

*Re: Zabbix and jails redirection*



			
				fred974 said:
			
		

> @matoatlantis I'll have to buy you a beer :beergrin  if you are near me {UK,Leeds}


Hehe, np mate. Well, you'll never know. Maybe some day.  



			
				fred974 said:
			
		

> ...Now my final question in this post... is it more secure to redirect my router to 19.168.0.198 and then use pf to redirect to wwwjail or to let the router to redirect to wwwjail directly?


No, don't .. on router redirect it to 192.168.0.115. No rdr rule on your FreeBSD host. You must understand that it doesn't matter that that IP is used in a jail. It's still the local IP address as any other on that 192.168.0.0/24 network. 

And when do you want to use rdr rule ? For example if your FreeBSD host is the actual router. That means it has a public IP on the egress (outgoing) interface, let's say bge0, and you have your LAN connected on your ingress interface bge1. You would set the 192.168.0.115 as an alias on your bge1 interface and you would do the following rdr rule in your pf.conf (strictly binding to www IP only here): 


```
nat pass on bge0 from $WWW_IP to any -> $IP_PUBLIC
rdr pass on bge0 from any to $IP_PUBLIC port {80,443} -> $IP_WWW
```

.. but that's not your case here.


----------



## fred974 (Mar 4, 2014)

*Re: Zabbix and jails redirection*

Thank you mate  post closed


----------

