# Almost There: CARP Issues



## Fatman (Feb 17, 2009)

I spent approximately a month trying to get this working however I just don't seem to be able to. At first I tried using FreeBSD but I got to a point where preempt would just not work. I replaced all the nics but still had the same problem. 

I have now installed OpenBSD4.4 and preempt seems to be workng!!!! 

All Carp interfaces are in Master on FWA and backups on FWB. Shutting down an interface on A cause B to take over. But now I've ran into a new issue. Trying to follow the traffic using tcpdump, i can see it on carp2[ext_if] but not on carp1[dmz_if]. Traffic to the WEB or RDP server works fine until the failover occurs. Ex: i see traffice on carp1 and carp2 when FWA Master
# tcpdump -n -i carp2
tcpdump: listening on carp2, link-type EN10MB
12:05:00.541875 x.x.x.x > 192.168.1.105.3389: S 1651566250:1651566250(0) win 16384 <mss 1260,nop,nop,sackOK> (DF)
12:05:00.625594 x.x.x.x.1619 > 192.168.1.105.3389: . ack 2242405241 win 17640 (DF)

# tcpdump -n -i carp1
tcpdump: listening on carp1, link-type EN10MB
12:06:14.813558 172.16.0.189.3389 > x.x.x.x.1623: S 3718839503:3718839503(0) ack 3600434287 win 65535 <mss 1460,nop,nop,sackOK> (DF)
12:06:14.937956 172.16.0.189.3389 > x.x.x.x.1623: P 1:12(11) ack 20 win 65516 (DF)

When FWB takes over, no traffic is seem on carp1 or 2.


My PF files are pretty much identical asides from the fw_addr.

################################################################################
# Macro Defines
################################################################################
lop_int="lo0"
hrt_int="fxp0"
ext_int="rl1"
int_int="rl0"
dmz_int="rl2"

ftp_ports="{ 21,60000:60049 }"
email_ports="{ 25,110 }"
webmail_ports="{ 32000,32001 }"
ssh_fw="{ 22 }"
ssh="22"
www="{80, 443}"
vpn="1194"
rdp="3389"

vpn_server="{172.16.0.200}"
www_server="{172.16.0.80}"
rdp_server="{172.16.0.189}"

trusted_www="192.168.1.1"
trusted_vpn="192.168.1.1"

fw_addr="192.168.1.104"
dmz_nets="{172.16.0.0/24}"
internal_nets="{192.168.20.0/24}"

nat on $ext_int from $dmz_nets to any -> ($ext_int)
nat on $ext_int from $internal_nets to any -> ($ext_int)

#rdr on $ext_int proto {tcp, udp} from $trusted_vpn to any port $rdp -> $rdp_server
rdr on $ext_int proto tcp from any to any port $www -> $www_server
rdr on $ext_int proto {udp} from any to any port 1194 -> $vpn_server
#rdr on $ext_int proto tcp from any to any port 22280 -> $www_server
rdr on $ext_int proto tcp from any to port $rdp -> $rdp_server


################################################################################
# Filtering Rules
################################################################################
# Set default policy
block log all

# Log any connection attempt to the firewall
#block in log on $ext_int from any to $fw_addr

# Allow SSH Access From Trusted on External To The FW
pass in log quick on $ext_int proto tcp from any to $fw_addr port $ssh_fw keep state

# Allow VPN Traffic To VPN Server
pass in quick on $ext_int proto udp from any to $fw_addr port $vpn keep state
pass quick proto udp from any to $vpn_server port $vpn keep state

# Allow DNS To DMZ
pass quick proto {tcp,udp} from any to $dmz_nets keep state

# Allow all Loopback
pass quick on $lop_int all

# Allow pfsync Updates In/Out
pass quick on $hrt_int proto pfsync keep state

# Allow CARP Advertisements In/Out
pass quick on {$hrt_int, $ext_int, $int_int, $dmz_int} proto carp keep state

# Allow WWW Traffic To WWW Server
pass in quick on $ext_int proto tcp from any to $fw_addr port $www keep state
pass quick proto tcp from any to $www_server port $www keep state

# Allow RDP traffic To RDP Server
pass in quick on $ext_int proto tcp from any to $fw_addr port $rdp keep state
pass quick proto tcp from any to $rdp_server port $rdp keep state

# Allow SSH Traffic To WWW Server
#pass in quick on $ext_int proto tcp from any to $www_server port 22280 keep state
#pass quick proto tcp from any to $www_server port 22280 keep state


# Allow HTTP Through
pass in quick on $ext_int proto tcp from any to $www_server port $www keep state
#pass out quick on $dmz_int proto tcp from any to $www_server port $www keep state

# Allow all outgoing traffic
pass in quick on $int_int all keep state
pass out quick on $ext_int all keep state

# Allow Pings
#pass in quick on $ext_int proto icmp from any to  keep state
pass out quick on $int_int proto icmp from any to any keep state

# Allow Pings to Firewall
#pass in quick on $ext_int proto icmp from any to $fw_addr keep state

# Allow Terminal Services
#pass in quick on $ext_int proto tcp from  to  port 3389 keep state
#pass out quick on $int_int proto tcp from  to  port 3389 keep state

# Allow SSL Through
#pass in quick on $ext_int proto tcp from any to  port 443 keep state
#pass out quick on $int_int proto tcp from any to  port 443 keep state

# Allow FTP Through
#pass in quick on $ext_int proto tcp from any to  port $ftp_ports keep state
#pass out quick on $int_int proto tcp from any to  port $ftp_ports keep state

# Allow Email Through
#pass in quick on $ext_int proto tcp from any to  port $email_ports keep state
#pass out quick on $int_int proto tcp from any to  port $email_ports keep state

# Allow Webmail Through
#pass in quick on $ext_int proto tcp from any to  port $webmail_ports keep state
#pass out quick on $int_int proto tcp from any to  port $webmail_ports keep state

# Allow DNS Through
#pass in quick on $ext_int proto { tcp, udp } from any to  port 53 keep state
pass out quick on $int_int proto { tcp, udp } from any to  port 53 keep state

Any help would be much appreciated, I need this up and running for a school project which is comming up next week.


----------



## Fatman (Feb 17, 2009)

Here's my configs:

FWA
inet 10.1.0.1 255.255.255.0 10.1.0.255 vhid 1 pass 001vCarpAdvert
inet 172.16.0.1 255.255.255.0 172.16.0.255 vhid 2 pass 001vCarpAdvert
inet 192.168.1.105 255.255.255.0 192.168.1.255 vhid 3 pass 001vCarpAdvert
inet 192.168.20.105 255.255.255.0 192.168.20.255 vhid 4 pass 001vCarpAdvert
inet 10.1.0.103 255.255.255.0 NONE
up syncif fxp0
inet 192.168.20.1 255.255.255.0 NONE
inet 192.168.1.103 255.255.255.0 NONE
inet 172.16.0.103 255.255.255.0 NONE

FWB
inet 10.1.0.1 255.255.255.0 10.1.0.255 vhid 1 pass 001vCarpAdvert advskew 100
inet 172.16.0.1 255.255.255.0 172.16.0.255 vhid 2 pass 001vCarpAdvert advskew 100
inet 192.168.1.105 255.255.255.0 192.168.1.255 vhid 3 pass 001vCarpAdvert advskew 100
inet 192.168.20.105 255.255.255.0 192.168.20.255 vhid 4 pass 001vCarpAdvert advskew 100
inet 10.1.0.104 255.255.255.0 NONE
up syncif fxp0
inet 192.168.20.104 255.255.255.0 NONE
inet 192.168.1.104 255.255.255.0 NONE
inet 172.16.0.104 255.255.255.0 NONE


----------



## Fatman (Feb 17, 2009)

ok, rebooting the servers has brought traffic to carp2[ext_if] after failing FWA.

FWB# tcpdump -n -i carp2
tcpdump: listening on carp2, link-type EN10MB
13:56:30.036500 173.32.139.31.2081 > 192.168.1.105.3389: S 1381720291:1381720291(0) win 16384 <mss 1260,nop,nop,sackOK> (DF)
13:56:32.929226 173.32.139.31.2081 > 192.168.1.105.3389: S 1381720291:1381720291(0) win 16384 <mss 1260,nop,nop,sackOK> (DF)
13:56:38.958281 173.32.139.31.2081 > 192.168.1.105.3389: S 1381720291:1381720291(0) win 16384 <mss 1260,nop,nop,sackOK> (DF)

# tcpdump -n -i carp1
tcpdump: listening on carp1, link-type EN10MB
*nothing*

pfsync0: flags=41<UP,RUNNING> mtu 1460
        pfsync: syncdev: fxp0 maxupd: 128
        groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33204
        groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:01
        carp: MASTER carpdev fxp0 vhid 1 advbase 1 advskew 100
        groups: carp
        inet 10.1.0.1 netmask 0xffffff00 broadcast 10.1.0.255
        inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x7
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:02
        carp: MASTER carpdev rl2 vhid 2 advbase 1 advskew 100
        groups: carp
        inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
        inet6 fe80::200:5eff:fe00:102%carp1 prefixlen 64 scopeid 0x8
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:03
        carp: MASTER carpdev rl1 vhid 3 advbase 1 advskew 100
        groups: carp
        inet 192.168.1.105 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::200:5eff:fe00:103%carp2 prefixlen 64 scopeid 0x9
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:04
        carp: MASTER carpdev rl0 vhid 4 advbase 1 advskew 100
        groups: carp
        inet 192.168.20.105 netmask 0xffffff00 broadcast 192.168.20.255
        inet6 fe80::200:5eff:fe00:104%carp3 prefixlen 64 scopeid 0xa


----------



## Fatman (Feb 17, 2009)

Found the issue, forgot to edit the sysctl file for forwarding.


----------



## DutchDaemon (Feb 17, 2009)

Devil in the details ..


----------

