# 8-STABLE currently broken?



## raphidae (Apr 1, 2010)

Yesterday (30/03) after updating a server (Dell PE 1950) several things started to break. The server had been rock-solid for months, and after downgrading to the 8-STABLE from 01-01-2010 everything returned to normal.

I'm not quite sure what's wrong as there are several things that break and dumps don't get written on panics.

What broke:

It's a mail server, running Communigate Pro (tried 5.1, 5.2 & 5.3) and accepting connections on both ipv4 and ipv6. The server uses v6 sockets for v4 addresses, like this:

```
CGServer 826 root   45u  IPv6 0xffffff00174c8a50      0t0  TCP [2001:610:xxx:xxx:xxx:xxx:xxx:200]:smtp (LISTEN)
CGServer 826 root   47u  IPv6 0xffffff0001faf370      0t0  TCP [::217.xxx.xxx.xxx]:smtp (LISTEN)
```

Directly after the upgrade I noticed that connections *out* to other ipv4 mailservers were no longer succeeding and ipfw was seeing some weird packets:

```
Mar 30 06:27:34 adinava kernel: ipfw: 65530 Accept TCP 1.23.2.0:28859 65.55.92.152:25 out via bce0
```

Obviously 1.23.2.0 is not a local IP, so I checked lsof:

```
CGServer 824 root   49u  IPv6 0xffffff00174b3a50      0t0  TCP [2001:610:xxx:xxx:xxx:xxx:xxx:200]:28859->[::65.55.92.152]:smtp (SYN_SENT)
```
Somehow the server was trying to connect to an ipv4 address from an ipv6 address, where the ipv6 address apparently overflows ipv4 storage and ends up being '1.23.2.0'...

For comparison, this is what it looks like when it works:

```
CGServer 105 root   94u  IPv6 0xffffff00a4ccd000      0t0  TCP [::217.195.117.200]:14532->[::65.55.92.152]:smtp  (ESTABLISHED)
```

At first I assumed this was a problem in the daemon, so I temporarily disabled ipv6 for CGatePro (the ipv6 is not yet added as MX anyway) and forced it to use ipv4 sockets. That worked, until I tried to reload the ipfw rules and hit a panic:


```
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff803e3b77
stack pointer           = 0x28:0xffffff8076d73890
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1467 (ipfw)
trap number             = 9
panic: general protection fault
cpuid = 0
Uptime: 4m24s
Cannot dump. Device not defined or unavailable.
panic: bufwrite: buffer is not busy???
cpuid = 0
Uptime: 4m24s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
Automatic reboot in 15 seconds - press a key on the console to abort
ipfw: ouch!, skip past end of rules, denying packet
```

It should have dumped (device is defined) and rebooted, but it hung there.

When I rebooted it my rules file (/etc/ipfw.rules.sh) was truncated to zero. Assuming this was an ipfw problem I left the rules out temporarily (I have IPFIREWALL_DEFAULT_TO_ACCEPT).

After ~4 minutes the server hung again, this time with the screen filled with 'ipfw: ouch!, skip past end of rules, denying packet' messages, these were also logged to /var/log/messages. This seemed a bit weird as only the default 'allow-all' rule was present.

So I decided to recompile the kernel without ipfw and reboot. After again ~4m I got the following panic:


```
dev = mfid0s1f, block = 1, fs = /var
panic: ffs_blkfree: freeing free block
cpuid = 3
Uptime: 4m34s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
```

Again the server didn't reboot but just froze (no num-lock LED action either).

I think a change to 8-STABLE between 01/01/2010 and 30/03/2010 seriously broke something, but I have no idea of how to go about finding out what exactly. There's no dumps and no logging other than the 'ipfw: ouch!' message.

Any ideas? dmesg & kernconf @ http://ra.phid.ae/dmesg.txt


----------



## phoenix (Apr 1, 2010)

Read the freebsd-ipfw and freebsd-stable mailing list archives.  This is covered in there.  Luigi is doing a lot of cleanups and fixes to IPFW right now.  Some of the cleanups broke things temporarily.  Updating your source tree to March 31 or later should fix things.

One of the dangers of running -STABLE without following the -stable mailing list.


----------



## raphidae (Apr 1, 2010)

I've tried removing ipfw completely, and that doesn't fix the problem.


----------



## raphidae (Apr 1, 2010)

Also, I do track the -stable mailinglist and I'm not seeing a heads-up or anything else on this.


----------



## raphidae (Apr 2, 2010)

The problem I'm having is definitely still present in 8-STABLE as of 2 april 2010. I updated source, recompiled and again when reloading ipfw rules and issuing a 'show' this happens:


```
[root@xxx ~]# ipfw show
00001                    0                    0 check-state
00004                    0                    0 allow ip4 from any to any via lo0
00006                    0                    0 allow ip6 from any to any via lo0
00579                    0                    0 count ip from any to any in via bce1 not proto udp not proto tcp not proto icmp not proto ipv6-icmp
00589                    0                    0 count ip from any to any out via bce1 not proto udp not proto tcp not proto icmp not proto ipv6-icmp
00900                    0                    0 count ip from any to any not // ***remark: handle traffic via inside adapter bce1
00904                    0                    0 deny ip4 from any to 10.0.255.255,10.255.255.255 in via bce1
00914                    0                    0 allow ip4 from 10.0.0.0/8 to me in via bce1
00924                    0                    0 allow ip4 from me to 10.0.0.0/8 out via bce1
00979                    0                    0 deny ip from any to any in via bce1
00989                    0                    0 deny ip from any to any out via bce1
01000                    0                    0 count ip from any to any not // ***remark: deny incoming traffic from bogons
01174                    0                    0 deny log logamount 200 ip4 from 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,0.0.0.0/8,169.254.0.0/16,192.0.2.0/24,224.0.0.0/3,127.0.0.0/8 to any in via bce0
03000                    0                    0 count ip from any to any not // ***remark: configure access for service 0 - SSH daemon and http-mgmt ports
03214                   14                 1080 allow ip4 from any to any proto tcp src-ip 85.147.xxx.xxx dst-ip me dst-port 22,280 in via bce0
03216                    0                    0 allow ip6 from any to any proto tcp { src-ip6 2001:610:xxx::/48 or src-ip6 2001:610:xxx::/48 } dst-ip6 me6 dst-port 22,280 in via bce0
03224                   12                 1168 allow ip4 from any to any proto tcp src-ip me src-port 22,280 dst-ip 85.147.xxx.xxx out via bce0
03226                    0                    0 allow ip6 from any to any proto tcp src-ip6 me6 src-port 22,280 { dst-ip6 2001:610:xxx::/48 or dst-ip6 2001:610:xxx::/48 } out via bce0
03274                    0                    0 unreach filter-prohib log logamount 200 ip4 from any to any proto tcp dst-ip me dst-port 22,280 in via bce0
03276                    0                    0 unreach6 admin-prohib log logamount 200 ip6 from any to any proto tcp dst-ip6 me6 dst-port 22,280 in via bce0
13000                    0                    0 count ip from any to any not // ***remark: configure access for service 1 - identd
13214                    0                    0 allow ip4 from any to any proto tcp dst-port 113 in via bce0 limit src-addr 4
13216                    0                    0 allow ip6 from any to any proto tcp dst-port 113 in via bce0 limit src-addr 4
13224                    0                    0 allow ip4 from any to any proto tcp src-port 113 out via bce0
13226                    0                    0 allow ip6 from any to any proto tcp src-port 113 out via bce0
13274                    0                    0 unreach filter-prohib log logamount 200 ip4 from any to any proto tcp dst-port 113 in via bce0
13276                    0                    0 unreach6 admin-prohib log logamount 200 ip6 from any to any proto tcp dst-port 113 in via bce0
23000                    0                    0 count ip from any to any not // ***remark: configure access for service 2 - mail daemon
23214                    1                   64 allow ip4 from any to any proto tcp dst-ip 217.195.xxx.xxx dst-port 25,80,110,143,443,993,995 in via bce0
23216                    0                    0 allow ip6 from any to any proto tcp dst-ip6 2001:610:xxx:0:217:195:xxx:xxx dst-port 25,80,110,143,443,993,995 in via bce0
23224                    1                   60 allow ip4 from any to any proto tcp src-ip 217.195.xxx.xxx src-port 25,80,110,143,443,993,995 out via bce0
23226                    0                    0 allow ip6 from any to any proto tcp src-ip6 2001:610:xxx:0:217:195:xxx:xxx src-port 25,80,110,143,443,993,995 out via bce0
23274                    0                    0 unreach filter-prohib log logamount 200 ip4 from any to any proto tcp dst-ip 217.195.xxx.xxx dst-port 25,80,110,143,443,993,995 in via bce0
23276                    0                    0 unreach6 admin-prohib log logamount 200 ip6 from any to any proto tcp dst-ip6 2001:610:xxx:0:217:195:xxx:xxx dst-port 25,80,110,143,443,993,995 in via bce0
56000                    0                    0 count ip from any to any not // ***remark: handle icmp
56914                    0                    0 allow ip4 from any to any proto icmp dst-ip me in via bce0 icmptypes 3,8,11
56916                    0                    0 allow ip6 from any to any proto ipv6-icmp dst-ip6 me6 in via bce0 ip6 icmp6types 1,2,3,4,128
56916                    0                    0 allow ip6 from any to any proto ipv6-icmp { dst-ip6 me6 or dst-ip6 fe80::/10 or dst-ip6 ff02::1 } in via bce0 ip6 icmp6types 134,135,136
56924                    0                    0 allow ip4 from any to any proto icmp src-ip me out via bce0 icmptypes 0,3,11
56924                    0                    0 allow ip4 from any to any proto icmp src-ip me out via bce0 icmptypes 8 keep-state
56926                    0                    0 allow ip6 from any to any proto ipv6-icmp src-ip6 me6 out via bce0 ip6 icmp6types 1,2,3,4,129
56926                    0                    0 allow ip6 from any to any proto ipv6-icmp src-ip6 me6 out via bce0 ip6 icmp6types 128 keep-state
56926                    0                    0 allow ip6 from any to any proto ipv6-icmp { src-ip6 me6 or src-ip6 fe80::/10 or src-ip6 ff02::1 } out via bce0 ip6 icmp6types 135,136
56974                    0                    0 deny log logamount 200 ip4 from any to any proto icmp dst-ip me in via bce0
56976                    0                    0 deny log logamount 200 ip6 from any to any proto ipv6-icmp dst-ip6 me6 in via bce0
60000                    0                    0 count ip from any to any not // ***remark: configure stateful shit for egress traffic
60124                    0                    0 allow ip4 from any to any proto udp src-ip me src-port 20001-65535 out via bce0 keep-state
60126                    0                    0 allow ip6 from any to any proto udp src-ip6 me6 src-port 20001-65535 out via bce0 keep-state
60154                    0                    0 allow ip4 from any to any proto udp dst-ip me dst-port 33434-33525 in via bce0 limit src-addr 4
60156                    0                    0 allow ip6 from any to any proto udp dst-ip6 me6 dst-port 33434-33525 in via bce0 limit src-addr 4
60174                    0                    0 deny log logamount 200 ip4 from any to any proto udp dst-ip me in via bce0
60176                    0                    0 deny log logamount 200 ip6 from any to any proto udp dst-ip6 me6 in via bce0
60184                    0                    0 unreach filter-prohib log logamount 200 ip4 from any to any proto udp src-ip me out via bce0
60186                    0                    0 unreach6 admin-prohib log logamount 200 ip6 from any to any proto udp src-ip6 me6 out via bce0
60224                    0                    0 allow ip4 from any to any proto tcp src-ip me src-port 20001-65535 out via bce0 keep-state
60226                    0                    0 allow ip6 from any to any proto tcp src-ip6 me6 src-port 20001-65535 out via bce0 keep-state
60274                    0                    0 unreach filter-prohib log logamount 200 ip4 from any to any proto tcp dst-ip me in via bce0
60276                    0                    0 unreach6 admin-prohib log logamount 200 ip6 from any to any proto tcp dst-ip6 me6 in via bce0
60284                    0                    0 unreach filter-prohib log logamount 200 ip4 from any to any proto tcp src-ip me out via bce0
60286                    0                    0 unreach6 admin-prohib log logamount 200 ip6 from any to any proto tcp src-ip6 me6 out via bce0
65529                    0                    0 allow log logamount 200 ip4 from any to any
65530                    0                    0 allow log logamount 200 ip6 from any to any
00001                    0                    0 check-state
00000                  113 15691514744433505686  ip from any to any
00000                    0                    0  ip from any to any

<panic>
```


----------



## phoenix (Apr 3, 2010)

raphidae said:
			
		

> Also, I do track the -stable mailinglist and I'm not seeing a heads-up or anything else on this.



Hrm, then it's on the -ipfw list.  I read too many lists to keep things organised sometimes.  

That's definitely the issue that is discussed, though (00000 rule and strange number ordering).  The last post from Luigi was that it was fixed on April 01.

You should go through the -ipfw archives, and considering chiming in on that thread.


----------

