# 'named: not enough free resources' gets box offline



## joint (Jan 12, 2010)

It happend couple times when the production box went offline but still running.
The Reset helped only.
Later on I found in: 
/var/log/messages 
a lot of strings matching exactly at the time when box went offline:

```
Jan  8 20:00:00 freebsdbox named[605]: client 212.0.221.131#63394: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 193.226.5.151#46650: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 4.68.25.4#38728: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 212.0.219.67#39168: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.101.62#21972: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.86.243#37008: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 212.93.158.242#21273: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.101.62#51570: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 212.0.209.131#24539: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 67.202.21.1#13739: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 85.37.17.17#52719: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.182.213#28260: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 193.226.5.151#34352: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 193.226.6.233#54745: error sending response: not enough free resources
```

Just in case may be interested in current output for network_card is:
`# # ifconfig -a`

```
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:1d:92:61:54:95
        inet xx.xx.69.211 netmask 0xffffffe0 broadcast xx.xx.69.223
        inet xx.xx.215.24 netmask 0xffffffff broadcast xx.xx.215.24
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet6 ::1 prefixlen 128 
        inet 127.0.0.1 netmask 0xff000000
```

Any ideas are very welcome on where to dig so to solve the issue ASAP, please.


----------



## SirDice (Jan 12, 2010)

Check the output of [cmd=]netstat -m[/cmd].


----------



## joint (Jan 12, 2010)

Here is:
`# # netstat -m`

```
66/1854/1920 mbufs in use (current/cache/total)
64/466/530/65536 mbuf clusters in use (current/cache/total/max)
64/448 mbuf+clusters out of packet secondary zone in use (current/cache)
0/199/199/32768 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
144K/2191K/2336K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
```


----------



## SirDice (Jan 12, 2010)

Check you firewall (if you have one running). Also make sure you don't have any network problems like a dodgy card or cable.


----------



## joint (Jan 12, 2010)

Ok, regarding making sure about any network problems like a dodgy card or cable -- it's hard to go for it, because the box inside the DataCenter.
So, I've just disabled IPFW to see if it could cause the issue.
And for your review, here is the:
`# # ipfw list`

```
[color="Red"]00100 allow ip from any to any via lo0[/color]
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 reset tcp from 173.45.76.212 to me dst-port 80
00500 reset tcp from 202.222.31.109 to me dst-port 80
00600 reset tcp from 74.143.62.214 to me dst-port 80
00700 reset tcp from 87.106.252.228 to me dst-port 80
00800 reset tcp from 115.68.53.195 to me dst-port 80
00900 reset tcp from 202.73.57.25 to me dst-port 80
01000 reset tcp from 195.154.126.140 to me dst-port 80
01100 allow tcp from any to me dst-port 80
01200 reset tcp from 202.73.57.25 to me dst-port 22
01300 reset tcp from 115.68.53.195 to me dst-port 22
01400 reset tcp from 202.154.57.33 to me dst-port 22
01500 reset tcp from 87.106.252.228 to me dst-port 22
01600 reset tcp from 74.143.62.214 to me dst-port 22
01700 reset tcp from 202.222.31.109 to me dst-port 22
01800 reset tcp from 173.45.76.212 to me dst-port 22
01900 reset tcp from 187.5.128.234 to me dst-port 22
02000 reset tcp from 124.42.41.162 to me dst-port 22
02100 reset tcp from 195.154.126.140 to me dst-port 22
02200 allow tcp from any to me dst-port 22
02300 reset tcp from 173.45.76.212 to me dst-port 21
02400 reset tcp from 202.222.31.109 to me dst-port 21
02500 reset tcp from 74.143.62.214 to me dst-port 21
02600 reset tcp from 87.106.252.228 to me dst-port 21
02700 reset tcp from 115.68.53.195 to me dst-port 21
02800 reset tcp from 202.73.57.25 to me dst-port 21
02900 reset tcp from 130.58.84.55 to me dst-port 21
03000 reset tcp from 75.151.84.17 to me dst-port 21
03100 reset tcp from 87.237.208.232 to me dst-port 21
03200 reset tcp from 85.17.45.56 to me dst-port 21
03300 reset tcp from 66.199.248.58 to me dst-port 21
03400 reset tcp from 195.154.126.140 to me dst-port 21
03500 allow tcp from any to me dst-port 21
65535 allow ip from any to any
```

*The strange thing for me is the very 1st line ouput stating
that Netword Card Interface is '*lo0*', while real is '*re0*'


----------



## SirDice (Jan 12, 2010)

joint said:
			
		

> The strange thing for me is the very 1st line ouput stating
> that Netword Card Interface is '*lo0*', while real is '*re0*'



lo0 is the loopback adapter (where 127.0.0.1 is bound to).


----------



## joint (Jan 12, 2010)

Ok, Thank you for your explanation regarding 'lo0'
and mega-appreciate you for the things pointed to me.
Apart from them is it any extra ideas of what may cause the issue?
In the mean time I'm runnig with IPFW disabled.
Here is the my system details:
`# # uname -a`

```
7.0-RELEASE FreeBSD 7.0-RELEASE #0: Thu Feb 28 05:32:36 UTC 2008     [email]root@orion.ispsystem.net[/email]:/usr/obj/usr/src/sys/ISPSYSTEM  amd64
```


----------



## SirDice (Jan 12, 2010)

joint said:
			
		

> Apart from them is it any extra ideas of what may cause the issue?
> In the mean time I'm runnig with IPFW disabled.


Out of ideas but I am wondering if it works with IPFW disabled. If that's the case then the problem is with the firewall and/or it's settings.



> Here is the my system details:
> # uname -a
> 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Thu Feb 28 05:32:36 UTC 2008     root@orion.ispsystem.net:/usr/obj/usr/src/sys/ISPSYSTEM  amd64


I do hope named isn't accessible from the outside. I strongly suggest updating.


http://security.freebsd.org/advisories/FreeBSD-SA-08:06.bind.asc
http://security.freebsd.org/advisories/FreeBSD-SA-09:04.bind.asc
http://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc


----------



## joint (Jan 12, 2010)

SirDice said:
			
		

> I do hope named isn't accessible from the outside. I strongly suggest updating.
> 
> http://security.freebsd.org/advisories/FreeBSD-SA-08:06.bind.asc
> http://security.freebsd.org/advisories/FreeBSD-SA-09:04.bind.asc
> http://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc



Ok, I've never done any patching before and would like to ask you let me know specifics about it.
1. There are /usr/src but it is empty...probably I have to do something for it?
2. Is it correct to do all 3 patches one by another rebooting the box everytime starting from the top link you've provided?
3. The current kernel I've got is not default GENERIC, 
   it is custom and has been created when I've installed ISPManager Server Control Panel http://ispsystem.com/en/
and not sure if I'll be able to start up the box once things will be patched.
Your thoughts & instructions, please?


----------



## SirDice (Jan 12, 2010)

1) http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/updating-upgrading.html
The source update mentions cvsup, there's no need to install it. Since 6.4 the base OS has csup which is similar in features and configuration.

2) You can do all patches at once and build just once.

3) Shouldn't be a problem but it does mean you can't use freebsd-update. If you want to use that you'll need to temporarily switch back to GENERIC.

As for updating, that last bind problem isn't fixed in 7.0 because 7.0 isn't supported anymore. If you update I suggest updating to 7.2-RELEASE-p6. This should be a relatively easy update. No need to rebuild any ports AFAIK. Do check /usr/src/UPDATING before starting.


----------



## joint (Jan 12, 2010)

Thank you very much!
I'll try to do all that in the night time.
As soon as I'll get things done I'll be back with the feedback.
Once again I appreciate you for your time & effort.


----------



## joint (Jan 24, 2010)

Here is the Feedback:
I Found that command 'top' has strange output now giving this line:

```
[I]top: sysctl(vfs.bufspace...) expected 8, got 4[/I]
```
I've got this right after I had made:

```
# csup -g -L2 /usr/local/etc/stable-supfile
# cd /usr/src && make buildworld && make installworld
# reboot
# mergemaster -Ui
# reboot
```

I assume that I have to change some values in sysctl.conf,
but I'm not sure which one exactly needs to be changed.
The Amount of RAM installed is 4Gb.
Any ideas, please?


----------



## DutchDaemon (Jan 24, 2010)

You mean you updated the world, but *not* the kernel? That's a recipe for disaster, and it is why 'top' is screwed: the kernel doesn't understand userland anymore. What made you take the steps listed above, instead of the proper procedure listed in /usr/src/Makefile?


----------



## joint (Jan 24, 2010)

Well, looks like now I'm screwed up for sure.
Is it any short-hand out_of_current_disaster recipe you may suggest to me?
Bear in mind, that I have not GENERIC kernel running.
And when I tried 'freebsd-update fetch' 'freebsd-update install' and then reboot - I lost box available by SSH and then had to wait for hours when Data-Center support getting me KVM_over_IP console connected and only 'freebsd-update  rollback' could get that box back online.
Is it any specific suggestions on this, please?


----------



## DutchDaemon (Jan 24, 2010)

*Either* use freebsd-update, *or* update from sources, and *if* you decide to update world/kernel from sources, _use the correct and full procedure_. 

You can use a non-GENERIC kernel with freebsd-update (by letting freebsd-update update the sources it uses to build its binaries, and building a custom kernel for your system, while keeping the GENERIC kernel in /boot/GENERIC) - but do *not* mix _freebsd-update_ and _make world_! Use one or the other, and stick with it.

I have no idea what your system is running right now, but I suggest you first run freebsd-update and reinstall and reboot the GENERIC kernel installed by freebsd-update (make sure 'src' is updated as well), and then try building a custom kernel again (keeping a copy of the generic kernel in /boot/GENERIC).

The Handbook has more on that subject.


----------



## dennylin93 (Jan 25, 2010)

If you have a CD/DVD with FreeBSD, you can use it to get a GENERIC kernel as well.


----------



## joint (Jan 28, 2010)

Ok, thank you guys for making things brighter now.
So, let me know, please, if I've got you the right way describing steps below with commands:
1. To update /usr/ports;

2. To update the 'src' with:

```
# csup -g -L2 /usr/local/etc/stable-supfile
```

3. To update FreeBSD -- I've got Release 7.0 for now:

```
# freebsd-update fetch
   # freebsd-update install
```

4. Build a custom kernel with needed OPTIONS enabled:

```
# cd /usr/src/sys/`uname -p`/conf
   # cp GENERIC ISPSYSTEM         --The ISPSYSTEM is the name of my current custom kernel + with 3 OPTIONS added to GENERIC 
   # vi ISPSYSTEM
... add those 3 OPTIONS that needed over again and after that go:
   # cd /usr/src          
 [color="DarkOrange"]  # make buildworld  --I'm NOT sure if should I do this command??[/color]
   # make buildkernel KERNCONF=ISPSYSTEM   
   # make installkernel KERNCONF=ISPSYSTEM
```

5. Reboot:

```
# shutdown -r now
```

*Are those steps above sound correct to follow?*
Please, correct me if I'm wrong somewhere.
Thank you!


----------



## joint (Jan 29, 2010)

dennylin93 said:
			
		

> If you have a CD/DVD with FreeBSD, you can use it to get a GENERIC kernel as well.


That is the case, that I've got this box remotely far away installed in Data_Center.
Thanks for paying attention to my post as well as advice


----------

