# Filesystem is full ( /: )



## Vib3 (Apr 22, 2009)

```
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a    496M    480M    -24M   105%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad0s1e    496M    2.0K    456M     0%    /tmp
/dev/ad0s1f    286G    2.6G    261G     1%    /usr
/dev/ad0s1d    1.2G    465M    658M    41%    /var


1       /COPYRIGHT
1       /bin
123     /boot
1       /cdrom
0       /compat
1       /dev
1       /dist
1       /entropy
2       /etc
0       /home
6       /lib
1       /libexec
1       /media
1       /mnt
1       /proc
4       /rescue
1       /root
4       /sbin
0       /sys
1       /tmp
2686    /usr
466     /var
```
---------------
I upgraded from 7.0 to 7.1 (freebsd-update upgrade -r 7.1-RELEASE)
First my var got full, but its fixed now.

Problem /: is full.

Runned fsck / no effect
I got 151M on /

Where is 329M of the data?


----------



## SirDice (Apr 22, 2009)

Here's a way to see where the data has gone:

```
# cd /
# du -sk * | sort -n
```

The biggest directory will be at the bottom of that list.


----------



## Vib3 (Apr 22, 2009)

```
du -sk * | sort -n
0       compat
0       home
0       sys
2       cdrom
2       dev
2       dist
2       media
2       proc
4       entropy
4       mnt
8       COPYRIGHT
12      tmp
54      root
172     libexec
990     bin
1788    etc
3744    rescue
3954    sbin
6030    lib
124936  boot
476624  var
2749610 usr
```

So in / its /boot with 123M.. But still doesnt explain where is 329M :/


----------



## DutchDaemon (Apr 22, 2009)

Vib3, use code tags around system output.


----------



## Vib3 (Apr 22, 2009)

Ok.

Any ideas ? How to clean ?

Just kernel there ~120M
Data is maybe corrupted, I think..


----------



## DutchDaemon (Apr 22, 2009)

Have you tried a reboot? This might be a stuck file descriptor hanging on to a file that's already been deleted from under it.


----------



## Vib3 (Apr 22, 2009)

DutchDaemon said:
			
		

> Have you tried a reboot? This might be a stuck file descriptor hanging on to a file that's already been deleted from under it.



Yeap. Server stopped from responding to network traffic, so I had to power it off.. And there was nothing in messages about that.


----------



## DutchDaemon (Apr 22, 2009)

Rather generic, but you may find some pointers here:
http://www.google.com/search?q=df+versus+du


----------



## Vib3 (Apr 22, 2009)

```
1788    ./etc
2       ./cdrom
2       ./dist
990     ./bin
22      ./boot/defaults
2       ./boot/firmware
124246  ./boot/kernel
2       ./boot/modules
2       ./boot/zfs
124936  ./boot
274     ./lib/geom
6030    ./lib
172     ./libexec
2       ./media
2       ./mnt/usbi
4       ./mnt
2       ./proc
3744    ./rescue
18      ./root/.irssi
2       ./root/.ssh
54      ./root
3954    ./sbin
3368090 .
```


----------



## SirDice (Apr 23, 2009)

My /boot/kernel only contains about 30MB, I'm wondering why yours has 125MB?


----------



## tangram (Apr 23, 2009)

SirDice said:
			
		

> My /boot/kernel only contains about 30MB, I'm wondering why yours has 125MB?



That's more or less the size of the binary kernel.


----------



## SirDice (Apr 23, 2009)

tangram said:
			
		

> That's more or less the size of the binary kernel.


Is there any other type of kernel? 


```
root@maelcum:/boot#du -sk * | sort -n
0       loader.conf
2       boot0
2       boot0sio
2       boot1
2       cdboot
2       device.hints
2       firmware
2       loader.rc
2       mbr
2       modules
2       pmbr
2       screen.4th
2       zfs
4       frames.4th
6       loader.4th
8       beastie.4th
8       boot
8       boot2
8       gptboot
16      loader.help
22      defaults
36      support.4th
272     loader
272     loader.old
288     pxeboot
30202   kernel.old
30738   kernel
```


----------



## tangram (Apr 23, 2009)

SirDice said:
			
		

> Is there any other type of kernel?



Wrong choice of words. I meant it using the GENERIC kernel configuration.


----------



## SirDice (Apr 23, 2009)

tangram said:
			
		

> Wrong choice of words. I meant it using the GENERIC kernel configuration.


I thought it would be something like that :e


----------



## anomie (Apr 23, 2009)

@Vib3: Something is obviously not adding up. How about a fsck of the / filesystem from single-user mode?


----------



## ephemera (Apr 23, 2009)

I once had a similiar problem, found out later that the FS snapshots I had made (in /.snap) were using up the extra space.


----------



## thetrojan01 (Apr 23, 2009)

I have the same problem too (/ is 108% full)

I will look this topic tomorow cuz i have to sleep!  good night everyone!


----------



## phoenix (Apr 23, 2009)

Also, you may want to boot into single-user mode, and check the size of / without any other filesystems mounted.  Run du from there.  It may be that there's something in /var (the mountpoint directory not the mounted filesystem) or /usr.


----------



## hydra (Apr 24, 2009)

Maybe check with this little tool:
sysutils/ncdu

After installing launch ncdu /


----------



## anomie (Apr 24, 2009)

ephemera said:
			
		

> I once had a similiar problem, found out later that the FS snapshots I had made (in /.snap) were using up the extra space.



@vib3: Assuming you haven't abandoned this thread... 

ephemera may be on to something with that suggestion. The du -ks invocation doesn't seem to pick up hidden (.) files. I have a quick find test you can run that will pick up said files. These points are demonstrated by the example that follows: 

```
%cat /dev/zero > .uh-oh
^C

%du -sk * | sort -n
2	tester.py
1632	foo-install-211.pdf
4036	python
14338	music

%find -x . -size +10000 -exec du -h {} \]

So, in your case, run: 
[code]# find -x / -size +10000 -exec du -h {} \;
```

See if it tells you a new story.


----------



## thetrojan01 (Apr 26, 2009)

*Solved mine problem*

I solved the problem for me! I had made a new kernel and didn't remove the old one GENERIC in /boot, so, the / it's 108% full because also the new kernel has 'root' as its owner.

It's now 64% full.

Also be careful if Desktop users run firefox as root... This creates big files in your directory. How you can avoid it: that's why there are adduser, su and sudo.
------------


----------



## jb_fvwm2 (Apr 26, 2009)

locate firefox | grep Cache
make a .sh removing Cache/*  after each firefox run
and it will be quicker


----------



## Vib3 (Apr 27, 2009)

```
%cat /dev/zero > .uh-oh

/: write failed, filesystem is full
cat: stdout: No space left on device

%df -h

Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a    496M    496M    -40M   109%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad0s1e    496M     50K    456M     0%    /tmp
/dev/ad0s1f    286G    2.6G    261G     1%    /usr
/dev/ad0s1d    1.2G    345M    779M    31%    /var

%find -x / -size +10000 -exec du -h {} \;

 14M    /.snap/.uh-oh
9.3M    /boot/kernel/kernel
 28M    /boot/kernel/kernel.symbols

after removing .uh-oh ->

%df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a    496M    481M    -25M   105%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad0s1e    496M     50K    456M     0%    /tmp
/dev/ad0s1f    286G    2.6G    261G     1%    /usr
/dev/ad0s1d    1.2G    345M    779M    31%    /var
```

For me there isnt GENERIC in /boot.
I used freebsd-update. 

How do I remove the old kernel ?


----------



## Vib3 (Apr 27, 2009)

ncdu helped nicely to getter better image of files in folders.
It said that my /rescue folder is 440.5MB.
/rescue folder is full on files:


```
.
.
-r-xr-xr-x  120 root  wheel  3793072 Apr 19 13:13 tunefs
-r-xr-xr-x  120 root  wheel  3793072 Apr 19 13:13 umount
-r-xr-xr-x  120 root  wheel  3793072 Apr 19 13:13 unlink
-r-xr-xr-x  120 root  wheel  3793072 Apr 19 13:13 vi
-r-xr-xr-x  120 root  wheel  3793072 Apr 19 13:13 whoami
-r-xr-xr-x  120 root  wheel  3793072 Apr 19 13:13 zcat
```

Should they be rather links and whats the best way to make them links ?


----------



## DutchDaemon (Apr 27, 2009)

And where would you like to link to? To the executables that are on your damaged file system, which is why you're using the executables in /rescue?

rescue(8)


----------



## phoenix (Apr 27, 2009)

Vib3 said:
			
		

> ncdu helped nicely to getter better image of files in folders.
> It said that my /rescue folder is 440.5MB.
> /rescue folder is full on files:
> 
> Should they be rather links and whats the best way to make them links ?



If you add *-i* to the ls command, you'll notice that they all have the same inode number -- they're all hardlinked to one single file.  Everything under /rescue is actually one massive crunchgen binary that does different things depending on the name that it is called by.  IOW, everything is normal here.

/rescue has all the tools you should need to rescue a system that won't boot, and/or that doesn't have /usr/bin available, or even one that doesn't have /bin available.


----------



## thetrojan01 (Apr 27, 2009)

how to remove old kernel: rm -rf /boot/oldkernel

be sure you aren't going to boot that next time!


----------



## anomie (Apr 27, 2009)

Vib3 said:
			
		

> ```
> %find -x / -size +10000 -exec du -h {} \][/quote]
> 
> Something is very wrong here -- you have only two files that are >5MB, but you're using 481MB on the / filesystem. Have you tried that fsck from single-user mode yet?
> ```


----------



## Vib3 (Apr 28, 2009)

phoenix said:
			
		

> Also, you may want to boot into single-user mode, and check the size of / without any other filesystems mounted.  Run du from there.  It may be that there's something in /var (the mountpoint directory not the mounted filesystem) or /usr.



Yes, it was the /var/db directory that was over 350M. Directory removed. Case closed -> Solved.

Thanks for everyone!


----------



## anomie (Apr 28, 2009)

I am very glad you got it solved, but that fix makes little sense. Based on your first post, /var has its own filesystem. Why was /var/db infringing on /'s space in your case?

---

edit: I think I see what you mean. In multiuser mode, you had /var mounted on top of a /var that already contained lots of files. Right?


----------



## tangram (Apr 28, 2009)

And if you deleted /var/db/ contents there goes /var/db/pkg also along with installed ports information...


----------



## DutchDaemon (Apr 28, 2009)

I'm sure the actual and useful /var/db/pkg was on the mounted /var, not the /var directory it was mounted on.


----------



## Vib3 (Apr 29, 2009)

DutchDaemon said:
			
		

> I'm sure the actual and useful /var/db/pkg was on the mounted /var, not the /var directory it was mounted on.



Yes.
Package information was on /var partition not on / partition,
so its ok


----------



## dvbme (Feb 6, 2010)

I searched high and low, found this thread that solved my problem.


----------



## sixtydoses (Feb 6, 2010)

phoenix said:
			
		

> Also, you may want to boot into single-user mode, and check the size of / without any other filesystems mounted.  Run du from there.  It may be that there's something in /var (the mountpoint directory not the mounted filesystem) or /usr.





			
				DutchDaemon said:
			
		

> I'm sure the actual and useful /var/db/pkg was on the mounted /var, not the /var directory it was mounted on.



I'm confused. Is it like having 2 /var's (one is mountpoint dir and another is mountpoint fs)? But that's not possible.


----------



## DutchDaemon (Feb 6, 2010)

Sure it is. In fact, it happens all the time. Do you have a separate /var partition on your system? Then it's mounted on a directory in the root partition called /var, which gets overlayed (overlaid?) by the mounted /var. Put some files on that underlying /var, and your disk statistics look all weird. Same goes for /tmp, /usr, and other typical mountpoints. If you want to check this, boot into single-user mode and check what's in /usr, /var and /tmp. Won't be too much .. but they do exist as legitimate directories.


----------



## sixtydoses (Feb 6, 2010)

Oh okay. I've always treated it as one and only one entity (directory /var mounted as /var file system). As in whatever that gets dumped in /var, will never affect the root partition. Thanks for the tip.


----------



## Beastie (Feb 6, 2010)

sixtydoses said:
			
		

> directory /var mounted as /var file system


If you have an independent /var partition, the device behind it - e.g. /dev/ad0s1e - is mounted on a directory (the mount-point) called /var, created* under the root partition /.
If you have no independent partition, files will be stored in the /var directory, under the root partition.

Check /etc/fstab and everything will become clear.


* actually, it's not "created", but simply extracted from the *base* distribution during the setup.


----------



## sixtydoses (Feb 6, 2010)

Beastie said:
			
		

> e.g. /dev/ad0s1e - is mounted on a directory (the mount-point) called /var, created* under the root partition /.



Yea. That's what causes the confusion. A dedicated device /dev/ad0s1e for /var shouldn't interfere /dev/ad0s1a for /.


----------



## Beastie (Feb 6, 2010)

With a separate /var partition, the /var directory (i.e. the mount-point) will be the only thing stored on the root partition (e.g. /dev/ad0s1a).
However, if you unmount the partition and copy/move files to /var, they will be stored on the root partition, obviously.


----------



## sixtydoses (Feb 6, 2010)

Ditto.


----------



## DutchDaemon (Feb 7, 2010)

sixtydoses said:
			
		

> Yea. That's what causes the confusion. A dedicated device /dev/ad0s1e for /var shouldn't interfere /dev/ad0s1a for /.



It's actually not uncommon to have a /var/log partition on a /var partition on a / partition ... so long as you mount them in the right order, it will work fine.


----------



## vincepoy (Feb 7, 2010)

SirDice said:
			
		

> My /boot/kernel only contains about 30MB, I'm wondering why yours has 125MB?



The GENERIC kernel by default compiles debugging symbols which will be big.

in the GENERIC kernel config file, there is the line:

```
makeoptions     DEBUG=-g
```

That increases the /boot/kernel by 67MBytes and if you backup to /boot/kernel.old, it's even bigger.

the way around it is when you build your kernel, put in "-DINSTALL_NODEBUG" after make so it will not install the symbol files, and you don't need to comment out the line in the kernel config.


```
make buildkernel KERNCONF=GENERIC -DINSTALL_NODEBUG
make installkernel KERNCONF=GENERIC -DINSTALL_NODEBUG
```


----------



## vincepoy (Feb 7, 2010)

thetrojan01 said:
			
		

> I solved the problem for me! I had made a new kernel and didn't remove the old one GENERIC in /boot, so, the / it's 108% full because also the new kernel has 'root' as its owner.
> 
> It's now 64% full.
> 
> ...



If you made a new kernel, it gets built in /usr/obj/usr/src/sys/GENERIC 

When you do the make installkernel

it will move /boot/kernel to /boot/kernel.old

and then install the new kernel in /boot/kernel

the reason when you installed the new kernel is due to the kernel debug symbol files which are 67MB in size as that's enabled by default in the GENERIC config file, this is mentioned in the /usr/src/UPDATING file, 20060118 entry.

As for root, what happens is when you do a upgrade atleast when I download the base install and even running install.sh to install, it first killed my /root symlink to /home/root so that /root is now using the / and then /var which was a symlink to /usr/var for me is now a directory so for the former, all I had to do was:


```
rm -rf /root ; ln -sf /home/root /root
```

and the /var problem can only be fixed in single-user mode since /var has devfs running on it for the named stuff so basically did the same thing for /var in single user mode.


----------



## vincepoy (Feb 7, 2010)

Vib3 said:
			
		

> For me there isnt GENERIC in /boot.
> I used freebsd-update.
> 
> How do I remove the old kernel ?



Now that I think of it, the base package does not include the kernel as that's part of the kernel set which does create a /boot/kernel/GENERIC, I think what you really want to do if it created the GENERIC is:


```
mv /boot/kernel/GENERIC /boot/kernel/kernel
```

It probably named the kernel as GENERIC so that it won't overwrite the existing kernel file.

If you build the kernel yourself with


```
make installkernel KERNCONF=GENERIC
```

The old kernel and it's modules will already move the contents of /boot/kernel to /boot/kernel.old and the new kernel goes into /boot/kernel.  If you wanted to really delete the old kernel, assuming that the new kernel works, then just:


```
rm -rf /boot/kernel.old
```


----------



## lsi (Jul 28, 2018)

Just to add some more detail - this can also happen if there are files in /mnt: 


```
# df -h
Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/ipsd0s1a    496M    481M    -25M   105%    /
[...]
# mount /dev/da0p1 /mnt/usbkey
# ls -l /mnt/usbkey
total 4
drwxrwxr-x  2 root  operator  512 Jul 26 21:04 .snap
# umount /mnt/usbkey
# ls -l /mnt/usbkey
total 2
drwxr-xr-x  7 root  wheel  512 Jul 25 02:09 data
# umount /mnt/usbkey
umount: /mnt/usbkey: not a file system root directory
# rm -rf /mnt/usbkey/data
# ls -l /mnt/usbkey
total 0
# df -h
Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/ipsd0s1a    496M    351M    105M    77%    /
[...]
```

In my case, this happened because files were copied to a mountpoint when the intended device was not mounted. In turn, this happened because the mount command was called from a backup script run from a cronjob. By default, /sbin is NOT in the path used by cron, so unless the path is modified, the full path to /sbin/mount must be provided in the script. If this isn't done, the call to mount fails - and the backup then backs up to the mountpoint directory, rather than the mounted device. If the mountpoint directory is under /mnt, it's on the root filesystem. So, in this case a failed mount results in the backup script filling the root filesystem. Slow claps for the admin involved. *cough* 

Thanks to this thread, fixed with 0 downtime.


----------



## SirDice (Jul 30, 2018)

lsi said:


> Just to add some more detail


This thread is more than 8 years old.


----------



## ShelLuser (Jul 30, 2018)

lsi said:


> Thanks to this thread, fixed with 0 downtime.


Other than what SirDice said: look into ZFS sometime, that can totally nullify these kinds of problems.


----------

