# Panic String: ffs_freefile: freeing free inode



## Seeker (Nov 20, 2009)

This is from my info file:

```
Dump header from device /dev/ad4s3b
  Architecture: i386
  Architecture Version: 2
  Dump Length: 161284096B (153 MB)
  Blocksize: 512
  Dumptime: Fri Nov 20 00:32:30 2009
  Hostname: blackhole.starforce.biz
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE-p4 #0: Fri Oct  2 12:21:39 UTC 2009
    root@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
  Panic String: ffs_freefile: freeing free inode
  Dump Parity: 3080968015
  Bounds: 15
  Dump Status: good
```
This is dumped core

```
#  kgdb kernel.symbols /var/crash/vmcore.15
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
dev = ad4s3f, ino = 1655978, fs = /usr
panic: ffs_freefile: freeing free inode
cpuid = 0
Uptime: 15h11m15s
Physical memory: 1011 MB
Dumping 153 MB: 138 122 106 90 74 58 42 26 10

Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from /boot/kernel/wlan_xauth.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_xauth.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /usr/local/modules/fuse.ko...done.
Loaded symbols for /usr/local/modules/fuse.ko
Reading symbols from /usr/local/lib/oss/modules/osscore.ko...done.
Loaded symbols for /usr/local/lib/oss/modules/osscore.ko
Reading symbols from /usr/local/lib/oss/modules/oss_audigyls.ko...done.
Loaded symbols for /usr/local/lib/oss/modules/oss_audigyls.ko
Reading symbols from /boot/kernel/fire_saver.ko...Reading symbols from /boot/kernel/fire_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fire_saver.ko
Reading symbols from /boot/kernel/radeon.ko...Reading symbols from /boot/kernel/radeon.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/radeon.ko
Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/drm.ko
#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc07e25f7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc07e28c9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc09d874b in ffs_freefile (ump=0xc4440c00, fs=0xc423e000, devvp=0xc4479e04, ino=7338, mode=65535) at /usr/src/sys/ufs/ffs/ffs_alloc.c:2088
#4  0xc09ee4a5 in handle_workitem_freefile (freefile=0xc44448a0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3709
#5  0xc09edb7a in process_worklist_item (mp=0xc44465a0, flags=Variable "flags" is not available.
) at /usr/src/sys/ufs/ffs/ffs_softdep.c:975
#6  0xc09ef622 in softdep_process_worklist (mp=0xc44465a0, full=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:848
#7  0xc09f1e07 in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:759
#8  0xc07bd0a9 in fork_exit (callout=0xc09f1990 <softdep_flush>, arg=0x0, frame=0xe4510d38) at /usr/src/sys/kern/kern_fork.c:810
#9  0xc0ac92e0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:264
(kgdb)
```

After this /usr was dirty and I had to run fsck against it.
There was immense amount of inode reconnect request that I had to perform and pressing _[y] + enter_ took me 5 minutes!  

I also remember that it wanted to create lost+found(I said yes) and Soft Updates have been mentioned...

What is a cause of this?


----------



## graudeejs (Nov 20, 2009)

instead of pressing y all the time you could add *-y* flag to fsck


----------



## Seeker (Nov 20, 2009)

I haven't expected so much requests to press *y*
Usually just 4-5 times and all is fixed.


----------



## graudeejs (Nov 20, 2009)

When I used UFS, sometimes after power failure (when I was a complete noob and didn't knew about -y) I was doing same practice.... there were few times I cursed on FreeBSD because it was very long process...


----------



## Seeker (Nov 20, 2009)

This err msg appeared many times

```
UNEXPECTED SOFT UPDATE INCONSISTENCY
```


----------



## Seeker (Nov 20, 2009)

This is crazy!
Now I have page faults...
Info:

```
# cat  /var/crash/info.16
Dump header from device /dev/ad4s3b
  Architecture: i386
  Architecture Version: 2
  Dump Length: 210821120B (201 MB)
  Blocksize: 512
  Dumptime: Fri Nov 20 15:21:58 2009
  Hostname: blackhole.starforce.biz
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE-p4 #0: Fri Oct  2 12:21:39 UTC 2009
    root@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
  Panic String: page fault
  Dump Parity: 308394019
  Bounds: 16
  Dump Status: good
```
Debug:

```
# kgdb kernel.symbols /var/crash/vmcore.16
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xffff0008
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc0addc87
stack pointer           = 0x28:0xe6c1ba2c
frame pointer           = 0x28:0xe6c1ba6c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 16558 (cc1plus)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 7h36m44s
Physical memory: 1011 MB
Dumping 201 MB: 186 170 154 138 122 106 90 74 58 42 26 10

Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from /boot/kernel/wlan_xauth.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_xauth.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /usr/local/modules/fuse.ko...done.
Loaded symbols for /usr/local/modules/fuse.ko
Reading symbols from /usr/local/lib/oss/modules/osscore.ko...done.
Loaded symbols for /usr/local/lib/oss/modules/osscore.ko
Reading symbols from /usr/local/lib/oss/modules/oss_audigyls.ko...done.
Loaded symbols for /usr/local/lib/oss/modules/oss_audigyls.ko
Reading symbols from /boot/kernel/fire_saver.ko...Reading symbols from /boot/kernel/fire_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fire_saver.ko
Reading symbols from /boot/kernel/radeon.ko...Reading symbols from /boot/kernel/radeon.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/radeon.ko
Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/drm.ko
#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc07e25f7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc07e28c9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc0ae3f2c in trap_fatal (frame=0xe6c1b9ec, eva=4294901768) at /usr/src/sys/i386/i386/trap.c:939
#4  0xc0ae41b0 in trap_pfault (frame=0xe6c1b9ec, usermode=0, eva=4294901768) at /usr/src/sys/i386/i386/trap.c:852
#5  0xc0ae4b5c in trap (frame=0xe6c1b9ec) at /usr/src/sys/i386/i386/trap.c:530
#6  0xc0ac926b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7  0xc0addc87 in get_pv_entry (pmap=0xc4a3fd74, try=0) at /usr/src/sys/i386/i386/pmap.c:2036
#8  0xc0ade17b in pmap_insert_entry (pmap=Variable "pmap" is not available.
) at /usr/src/sys/i386/i386/pmap.c:2203
#9  0xc0ae112e in pmap_enter (pmap=0xc4a3fd74, va=771317760, access=2 '\002', m=Variable "m" is not available.
) at /usr/src/sys/i386/i386/pmap.c:3114
#10 0xc0a199e7 in vm_fault (map=0xc4a3fce8, vaddr=771317760, fault_type=2 '\002', fault_flags=8) at /usr/src/sys/vm/vm_fault.c:891
#11 0xc0ae409b in trap_pfault (frame=0xe6c1bd38, usermode=1, eva=771317760) at /usr/src/sys/i386/i386/trap.c:829
#12 0xc0ae49e7 in trap (frame=0xe6c1bd38) at /usr/src/sys/i386/i386/trap.c:397
#13 0xc0ac926b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#14 0x0850c8ff in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
```
I've run memtest86+(booted) for 4 hours and all was clean


----------



## Seeker (Nov 20, 2009)

Also this:

```
# cat /var/crash/info.17
Dump header from device /dev/ad4s3b
  Architecture: i386
  Architecture Version: 2
  Dump Length: 137056256B (130 MB)
  Blocksize: 512
  Dumptime: Fri Nov 20 21:45:33 2009
  Hostname: blackhole.starforce.biz
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE-p4 #0: Fri Oct  2 12:21:39 UTC 2009
    root@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
  Panic String: page fault
  Dump Parity: 4186440487
  Bounds: 17
  Dump Status: good
```
Debug:

```
# kgdb kernel.symbols /var/crash/vmcore.17
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x27
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc08574b6
stack pointer           = 0x28:0xe6c29890
frame pointer           = 0x28:0xe6c298b8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 7234 (csup)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 6h22m19s
Physical memory: 1011 MB
Dumping 130 MB: 115 99 83 67 51 35 19 3

Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from /boot/kernel/wlan_xauth.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_xauth.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /usr/local/modules/fuse.ko...done.
Loaded symbols for /usr/local/modules/fuse.ko
Reading symbols from /usr/local/lib/oss/modules/osscore.ko...done.
Loaded symbols for /usr/local/lib/oss/modules/osscore.ko
Reading symbols from /usr/local/lib/oss/modules/oss_audigyls.ko...done.
Loaded symbols for /usr/local/lib/oss/modules/oss_audigyls.ko
Reading symbols from /boot/kernel/fire_saver.ko...Reading symbols from /boot/kernel/fire_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fire_saver.ko
Reading symbols from /boot/kernel/radeon.ko...Reading symbols from /boot/kernel/radeon.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/radeon.ko
Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/drm.ko
#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc07e25f7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc07e28c9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc0ae3f2c in trap_fatal (frame=0xe6c29850, eva=39) at /usr/src/sys/i386/i386/trap.c:939
#4  0xc0ae41b0 in trap_pfault (frame=0xe6c29850, usermode=0, eva=39) at /usr/src/sys/i386/i386/trap.c:852
#5  0xc0ae4b5c in trap (frame=0xe6c29850) at /usr/src/sys/i386/i386/trap.c:530
#6  0xc0ac926b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7  0xc08574b6 in vfs_hash_get (mp=0xc44465a0, hash=71515, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/vfs_hash.c:72
#8  0xc09f8619 in ffs_vgetf (mp=0xc44465a0, ino=71515, flags=2, vpp=0xe6c299b8, ffs_flags=0) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1399
#9  0xc09f8b1e in ffs_vget (mp=0xc44465a0, ino=71515, flags=2, vpp=0xe6c299b8) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1379
#10 0xc0a0506b in ufs_lookup (ap=0xe6c29a00) at /usr/src/sys/ufs/ufs/ufs_lookup.c:600
#11 0xc0af8822 in VOP_CACHEDLOOKUP_APV (vop=0xc0c6c200, a=0xe6c29a00) at vnode_if.c:153
#12 0xc0851e7c in vfs_cache_lookup (ap=0xe6c29a84) at vnode_if.h:83
#13 0xc0afa4f6 in VOP_LOOKUP_APV (vop=0xc0c6c720, a=0xe6c29a84) at vnode_if.c:99
#14 0xc0858821 in lookup (ndp=0xe6c29b48) at vnode_if.h:57
#15 0xc0859569 in namei (ndp=0xe6c29b48) at /usr/src/sys/kern/vfs_lookup.c:215
#16 0xc086705f in kern_lstat (td=0xc50f6000, path=0x28449080 <Address 0x28449080 out of bounds>, pathseg=UIO_USERSPACE, sbp=0xe6c29c18)
    at /usr/src/sys/kern/vfs_syscalls.c:2184
#17 0xc086721f in lstat (td=0xc50f6000, uap=0xe6c29cfc) at /usr/src/sys/kern/vfs_syscalls.c:2167
#18 0xc0ae4505 in syscall (frame=0xe6c29d38) at /usr/src/sys/i386/i386/trap.c:1090
#19 0xc0ac92d0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
```

Maybe, there is an issue with CPU, or it's L1, or L2, or Lx cache??
Which tool should I use? :stud


----------

