# Kernel Panic with ZFS



## cliffboss (Nov 20, 2011)

Hello,

I am currently experiencing some really nasty stability problems today. It all started with a power outage this morning. When the power came back I booted up the server and ran *zpool scrub* on the array. Later in the day I check the status on the scrub and could not ssh into the box. I power cycle the machine and ssh back in to the machine. About 10 minute later ssh disconnects and the machine stops responding. 

Next I plug my monitor into the box and watch what was going on. Like before around the 10 minute mark I see a kernel panic happen. The message states that it could not write the crash dump to the disk :\

I then powered cycle the machine once it came back up I then rebooted it. The reboot takes a very long time waiting to process to shutdown. It gets to the syncing of the nodes, spiting out a bunch of 4s, 2s and 1s and then says time out. Next the screen says that it is emptying the buffer and a bunch of numbers are written to the screen. After about 2-5min of this I get another kernel panic.

Tried the above again and the same result. :\

Any help trying to diagnosis the issue?

Notes: 

Up to today my systems has been stable with no crashes what so ever.
I am running with on 2GB of ram (which i know is sub optimal, but the machine is really only used to store and watch tv shows)
 I also read that upgrading to 8.2 stable fixes a lot of issues with ZFS, not sure if this will fix my problem

Thanks,
Cliffboss


----------



## gkontos (Nov 20, 2011)

Hi,

Please describe a bit further your setup. 
Is it a root on ZFS installation and how many disks are you using ?

[CMD=""]# zpool status[/CMD]

Also, have a look at /var/log/messages. A power failure can cause problems to disks. Older versions of ZFS behave strange when they can not read or write from a disk.


----------



## cliffboss (Nov 20, 2011)

Current setup:

Intel Core 2 Duo E6600
2GB of ram
4x 2TB Western Digital Green drive
1x 640GB Western Digital drive

 ZFS is not on the root
 The drives are plugged into the on board stat controller
 The 4 Green drives are in a RAIDZ setup


```
FileServer# zpool status
  pool: storage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad6     ONLINE       0     0     0
            ad8     ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            ad12    ONLINE       0     0     0

errors: No known data errors
```

See the attachment for the last 24hours of the messages log


----------



## tingo (Nov 20, 2011)

Hint: it could be a hardware problem. Run memtest86+ for several hours or overnight.


----------



## Epikurean (Nov 20, 2011)

I had the exact same problem.
What I experienced was, that the kernel panics where caused by extensive swapping etc.
I bootet into single user mode, forced an unmount of the zpool and mounted it again. So far, the problem didn't return.


----------



## cliffboss (Nov 20, 2011)

Epikurean said:
			
		

> I had the exact same problem.
> What I experienced was, that the kernel panics where caused by extensive swapping etc.
> I bootet into single user mode, forced an unmount of the zpool and mounted it again. So far, the problem didn't return.



This is done by exporting and importing the zpool right?


----------



## phoenix (Nov 20, 2011)

You can try booting off a 9.0 install CD or an mfsBSD CD (they support ZFSv28), and doing a transaction rollback import:
`# zpool import -F <poolname>`
Then do an export, and a normal import, and another export.  Then boot into the normal OS and try to import the pool.


----------



## gkontos (Nov 20, 2011)

@cliffboss,

Your attachment didn't go through for some reason.

I would first try what phoenix suggested. Upgrading to 8.2-STABLE and ZFS V28 might also be a good step.


----------



## cliffboss (Nov 21, 2011)

phoenix said:
			
		

> You can try booting off a 9.0 install CD or an mfsBSD CD (they support ZFSv28), and doing a transaction rollback import:
> `# zpool import -F <poolname>`
> Then do an export, and a normal import, and another export.  Then boot into the normal OS and try to import the pool.



I tried this, but it failed horribly.

The *zpool import -f* failed complaining that the GPT table has been corrupted for each of the disks 

Looks like I am going to rebuild the pool from a back up.


----------



## Terry_Kennedy (Nov 21, 2011)

cliffboss said:
			
		

> Like before around the 10 minute mark I see a kernel panic happen. The message states that it could not write the crash dump to the disk :\


Crash dumps have not worked reliably (or at all, actually) for me in years. The last time they worked for me was in non-SMP FreeBSD 5.x.

There are actually a large number of things which work to prevent crash dumps from operating reliably. There's a discussion of one of them (and some tangents) on the freebsd-current mailing list.

I've found that the best way to find out what caused the panic is to have some way of recording the console output (either a serial console logged somewhere, or certain types of KVM / remote console support). The info the developers need has likely scrolled off your screen by the time you get the message about not being able to access the crash dump.



> I then powered cycle the machine once it came back up I then rebooted it. The reboot takes a very long time waiting to process to shutdown. It gets to the syncing of the nodes, spiting out a bunch of 4s, 2s and 1s and then says time out.


That's not good. Those are dirty buffers which need to be written back to disk. If the number doesn't decrease, it can indicate a hardware or driver problem (writes never complete) or some sort of buffer chain corruption (which could also be hardware or software) where it has lost track of what exactly it is doing.

Normally I don't immediately say "go run memtest86+", but in this case I think it might be a good idea, just to rule out any memory issues.

If you are using any type of "smart" disk controller (such as a RAID card, even if you aren't using it in RAID mode), you should check to see if there are any firmware updates for the card.



> I am running with on 2GB of ram (which i know is sub optimal, but the machine is really only used to store and watch tv shows)


ZFS really, really wants to have a lot of RAM. The folks that originally developed it at Sun seem to have had the expectation that it would be running in a 64-bit environment with lots of RAM, which was not an unreasonable assumption given the hardware Sun was selling.



> I also read that upgrading to 8.2 stable fixes a lot of issues with ZFS, not sure if this will fix my problem


8-STABLE has ZFS 28/5, while 8.2-RELEASE has ZFS 15/4, IIRC. ZFS 28 is a much newer code base, and FreeBSD development is being done with ZFS 28. Some patches might be able to be backported to the older ZFS 15 implementation, but the best testing and coverage is likely to be under ZFS 28.



> The zpool import -f failed complaining that the GPT table has been corrupted for each of the disks
> 
> Looks like I am going to rebuild the pool from a back up.


That sounds like something ran wild and started writing in inappropriate places on the disk drives. Even once you fix the underlying problem, the pool may not be recoverable. I'd wait for an expert opinion (or go ask on the freebsd-fs mailing list) before clobbering the pool, unless you're sure you have a good backup.

I don't know of any definite hardware issue that could cause this, given the hardware you listed. There was a bug in the 3Ware twa-class firmware which caused erroneous writes to low block numbers (since fixed), and Samsung had a firmware bug on some drive models which could cause data corruption.


----------



## cliffboss (Nov 21, 2011)

Just an update, last night before bed I decide to turn off ZFS by removing the line from the rc.conf file. 

So far it has been stable for a little over 8 hours. 

Bad news this seems that ZFS is the problem. 

When I get home from work I am going to re enable ZFS again and see what happens. Maybe I proper start/shutdown might fix things.


----------



## gkontos (Nov 21, 2011)

cliffboss said:
			
		

> I tried this, but it failed horribly.
> 
> The *zpool import -f* failed complaining that the GPT table has been corrupted for each of the disks
> 
> Looks like I am going to rebuild the pool from a back up.


The good thing is that you have backups.
But don't give up yet. It looks like the power failure messed up somehow your GPT tables.

[CMD=""]# gpart show[/CMD]

Will show you if there are any problems and :

[CMD=""]# gpart recover /dev/adx [/CMD] 

Might be able to fix this for you. you might want to have a look at gpart(8)() before.

If you do decide to restore from a backup. I would suggest the use of labels instead. See glabel(8)() for better reference.

George


----------



## phoenix (Nov 21, 2011)

cliffboss said:
			
		

> I tried this, but it failed horribly.
> 
> The `# zpool import -f` failed complaining that the GPT table has been corrupted for each of the disks
> 
> Looks like I am going to rebuild the pool from a back up.



Note:  *-f* and *-F* do two *very* different things.  You want to use the capital F.


----------



## cliffboss (Nov 21, 2011)

phoenix said:
			
		

> Note:  *-f* and *-F* do two *very* different things.  You want to use the capital F.



-F told me there was not such switch. I used -f thinking you had done a typo...


----------



## cliffboss (Nov 21, 2011)

gkontos said:
			
		

> The good thing is that you have backups.
> But don't give up yet. It looks like the power failure messed up somehow your GPT tables.
> 
> [CMD=""]# gpart show[/CMD]
> ...



Just did what you said above and then did a [CMD=""]zpool import storage[/CMD]
and it worked.

I checked the status of the zpool and it said it re silvered 3k.

Now my question is can I trust the data on the drive, even when the GPT were corrupted and restored?


----------



## gkontos (Nov 21, 2011)

cliffboss said:
			
		

> Just did what you said above and then did a [CMD=""]zpool import storage[/CMD]
> and it worked.
> 
> I checked the status of the zpool and it said it re silvered 3k.
> ...



Good question! I honestly can't advise on that but it doesn't appear to be like a faulty drive.
You many want to use smartmontools just to keep an eye on those drives.

Regards,
George


----------



## cliffboss (Nov 22, 2011)

Already had that program running from day 1. It reports all drives as healthy. As for one of the drive being faulty, doubt it since FreeBSD was reporting all 4 drives having corrupted GPT [Chance of all be duds highly unlikely]

What I'll probably do unless anyone else has any insight is destroy the pool, format and recreate the pool from a backup. [Only have had the systems up for a month now so they aren't that out of date]


----------



## phoenix (Nov 22, 2011)

cliffboss said:
			
		

> -F told me there was not such switch. I used -f thinking you had done a typo...



-F was added in a later version of ZFS than was shipped in FreeBSD 8.2.  It's part of the ZFSv28 that's included with FreeBSD 9.0 and 8-STABLE.  Hence why I said you have to use a 9.0 install CD (drop to shell) or an 8.2 mfsbsd CD (that includes ZFSv28).


----------



## cliffboss (Nov 22, 2011)

phoenix said:
			
		

> -F was added in a later version of ZFS than was shipped in FreeBSD 8.2.  It's part of the ZFSv28 that's included with FreeBSD 9.0 and 8-STABLE.  Hence why I said you have to use a 9.0 install CD (drop to shell) or an 8.2 mfsbsd CD (that includes ZFSv28).



I did grab a copy of 9.0 rc2


----------

