# Problems with swap on ZFS ZVOL?



## nsayer (Dec 6, 2010)

I've recently set up a VPS at rootbsd. I set it up with GPT+ZFS almost the same way as my old physical machine (the difference is that I'm not using a mirror for the tank, since the underlying storage has redundancy, and the virtual disk is much smaller than the physical disk I had before): the only things on the disk are the freebsd-boot partition and the tank. Swap is provided on a zvol.

The system only has 768 MB of RAM, but this page has some tuning that suggests that it ought to work.

Under load, the system winds up using a couple dozen MB of swap, but then all of a sudden it locked up with repeated messages:


```
swap_pager: indefinite wait buffer: bufobj:
```

All I get from teh g00g on this is a thread from September of 2007 on freebsd-current.

Is there still problems swapping to a zvol? What can I do to make it more reliable?


----------



## graudeejs (Dec 6, 2010)

You forgot to mention FreeBSD version....

I didn't have problems with swapping.... Once I made bug in a program, that filled ram and swapped 500MB in a about minute  until I reset PC manually
Check this, maybe you're missing something related to swap itself:
http://wiki.bsdroot.lv/freebsd:quick_tips#create_swap_on_zfs

Or maybe it's memory amount issue or old FreeBSD release


----------



## nsayer (Dec 6, 2010)

Sorry I forgot to mention it - 8.1-RELEASE.


----------



## nsayer (Dec 6, 2010)

I checked out the page you referenced. Turning checksums off is a good idea. I'll do that. I also read elsewhere that creating the zvol with a 4k block size was a good idea. 

The example also creates a traditional swap partition. I didn't do that, and now it would be a day of work to repartition the drive. But if you're not usig that swap partition, then your results should still be valid. 

But this is a very RAM constrained (virtual) machine. I wonder if that's the difference.


----------



## danbi (Dec 7, 2010)

Some people say enabling compression on the swap volume helps as well.


----------



## nsayer (Dec 7, 2010)

danbi said:
			
		

> Some people say enabling compression on the swap volume helps as well.



That's counter-intuitive. Do you have any references that explain why?


----------



## graudeejs (Dec 7, 2010)

On one hand you need to read less data... on slow disks and fast cpu, this could be beneficial (perfect for floppy disks ).
On other hand you need to decompress data, which means, more cpu cycles, which means bigger latency.


----------



## uchman (Jun 14, 2011)

Was this issue resolved? I'm having the same problem.


----------



## nsayer (Sep 16, 2012)

No, this is still going on. I just caught my VPS stuck with this same problem.

The provider gave us all a RAM upgrade for free not too long ago. It's up to 1.3 GB of RAM, but that still isn't a lot for a ZFS equipped server.

I have a single zvol for swap, but things like building a JVM from the ports will easily wind up getting it stuck this way.


----------



## gkontos (Sep 16, 2012)

You are using ZFS on root on a very low memory system. (less significant in this case) 
You are using ZFS on a virtualized vdev (more significant in this case)

Backup and reinstall using UFS or UFS+J.

If you want to implement a ZFS on root then get a dedicated server.


----------



## nsayer (Oct 18, 2014)

So this is continuing to happen on FreeBSD 9.2. I can't `buildworld` 9.3 with the system running multi-user because with swap on a zvol it hangs, and without swap it runs out of RAM.

Is swapping on a zvol ever going to work properly?


----------



## usdmatt (Oct 18, 2014)

I'm sure there are a few posts about problems with swap on ZFS on here. If at all possible, just don't do it. Create a completely separate partition during install and just use that like any other normal install.

The machine uses swap when it's low on RAM, but all writes to ZFS are cached in RAM for 5-30 seconds before being flushed to disk as a single transaction, so you're trying to use RAM in order to write data to disk, when you're trying to use disk because you're out of RAM...

I have no doubt there are possibly things that could be improved for swap on ZFS, but in the end it just is not appropriate to use something higher level like ZFS, that requires CPU and RAM resources, for swap. The system should just have direct access to a swap partition on the disk.


----------



## Beeblebrox (Oct 18, 2014)

Swap on ZFS has been shown to be a bad idea, but exponentially so when you have memory constraints. Minimum recommended RAM is 4 GB for ZFS, and although ZFS can work with less than 4 GB, throwing swap into this is going to give you problems.


----------



## Ordoban (Oct 18, 2014)

If you disable both caches for the ZFS swap partition, it works without issue.


----------



## nsayer (Oct 18, 2014)

I did go through the flags and noticed that primarycache and secondarycache were not set to disabled. I've changed that and will try that out.

Unfortunately, I have read all of the advice not to swap onto a zvol, but at this point reconfiguring this (virtual) machine would be too painful, so getting it to work minimally acceptably would be best.


----------



## nsayer (Oct 20, 2014)

Turns out it still deadlocks... it just doesn't deadlock with the "indefinite wait buffer" message on the console.

I tried setting vfs.zfs.arc_max to 128M 128 MB (the system has 1.5G 1.5 GB of RAM). Are there any other settings I could try?


----------



## noobster (Oct 20, 2014)

It doesn't work properly in the versions you mentioned even if you have a lot of memory. It will deadlock when it starts swapping, although not necessarily immediately. This may be fixed in the long run but for now a swap partition (or swap file on UFS) is the only reliable solution as far as I know.


----------

