# 'usem' State in `top` (dtrace help?)



## ranvel (Feb 28, 2019)

Hello!

I'm working on a problem and I don't know how to find out more about it! 

I'm having an application that is hanging at a certain point on my production system and in `top` it is listed as _usem_ (screenshot attached). If it is helpful, the processing that is hanging is part of a multiprocessing script. In terms of system, I'm on 12.0-RELEASE-p3.

As far as I can tell, this "event on which the process waits" (from the top `man` page), _usem_ stands for "unlink semaphore", but I can't really find out anything more about the event itself and why it is happening. 

Can anyone offer any advice? I've tried a few base dtrace one-liners and dwatch commands from this page, but I haven't been successful in finding any more useful information on the issue. I'd ideally like to be able to identify the reason why the process doesn't die. 

Thanks!


----------



## cneira (Mar 3, 2019)

Have you tried running truss? Also if the process is not dying you could take a core dump using gcore and then use gdb to check what was doing at that time.


----------



## Bobi B. (Mar 3, 2019)

You can peek at where process is stuck in the kernel with procstat(1): `procstat -kk <pid>`. Also take look at other information `procstat` can provide, like a list of file descriptors and virtual memory mappings.

You can peek with truss(1), as well, but it won't output anything, until syscall returns: `truss -p <pid>`.

What does this script do?

PS: A bit brave to run ZFS with just 1 Gig of RAM; is it necessary? Can it be replaced with UFS?


----------



## Kiiski (Mar 6, 2019)

One place to find alternatives for debugging Python code:
Python Debugging Tools


----------



## ranvel (Mar 8, 2019)

Thanks so much for these responses. I don't know why this problem didn't come up in my dev environment, but I was able to fix it. It was a multiprocessing script that maybe wasn't joining the child processes to the parent processes.

Bobi B: My understanding was that as long as I kept an eye on the ARC, it wouldn't be an issue. I was running with 3 GB on the dev environment and it didn't look like I was able to take advantage of the resources. The files on this server from now on will be logs and duplicates. I'm using dedup, which I know is frowned upon, but the hash table should be mostly duplicates, right?


----------



## Bobi B. (Mar 8, 2019)

ranvel said:


> Thanks so much for these responses. I don't know why this problem didn't come up in my dev environment, but I was able to fix it. It was a multiprocessing script that maybe wasn't joining the child processes to the parent processes.


That would be easy to tell by looking at the output of `ps dax`.


ranvel said:


> Bobi B: My understanding was that as long as I kept an eye on the ARC, it wouldn't be an issue. I was running with 3 GB on the dev environment and it didn't look like I was able to take advantage of the resources. The files on this server from now on will be logs and duplicates. I'm using dedup, which I know is frowned upon, but the hash table should be mostly duplicates, right?


Oh? `dedup` is known to be quite memory-hungry. Check out this Oracle article on ZFS Deduplication. You better put logs on a separate dataset/filesystem and implement log rotation, and see if you can avoid persisting duplicate files at all, if possible. I haven't work with a data set that would benefit from dedup, therefore my experience is more or less nil. However I believe you can't really turn off dedup once enabled.


----------



## SirDice (Mar 8, 2019)

ranvel said:


> I'm using dedup, which I know is frowned upon, but the hash table should be mostly duplicates, right?


It's not frowned upon, but you need to have around 5GB of RAM per 1TB of storage in order to make this work.

As it's mostly log files, I assume these are plain text? Then you're probably better off enabling compression.


----------



## ranvel (Mar 25, 2019)

Thanks to you both - I never knew about the 'd' option for `ps`. That's super helpful. 

I have *dedup* enabled only on a single dataset of files where the only thing that is changed in around 16 GB of files is in-file metadata (as opposed to extended attributes). So if I am thinking about this right, everything would be deduplicated excepted for the block where that metadata is changed, which is very small in my testing.


----------

