# iNotify for FreeBSD?



## IT_Architect (Mar 4, 2013)

Is there an iNotify for FreeBSD?  Kqueue has to open every directory it watches, so too many directories will run it out of the max number of open files (typically 1024).  That would never work on a web server.

Thanks!


----------



## fluca1978 (Mar 4, 2013)

I don't know, it seems there is none.
Anyway I believe this is the wrong forum, it is better placed in the _(kernel) development_ area.
Moreover I cannot imagine why a single process should monitor more than 1024 directories, and since it has to monitor directories, probably splitting the process into hierarchical processes will work better.


----------



## IT_Architect (Mar 4, 2013)

fluca1978 said:
			
		

> I cannot imagine why a single process should monitor more than 1024 directories


Real-time anti-malware solutions all require it.  It's the limitation they run into when attempting to implement kqueue.  Not having the functionality of inotify is a show-stopper.  I was hoping there was a way around this limitation in the newer FreeBSDs.

Thanks!


----------



## Carpetsmoker (Mar 4, 2013)

> will run it out of the max number of open files (typically 1024)



Where do you get the 1024 limit?


```
[~]% sysctl -d kern.maxfiles
kern.maxfiles: Maximum number of files
[~]% sysctl -d kern.maxfilesperproc
kern.maxfilesperproc: Maximum files allowed open per process

[~]% sysctl kern.maxfiles
kern.maxfiles: 12328
[~]% sysctl kern.maxfilesperproc
kern.maxfilesperproc: 11095
```

You can increase these further, but these are the defaults on my FreeBSD 8.2 and 9.0 systems...


----------



## wblock@ (Mar 4, 2013)

The mailing lists (maybe freebsd-current) are probably a better source for this type of information.


----------



## SirDice (Mar 4, 2013)

IT_Architect said:
			
		

> Is there an iNotify for FreeBSD?


No, iNotify is a Linux kernel feature.


----------



## kpa (Mar 4, 2013)

Real-time anti-malware for FreeBSD? I take it's for checking Windows malware on a fileserver running FreeBSD


----------



## IT_Architect (Mar 4, 2013)

Carpetsmoker said:
			
		

> Where do you get the 1024 limit?...You can increase these further, but these are the defaults on my FreeBSD 8.2 and 9.0 systems...



I was reading here.  http://git-annex.branchable.com/design/assistant/inotify/
and here: http://en.wikipedia.org/wiki/Gamin


> "Under Linux, Gamin uses inotify or dnotify to monitor filesystem activity. Under FreeBSD, Gamin uses kqueue/kevent as the native FreeBSD kernel event notification mechanism, but in this case kqueue has one major drawback as compared to inotify: inotify is the filename-based monitoring facility and kqueue uses file descriptors for identification of monitored files, so, under FreeBSD, gam_server has to open each file in the monitored directory. This can be an issue when monitoring directories with many files stored, because system could easily reach its kernel limits on maximum file descriptor count (kern.maxfiles and kern.maxfilesperproc)."





			
				SirDice said:
			
		

> No, iNotify is a Linux kernel feature.





			
				wblock@ said:
			
		

> The mailing lists (maybe freebsd-current) are probably a better source for this type of information.



Perhaps so.  I never heard if inotify for FreeBSD, but while searching I found this http://www.freebsd.org/cgi/man.cgi?query=inotify&apropos=0&sektion=0&manpath=SuSE+Linux/i386+11.3&format=html a post on this forum. http://forums.freebsd.org/showthread.php?t=29988
and this: https://github.com/skirge/fsnotifier-freebsd/blob/master/inotify.c which is based on kqueue.  I was hoping.



			
				kpa said:
			
		

> Real-time anti-malware for FreeBSD? I take it's for checking Windows malware on a fileserver running FreeBSD



No, the problem is bots guessing passwords and placing exploits on their pages.  You can have users come up with passwords that meet all of the complexity requirements but the bot finds them because they use things that are easy for them to remember.  When their site gets blacklisted, and it's my fault.  Firewall policies are problematic because bots know how to stay under the radar.  If you set the policy below the robot, it also blocks customers.

There is a program designed to catch web exploits. Linux Malware Detector (LMD).  I deployed that.  The problem is, I put a known exploit out there that I saved from a previous incident, and it didn't find it, while ClamAV, which is not designed for that type of exploit, did find it.  So I'm just trying to come up with ways to catch things before they get blacklisted by Google and Yahoo!/Bing.  ClamAV works fine for a daily scan, and even hourly with parameters that check only the files that have changed in the past hour.  However, the right way is to scan files the moment they are created or change.  IMHO, this has become a necessity in this day and age.

Thanks all!


----------



## decke (Apr 26, 2013)

iNotify would also be useful for all the cloud synchronization services that start to become popular. They all have the same problem that they need to watch folders that get automatically synchronized to the cloud when they change or files are added.


----------



## ta0kira (Apr 26, 2013)

Have you considered enabling audit(4) and using auditpipe(4)?

Kevin Barry


----------



## SirDice (Apr 26, 2013)

IT_Architect said:
			
		

> There is a program designed to catch web exploits. Linux Malware Detector (LMD).  I deployed that.  The problem is, I put a known exploit out there that I saved from a previous incident, and it didn't find it, while ClamAV, which is not designed for that type of exploit, did find it.


Unfortunately all those types of programs all work signature based. This means it will never catch everything. If I write a brand new exploit (or something specifically targeted at your site) these programs will never catch it. Just keep a sharp eye on your logs, analyse them daily. Filter out the obvious crap and see what's left. If you do that on a regular basis you will quickly see things that are out of place, investigate those.



> So I'm just trying to come up with ways to catch things before they get blacklisted by Google and Yahoo!/Bing.


Make sure you set up an abuse email address or some other way people can use to complain. Read them, investigate and act on them as quickly as possible. You usually only get put on banlists when there's no way to comlain or you simply don't act on any abuse complaints.


----------



## decke (Apr 28, 2013)

From a user perspective having to enable audit seems wrong. The audit implementation could probably be a good example on how to get the information in the kernel but it needs a proper API for the userland.


----------



## ta0kira (Apr 29, 2013)

decke said:
			
		

> From a user perspective having to enable audit seems wrong. The audit implementation could probably be a good example on how to get the information in the kernel but it needs a proper API for the userland.


I suppose it would also pose a problem with jailed processes and other non-privileged processes, but I made the suggestion given @IT_Architect's goal of server security. I certainly wouldn't use audit as a substitute for inotify myself because every occasion I've had to monitor files or directories in real time only involved a handful of each.

Kevin Barry


----------



## priyadarshan (Nov 20, 2014)

`fswatch`, albeit sporting for now a GNU license, could perhaps help? https://github.com/emcrisostomo/fswatch

But, regarding FreeBSD, the README says:

_The kqueue monitor, available on any *BSD system featuring kqueue, requires a file descriptor to be opened for every file being watched. As a result, this monitor scales badly with the number of files being observed and may begin to misbehave as soon as the fswatch process runs out of file descriptors. In this case, fswatch dumps one error on standard error for every file that cannot be opened._

(PS: I also posted this on similar Thread 42792)


----------



## b7j0c (Nov 22, 2014)

I personally would like to see an iNotify API. I know its a Linux thing but it is very convenient.

In many cases I just use Go for these types of problems, it provides an fsnotify interface across platforms that is "good enough".


----------



## sweeney (Dec 21, 2014)

b7j0c said:


> I personally would like to see an iNotify API. I know its a Linux thing but it is very convenient.
> 
> In many cases I just use Go for these types of problems, it provides an fsnotify interface across platforms that is "good enough".



There's one in the ports tree.  From the pkg-descr:

This library provides inotify-compatible interface for applications,
that need to monitor changes happening in a filesystem.  It can be useful
when porting Linux applications, which often use inotify interface.

The IN_OPEN, IN_CLOSE_WRITE and IN_CLOSE_NOWRITE events are not yet
implemented, so the relevant tests are known to fail.

Author: Dmitry Matveev <me@dmitrymatveev.co.uk>
WWW: https://github.com/dmatveev/libinotify-kqueue​If you install from `pkg` it even gives you a nice little warning that you may want to bump the kern.maxfiles value
​


----------



## tanked (Dec 21, 2014)

Could Dtrace help here? I'm sure there would be a way to get it to alert on file access, though I don't know how you would integrate that with other applications.


----------



## sweeney (Dec 22, 2014)

sweeney said:


> There's one in the ports tree.



For completeness, the port is devel/libinotify.


----------



## mrtonyg (Dec 28, 2014)

Much like the OP I have been searching for program that will monitor a file for changes.

After much searching I found a simple and excellent solution in this page: http://doc.geoffgarside.co.uk/kqueue/
Scroll to the bottom of the page and select: EVFILT_VNODE
There you will find the c source code for the excellent utility. Compile and enjoy!

By the way, I did try `fswatch`: https://github.com/emcrisostomo/fswatch. But could only get it to compile in FreeBSD 10 and I needed something that worked in v9. The issue with `fswatch` is that is written in c++ and the c++ compiler in FreeBSD 9.x is not working properly.

Just for clarification, FreeBSD has an `fswatch` in the ports but it is not related to the `fswatch` I just mentioned. The `fswatch` in the ports is a hash generator for files...security/fswatch


----------



## PacketMan (Dec 28, 2014)

decke said:


> iNotify would also be useful for all the cloud synchronization services that start to become popular. They all have the same problem that they need to watch folders that get automatically synchronized to the cloud when they change or files are added.



Agreed. Some sort of 'Notify' is getting more and more desirable/needed. Cloud sync is taking off like crazy.


----------



## tingo (Jan 2, 2015)

Perhaps devel/fam could be used?


----------



## wblock@ (Jan 2, 2015)

devel/gamin is another, maybe newer thing like fam.


----------



## Carpetsmoker (Jan 3, 2015)

But those all are all just frontends for kqueue, so don't solve the problems of having to open every directory it's watching.


----------



## priyadarshan (Jul 21, 2015)

Two more to consider:

sysutils/filewatcherd home page: https://github.com/faelys/filewatcherd/
sysutils/watchman home page: https://facebook.github.io


----------



## Andrew Schmidt (Apr 4, 2016)

priyadarshan said:


> Two more to consider:
> 
> sysutils/filewatcherd home page: https://github.com/faelys/filewatcherd/
> sysutils/watchman home page: https://facebook.github.io



I am no expert.

filewatcherd doesn't seem to watch the fs, doesn't seem to be kernel based, or like watchman, uses kqueue.

I haven't used FreeBSD since 4.4 and I just installed 10.2 and upgraded to 10.3- it's been a while.  Please forgive me if I'm off  base.


----------



## zirias@ (Apr 4, 2016)

Andrew Schmidt said:


> filewatcherd doesn't seem to watch the fs, doesn't seem to be kernel based, or like watchman, uses kqueue.
> [...] Please forgive me if I'm off  base.



No, you aren't; seeing this thread, I wanted to comment roughly the same about devel/libinotify, which Carpetsmoker already did briefly:

All these tools/libraries have to use what the OS gives them, which is either just polling (bad) or kqueue (nice) on FreeBSD. In the case of devel/libinotify, the API exposed is that of Linux' inotify, which is great for quick porting of Linux software -> it will use kqueue without actually changing a lot of code. But in the corner cases where the problem is actually limitations of kqueue, of course none of all these suggestions will do any good.


----------



## Andrew Schmidt (Apr 6, 2016)

So all is lost?  Should I switch my server back to Linux?  I'm not asking to be "snarky" or whatever, but what are the chances of a new API being developed that is useful as a FS monitor?

I would MUCH rather stick with FreeBSD.  MUCHO


----------



## zirias@ (Apr 6, 2016)

Andrew Schmidt said:


> So all is lost?  Should I switch my server back to Linux?  I'm not asking to be "snarky" or whatever, but what are the chances of a new API being developed that is useful as a FS monitor?


Well, I didn't look into kqueue so far, but used inotify in an own project ... that's because I am new to FreeBSD. So I can only write on this topic what I read about it.

From what I understand, kqueue IS useful as a FS monitor and this is the purpose of this API. The only drawback compared to Linux' inotify is the need for multiple open file descriptors while inotify works on one single file descriptor that is configured to report you events for whatever you are interested in. This drawback is only relevant when monitoring a huge number of files/directories simultaneously. This thread has already seen the hint that the kernel allows to increase limits here and that in the default settings, you already have a high number -- so are you sure you actually need something that's not possible with kqueue?


----------



## Andrew Schmidt (Apr 6, 2016)

Zirias said:


> Well, I didn't look into kqueue so far, but used inotify in an own project ... that's because I am new to FreeBSD. So I can only write on this topic what I read about it.
> 
> From what I understand, kqueue IS useful as a FS monitor and this is the purpose of this API. The only drawback compared to Linux' inotify is the need for multiple open file descriptors while inotify works on one single file descriptor that is configured to report you events for whatever you are interested in. This drawback is only relevant when monitoring a huge number of files/directories simultaneously. This thread has already seen the hint that the kernel allows to increase limits here and that in the default settings, you already have a high number -- so are you sure you actually need something that's not possible with kqueue?



I want to monitor my plex library.  I'll probably have to switch back to Linux. A viable FS monitor kqueue is not.  It may be good for watching a folder or a maybe a few, but an entire ZFS volume?

This is a basic function of any modern operating system, just not FreeBSD?


----------



## Carpetsmoker (Apr 6, 2016)

Andrew Schmidt said:


> I want to monitor my plex library.  I'll probably have to switch back to Linux. A viable FS monitor kqueue is not.  It may be good for watching a folder or a maybe a few, but an entire ZFS volume?
> 
> This is a basic function of any modern operating system, just not FreeBSD?



Again, it *does* work for FreeBSD, it just uses up file descriptors. The default limit is about 12k, which is rather low − conservative upper limits are not necessarily a bad thing, by the way.
The current default for fs.inotify.max_user_watches is 524k. I'm not sure what a feasible upper limit is for the maximum number of open file descriptors, but you should be able to set it to something in the order of hundreds of thousands, if not millions.

Does inotify scale *better* here? Sure. But I don't think that kqueue is as dysfunctional as you make it out to be ...


----------



## tobik@ (Apr 6, 2016)

Carpetsmoker said:


> The default limit is about 12k, which is rather low − conservative upper limits are not necessarily a bad thing, by the way.


The default limit for kern.maxfiles is much higher normally since it's scaled based on how much memory you have (same for kern.maxfilesperproc): http://fxr.watson.org/fxr/source/kern/subr_param.c#L260

For example here on my ThinkPad with 4 GB kern.maxfiles is 124965 and on a server with 20 GB it's 651785.

Andrew, obviously you seem to have hit some kind of a practical problem/limitation with kqueue. Can you tell us more?


----------



## Andrew Schmidt (Apr 8, 2016)

tobik said:


> The default limit for kern.maxfiles is much higher normally since it's scaled based on how much memory you have (same for kern.maxfilesperproc): http://fxr.watson.org/fxr/source/kern/subr_param.c#L260
> 
> For example here on my ThinkPad with 4 GB kern.maxfiles is 124965 and on a server with 20 GB it's 651785.
> 
> Andrew, obviously you seem to have hit some kind of a practical problem/limitation with kqueue. Can you tell us more?



I can't tell you more, I don't have the knowledge, my limitation.  Plex devs seem to think kqueue just won't work and can't/won't implement support for it.

I keep seeing issues with descriptor limits, possibly differences in media locations, and the fact that folders being monitored must be open.  I'm not sure what that last part means, I don't use the GUI, so everything seems "open" to me. 

I, myself, can't find much info about kqueue except for monitoring sockets.  I haven't (yet) tried experimenting with it, due to the fact I'm a single father with a full-time job...  Haven't had much time.


----------



## sremick (Apr 12, 2016)

For what it's worth: Andrew isn't alone here, and there are many other FreeBSD users (either directly, or by virtue of using Plex on FreeNAS which is an extremely-popular option) who are foaming at the bit for this. Unfortunately there is a strong sense among the Plex devs (who certainly know a lot more about inotify and the inner-workings and needs of Plex than I) that kqueue just is not a workable option for this need. From reading the historical posts in this thread, it seems Plex is not unique in this either. It would be really great if instead of end-users being caught in the middle, frustrated at the loss of functionality and trying to relate info back and forth, there could be a direct dialog between FreeBSD devs and Plex devs ("gbooker02" would be a prime contact) to sort this all out. There must be a workable solution by one team or another, but right now it really seems we're in a stalemate due to miscommunications/misunderstandings between the two projects about what the options are and what can and cannot be done with them.

Plex on FreeBSD already gets second-fiddle to Linux due to lack of Gracenote support. I don't hold any hope of an answer there in that 3-way problem. But the kqueue/inotify issue is strictly between Plex and FreeBSD so it seems that surely there could be an answer if we could only get the proper people talking to each other...


----------



## Adrian Williamson (Apr 12, 2016)

Whilst I can't pretend to know anything about inotify or kqueue, I am one of those 'Plex on FreeNAS' users who would love it if Plex could automatically detect changes to my media dataset and update it's library accordingly.


----------



## gbooker (Apr 13, 2016)

I created an account over here to hopefully help with the dialog.

From my assessment, using kqueue to monitor for file changes requires opening the fd for every file and directory contained within the directory tree.  I've contacted a few Plex users to ask the size of these library as well as the amount of RAM they have to better assess the feasibility.  One example I obtained the user has over 500,000 files and directories with 16G of RAM.  If he were on FreeBSD (not Windows) and a kqueue-based file change monitoring was done with his library, the process would exceed the limit of the maximum number of open files.  Several others come close to the limit and this is among a sample of about 10 users I've personally asked.  There are several "build threads" in Plex's forums that tell you they would either come close or exceed the limit as well.  Add to this that Plex has recently added support for photo libraries and many of Plex's users are avid or professional photographers, it is easy to see how these directory trees can grow to be enormous in terms of file count.  Given how detrimental hitting these limits would be to the application as a whole, this would also require reading the limit as well as a scan to see if the limit is in danger of being hit before enabling the function.  Furthermore, if enabled, the count would need to be monitored to disable it if the library were to grow dramatically (such as first-run setup adding all the libraries) and be again in danger of hitting the limit.

On the practice of using kqueue:  When using kqueue, the code is not told a path for the change but rather the fd.  This means the application must keep a mapping from fd -> path.  Furthermore, the information about the change is somewhat limited, meaning that if the fd corresponds to a directory, that directory must nearly always be rescanned.  This means there must be a mapping from path -> fd to determine if a file/directory within the scanned directory is already monitored or not.  (In reality these maps would be to a common data structure rather than just path <-> fd.)  This is a large amount of accounting that must be done by the application.  So large in fact that an initial stab at implementing this monitoring for FreeBSD produced code about as large as the monitoring for MacOS, Linux, and Windows combined (as well as including the common functions used across all platforms).  Some of this can be saved by using `udata` inside `kevent` but at most that would be one map and very few lines of code.

On the philosophy of using kqueue for FS monitoring:  I see how kqueue has great purposes, but it strikes me as its intent was to monitor sockets more so that files.  Extending this use to directories/files seems to me like a bit of a hack, but I suppose it works well enough for a small set.  When it is scaled to such a large number as would be used in several users' Plex libraries, it looks far more like it is being used for a task it was never designed to handle.  Even more so when these libraries are mostly quiescent with occasional additions/subtractions/modifications of files/directories.  Holding hundreds of thousands of fds open for what is likely a 10s of changes a day seems excessive.  It strikes me that FreeBSD really needs some kernel API that's truly designed for file system monitoring over using an existing API that seems ill-suited for this scale.

On inotify:  Personally, I'd prefer FSEvents over inotify as inotify still requires opening an fd-like object for every directory.  It does give information about changes to files contained within the directory though and provides path information on the changed item.  This does reduce the amount of accounting the application must perform but there's still some for every directory.  The FSEvents monitors an entire directory tree at once and provides rich information in its callback.  This is much easier on the developer.


----------



## mnd999 (Apr 15, 2016)

This sounds like something that might need to go to freebsd-current to get the attention of those in the know.


----------



## rigoletto@ (Jan 2, 2018)

I know I am necro-posting but I am digging in the subject a bit on IRC (#freebsd) and I am leaving it here for eventual future interested people.

It seems kqueue(2) is just a event notification framework, but would be "very easy" (for who knows how to do that, not me) to implement FS monitoring system (like FSEvents) over kqueue(2).

In other words, if someone implement FSEvents/inotify equivalent he/she would just need to "connect" it to kqueue(2) to have a proper FS filesystem monitoring tool.


----------



## tobik@ (Jan 2, 2018)

But isn't this what devel/libinotify does?


----------



## rigoletto@ (Jan 2, 2018)

tobik@

Based on what I understood, I think devel/libinotify is just an API (kqueue <-> inotify) wrapper, but it would be possible ("very easy") to write a proper FS monitoring system (which would just need to use a few file descriptors  - like FSEvent/inotify) and connect it to the kqueue(2) framework to "asynchronously dispatch information to userspace".


----------



## tobik@ (Jan 2, 2018)

Ah ok.

Btw, who claims that it is "very easy" and why haven't they done it already?


----------



## rigoletto@ (Jan 2, 2018)

This is the question I do not have the answer. Probably people who does not have interest in this kind of function.

EDIT:

Btw, I do not have idea how filesystem monitoring works but it seems lang/go is quite used for this kind of objective.


----------



## Snurg (Jan 2, 2018)

Plex is not the only application that needs to watch a lot of directories/files.

Thus just my interested layman's question regarding the practicability of the kqueue approach, which was allegedly designed for watching sockets:
*How long does it take to traverse half a million files, only to open the file handles kqueue requires to work?*

Personally I subjectively feel the FSEvents approach looks much more elegant.


----------



## rigoletto@ (Jan 2, 2018)

Well,* just for the record*, I downloaded the Linux kernel (4.14.11). The inotify stuff is in fs/notify/inotify/*. There are 5 files, including the Makefile, with a total of 1061 lines (including comments).

I do not know if it actually is what it seems to be, but does not seem that much of code indeed (C code I guess). Well, there may have a lot of more code somewhere, and I also can't help in availing the complexity of that implementation.

EDIT:

Btw, everything in fs/notify folder result in a total of 20+ files and 4588 lines of code in total.


----------



## poorandunlucky (Jan 5, 2018)

Andrew Schmidt said:


> I want to monitor my plex library.  I'll probably have to switch back to Linux. A viable FS monitor kqueue is not.  It may be good for watching a folder or a maybe a few, but an entire ZFS volume?
> 
> This is a basic function of any modern operating system, just not FreeBSD?



Plex detects changes just fine on my system...  (?)

I use minidlna now, though...

Also, sorry to butt-in like this, but could someone just kind of summarize what the difference between kqueue and inotify is?

I was under the impression kqueue was fine to monitor filesystem changes... even if it has "open files" which I'm sure is just a number, and it just watches for any access at the kernel level...  I'm not even sure how else something could detect changes to a folder...  I mean, a folder and a file are the same, no?  just an inode...  no?  I'm just not sure where this is all going...


----------



## rigoletto@ (Jan 5, 2018)

poorandunlucky

See: Thread 38162/page-2#post-317478


----------



## ralphbsz (Jan 5, 2018)

The code in the inotify directory is the tip of the iceberg.  The real work happens in the file systems: those need to generate the events that inotify then channels to consumers.  Not all file systems do that.  Some file systems use other mechanisms to get the same effect.  One that's popular for large systems is to run DMAPI (which wasn't intended for getting file change notifications, but for backup and HSM), and issue DMAPI events.  Another approach is to not attempt to notify on every change, but instead implement fast scanning of the whole file system metadata; for large and networked systems it can actually be more efficient to regularly scan all the file system than to continuously monitor for all changes.

So the real number of lines of code for the whole family of file change notification is much higher, and not in a single place.


----------



## Snurg (Jan 5, 2018)

ralphbsz said:


> ... *consumers*.


I know you didn't mean it that way, but I think maybe it's true.
If I am right this immediate notifying is mainly a typical desktop/laptop, thus a "consumer" functionality.
And then this would explain, why this for the developers of a server OS has no high priority.
...


ralphbsz said:


> So the real number of lines of code for the whole family of file change notification is much higher, and not in a single place.


It is actually very complex. The whole stuff, the inode->path associations etc, that are described by gbooker above.This is to be done by kqueue, an application allegedly made for another purpose.

I guess Apple has made a quite different, maybe more sophisticated approach: when a file is opened, then there is a check whether it matches the FSEvent rules.
If and only if it matches, it becomes necessary to store and watch the inode until the file gets closed.
But, this would require deep integration into the filesystem, instead just putting a layer over it. Much work, which possibly only a small fraction of FreeBSD users would use.


----------



## rigoletto@ (Jan 5, 2018)

Snurg said:


> If I am right this immediate notifying is mainly a typical desktop/laptop, thus a "consumer" functionality.



But not just! For instance, this very useful for syncing services, specially large ones. 

EDIT:

I heard iXsystems is porting the ZFS native encryption and it should (hopefully) land in 12-RELEASE, in about a year.

This functionality (notify) seems something of their interest too, specially due to FreeNAS. However, I am not aware of any work of them about that.


----------



## ralphbsz (Jan 5, 2018)

When I said "consumer" above it, I didn't mean desktop/laptop as opposed to enterprise/server computing, but I meant the party that uses the change notification events (eats them, or consumes them).

And indeed, these type of mechanisms are used in some large enterprise systems.  Not to update a little file browser window on your laptop screen (duh).  That's why there are some industrial-strength versions of notifications in some systems.  They tend to not use inotify or kqueue, but their own mechanisms, for efficiency sake.


----------



## poorandunlucky (Jan 6, 2018)

ralphbsz said:


> [...] Some file systems use other mechanisms to get the same effect.  One that's popular for large systems is to run DMAPI (which wasn't intended for getting file change notifications, but for backup and HSM), and issue DMAPI events.  Another approach is to not attempt to notify on every change, but instead implement fast scanning of the whole file system metadata; for large and networked systems it can actually be more efficient to regularly scan all the file system than to continuously monitor for all changes. [...]



This.

For things like Plex libraries, IMO just scanning directories' metadata periodically seems quite enough...  I don't think mechanisms like kqueue or inotify were meant to monitor for such trivial changes...  

https://wiki.netbsd.org/tutorials/kqueue_tutorial/

Indeed, it wasn't meant to monitor file system changes... I concur that maybe it's a hack to do it on a small scale, but it's definitely not meant to be used to monitor a collection of files for changes, or folders for new/deleted/modified files...

Mercurial does that pretty well, AFAIK...  why not just use a script like that?


----------



## rigoletto@ (Feb 14, 2018)

Is there by any chance the object of this thread be added to SummerOfCodeIdeas page? Or is this too much work for GSoC?


----------



## priyadarshan (Jun 12, 2018)

gbooker said:


> I created an account over here to hopefully help with the dialog.
> 
> On inotify:  Personally, I'd prefer FSEvents over inotify as inotify still requires opening an fd-like object for every directory.  It does give information about changes to files contained within the directory though and provides path information on the changed item.  This does reduce the amount of accounting the application must perform but there's still some for every directory.  The FSEvents monitors an entire directory tree at once and provides rich information in its callback.  This is much easier on the developer.



Thank you gbooker, it seems you posted only once, but what you wrote was quite informative and useful. Your mentioning of FSEvents (macOS) brought me to investigate illumos/solaris File Event Notification system, which is now used by fswatch. Now we are using OmniOS on several servers because of that. Thank you, and thanks to this forum, because it always demonstrated to be a fount of unexpected wisdom.


----------



## rigoletto@ (Jun 16, 2018)

Apparently this will finally happen: Thread 66293.


----------



## unitrunker (Jun 16, 2018)

FreeBSD already has fdescfs(4). This could be extended to browse file descriptors for a specific process or process tree.


----------



## rigoletto@ (Jun 16, 2018)

This can already be done using kqueue(2) but that scale poorly because it does need to open a file descriptor per file, while inotify uses one per directory and FSEvents one per file system.

"All" of that is in this thread already.


----------



## unitrunker (Jun 16, 2018)

lebarondemerde said:


> "All" of that is in this thread already.



I read all three pages. The discussion seems to be biased towards how inotify / kqueue / FSEvents operate. Tracking all file handles for a specific (and worrisome) process might be an equally useful approach. Can this be combined with filtering by device or parent directory?

An inotify implementation would be great - but an even more flexible mechanism would be event better.


----------



## rigoletto@ (Jun 16, 2018)

I am not the best person to comment about techinical details, there are a lot people more capable than me in here, but I suppose (between the existing implementations) the most flexible is the AIX one; however no eye seen in there of course.

By the IBM article the files/directories to be watched should/can be mounted using something like a special fs:



> *Monitoring events with the AIX Event Infrastructure*
> The AIX Event Infrastructure is contained in the bos.ahafs fileset on AIX 6.1 TL 6 and AIX 7.1. To monitor events, first install the bos.ahafs fileset and mount an instance of the AIX Event Infrastructure file system:
> 
> mkdir /aha
> ...



Source.


----------



## gbooker (Aug 31, 2018)

Wow activity here again; I was worried that I had inadvertently killed this thread with my last reply.  I had stopped checking at one point so I didn't see until now see there is discussion here once more.

To clarify my comparisons of kqueue to inotify/fsevents I'll give a quick and dirty overview of the process for monitoring a file-system tree:
Setup:
With kqueue, the user-space developer must:

Enumerate the entire directory structure recursively
Open an FD for every directory and file
Add accounting to lookup by FD as well as by path (needed later)
With inotify:

Create inotify context
Enumerate the entire directory structure recursively
Register a watch in the context for every directory (not files)
Add accounting to lookup path by watch id (needed later)
With fsevents:

Register a single watch for the directory tree (no recursion)
Watching for changes:
With kqueue:

Loop over kevent giving it a giant array of fds from the accounting to get list of FDs changed
For each FD indicating change:
Lookup in accounting by FD
If a directory changed, scan dir to figure out what the change was
Use the by path accounting to determine file/directory additions and removals
If a file/directory was added, recursively add them (as in setup above)
If a file/directory was removed, recursively remove them from accounting and close the fds

With inotify:

Loop over read of inotify context
For each object read:
Lookup the path by watch id (provided in object) and add file/dir name (also provided) to get full path of change
Take action on the path in the object
If a directory was added, recursively add directories (as in setup above)
If a directory was removed, recursively remove directories from accounting and unregister with the context

With fsevents:

Read the path of the updated file/directory given in the callback
Take action on this path

I hope with the above it makes it more clear why I'd prefer something akin to the fsevents API oven inotify as it's kinder on the user-land developer.  With fsevents there is one object to open per monitored directory tree and it gives rich contextual information in the callback.  The only accounting needed by the user-land program is which directory trees are monitored and close that monitor if the entire directory tree disappears.  With inotify the user-land program must have accounting for every directory, but at least the notification there yields the path of what was modified or added and what the change in the FS was.  The most work is kqueue since the user-land program only has the fd of the FS change and no other contextual information.  This requires all the accounting I mentioned above.

As per the rationale for having FS monitoring, I can attest to the difference myself.  I used to be running ZFS on Ubuntu and I transitioned over to FreeNAS.  When I was on Ubuntu, if I rsynced a file over to my media directories, within seconds it was detected, scanned, and playable on the Plex client running on the TV.  When I changed to FreeNAS, I had to wait for a periodic scan to pick up the new file or manually kick it off myself.  I truly missed the automatic FS monitoring.  At the moment, I'm currently transitioning my media storage/server back to ZFS on Ubuntu and when I would copy a file to a media pool on the Ubuntu side, it'd show up in Plex in seconds once more without any interaction on my part.  It was quite satisfying to automatic FS change detection once more and reminded me of how much I missed it.

So I'll appeal for a new FS change API that gives rich callback information, such as path of the change, what the change was (add/modify/delete), and this be done on a directory-tree as a whole.


----------



## priyadarshan (Sep 1, 2018)

Thank you, very instructive.


----------



## rigoletto@ (Sep 1, 2018)

gbooker First, thank you for your quite interesting input!

Looking on the "AIX Event Infrastructure" example, can you guess (it is closed source) of how it works?

By what I understood that allow to the developer to use it to watch from individual files (what would be usefull for situations previously pointed by unitrunker) to entire file systems.

Cheers!


----------

