# Function fallocate() & splice() from FreeBSD.



## nagual2 (Oct 29, 2012)

Some time ago I began to develop a test file systems for Linux and FreeBSD
https://github.com/nagual2/Test

And to assess the effects of file fragmentation on performance file system I needed to implement add and remove blocks in the middle of the file - "hole punching" I can't find an analogue functions fallocate() and splice() under FreeBSD. And while I'm running linux emulator these functions are not available when compiling:

```
# if HAVE_LINUX_FALLOCATE
return fallocate (fd, mode, offset, len);
# else
return EIO_ENOSYS ();
# endif
```
Function posix_fallocate() not analgichna fallocate().
How do I implement fallocate() and splice() on FreeBSD? Or how to use those that are in the emulator linux?


----------



## expl (Oct 29, 2012)

You can always implement these syscalls in userworld, but then it would be unfair to compare them to Linux since obviously due to design they will run much slower. In conclusion you should not write I/O tests that will target multiple OSes using Linux only not portable syscalls...


----------



## nagual2 (Oct 29, 2012)

I am developing a test with Perl using library IO::AIO is implemented as an interface to libeio (http://software.schmorp.de/pkg/libeio.html). But it turned out that FreeBSD does not work. Do I understand correctly that there are no analogues FreeBSD fallocate() and splice()? As for the file system ZFS and UFS UFS2 admit they add - remove the block of any length with any offset in the file, or there are restrictions on reversal of file systems?


----------



## SirDice (Oct 29, 2012)

nagual2 said:
			
		

> Do I understand correctly that there are no analogues FreeBSD fallocate() and splice()?



Why would we need them?



> *This is a nonportable, Linux-specific system call*.  For the portable,
> POSIX.1-specified method of ensuring that space is allocated for a file, see
> posix_fallocate().


http://www.kernel.org/doc/man-pages/online/pages/man2/fallocate.2.html

Same for splice():


> CONFORMING TO
> *This system call is Linux-specific.*


http://www.kernel.org/doc/man-pages/online/pages/man2/splice.2.html


----------



## nagual2 (Oct 29, 2012)

SirDice said:
			
		

> Why would we need them?


I manipulate large files. I'm need "hole punching".


----------



## SirDice (Oct 29, 2012)

I understood that, I'm just wondering why FreeBSD would need them.


----------



## nagual2 (Oct 29, 2012)

And how can you add a block in the middle of the file length of 20 gigabytes? Without fallocate()? Without the "hole punching"? In the middle of the file, not the end.


----------



## SirDice (Oct 29, 2012)

Like you would solve it without those functions, copy the data until the point you need to insert, insert the data and copy the rest of the original file. Delete old data.


----------



## nagual2 (Oct 29, 2012)

It's impossible. File sizes over 10 gigabytes.


----------



## expl (Oct 29, 2012)

Well FreeBSD has ftruncate() syscall, thats is very similar and differently from fallocate is portable.


----------



## nagual2 (Oct 29, 2012)

expl said:
			
		

> Well FreeBSD has ftruncate() syscall, thats is very similar and differently from fallocate is portable.



function ftruncate() truncates the end of the file, it does not set the length of the deleted fragment.
Ð¡an not be done splice() from ftruncate().


----------



## expl (Oct 29, 2012)

nagual2 said:
			
		

> It's impossible. File sizes over 10 gigabytes.



It is possible, just need to do it in reverse order. Allocate space at the end of the file equal to the size of your 'hole', then move data from before the allocation bit by bit (use buffered and not raw read/write functions, they are optimized for this kind of thing) towards the end until you reach the position where you need to insert the data, write the change and quit.

edit:
I am pretty sure thats how fallocate works, it's just done at kernel level so it has advantage of knowing exact block layout so it can optimized much better.


----------



## nagual2 (Oct 29, 2012)

This is possible but it is very slow and it's creators extra unnecessary load on the drive.


----------



## expl (Oct 30, 2012)

nagual2 said:
			
		

> This is possible but it is very slow and it's creators extra unnecessary load on the drive.



Can you better explain why would you test this sort of thing? No well designed software will ever need/use such behavior so why test for it?


----------



## nagual2 (Oct 30, 2012)

To quickly create test conditions performance file system under the maximum fragmentation are two options - one file large file and many small. So if one more - provide read and write operations on the test disk from 0% filled with files up to 100% and calculate the average read and write speeds. As soon as the read and write speeds stop falling, we believe the maximum fragmentation and the resulting rate for the FS. Actually more complicated: If we write then write in the middle of inserting a file with an arbitrary offset from the start and arbitrary length unit. If you remove the block then do not remove it completely and cut out of the middle with an arbitrary displacement and arbitrary length and then add it to the end. What are the functions that will allow it to implement?

Any ideas?


----------



## expl (Oct 30, 2012)

nagual2 said:
			
		

> To quickly create test conditions performance file system under the maximum fragmentation are two options - one file large file and many small. So if one more - provide read and write operations on the test disk from 0% filled with files up to 100% and calculate the average read and write speeds. As soon as the read and write speeds stop falling, we believe the maximum fragmentation and the resulting rate for the FS. Actually more complicated: If we write then write in the middle of inserting a file with an arbitrary offset from the start and arbitrary length unit. If you remove the block then do not remove it completely and cut out of the middle with an arbitrary displacement and arbitrary length and then add it to the end. What are the functions that will allow it to implement?
> 
> Any ideas?



You can't achieve accurate maximum fragmentation from userworld, actual fragmentation will vary a lot depending on filesystem and physical media. But if you want to simulate fragmentation anyways you have to use many files to make it as accurate and portable as possible.


----------



## nagual2 (Oct 30, 2012)

I understand that the length of the recording unit should be a multiple of the block size file system.


----------

