# Reading files larger than 606666752 bytes



## noobster (Apr 18, 2009)

I'm trying to read a large file in C using open() and read(). I am doing this is a non-blocking way using a kqueue. Everything works fine until 606666752 bytes have been read from the file descriptor, but at that point kevent stops returning (i.e., no bytes are ready to be read). For smaller files, this is not an issue. Is there any setting I have to change to be able to read large files? Thanks in advance.


----------



## Mel_Flynn (Apr 19, 2009)

Are you testing for EV_ERROR or similar?

FreeBSD off_t should handle up to OFF_MAX, which is nowhere near the value you're seeing. Would be easier to see your code...


----------



## noobster (Apr 19, 2009)

If I run the code posted in the other thread (https://forums.freebsd.org/showpost.php?p=20582&postcount=3) on a large file, for some reason it hangs at 606666752 bytes. Any idea why?


----------



## graudeejs (Apr 19, 2009)

check if there is limit set in /etc/login.conf


----------



## noobster (Apr 19, 2009)

All settings seem to be set to unlimited. Also, I'm sure I didn't edit this file.


----------



## Mel_Flynn (Apr 19, 2009)

Let us know if this is the same problem as EOF, you may want to warnx("%lli", ke.data) just before going to the next loop sequence.


----------



## noobster (Apr 19, 2009)

Ah, it looks like that ke.data contains the value 606666752 in the first loop instead of the correct file size. Therefore, it only reads 606666752 bytes. I suspect this is because of the limited size of that struct field, but I have no idea how to get around this limitation. Anyone?


----------



## Mel_Flynn (Apr 19, 2009)

When ke.data - BUFSIZ reaches <=0, issue EV_CLEAR on the next invocation? Not sure it'll work, but from coarse reading of the code, EV_CLEAR should clear the kernel copy of the event. Then you need to solve the problem to detect when really EOF.


----------



## noobster (Apr 19, 2009)

Thanks for the suggestion, but it didn't work.


----------



## Mel_Flynn (Apr 19, 2009)

Is this file INT_MAX+606666752 bytes? Cause I can't reproduce it with a 640M file.


----------



## noobster (Apr 19, 2009)

The file I'm using is 4901634048 bytes, which is 4.6 gigabytes. So the file is much bigger than INT_MAX.


----------



## noobster (Apr 19, 2009)

It looks like the problem is with the file I am using for testing. Other large files work fine. I don't know exactly what's wrong with the culprit, but it seems as if the file size is much larger than the actual file contents. Weird.


----------



## Mel_Flynn (Apr 20, 2009)

Sparse file maybe? Tho, EOF should be at what stat(2) says. I'd copy it to another dir. It's possible the directory entry is lying. If so, do fsck -y in single user.


----------

