# Unable to delete some files



## potzilov (Feb 3, 2022)

Hi guys,

Recently I've discovered that there are some files that I'm unable to delete. See the screenshot below from `mc` (in /root directory).






A simple `ls` does show the file:





`ls` with other switches, e.g. `ls -la` does not show the file.

It's difficult for me to say what caused this, perhaps a power outage. What I do see is that there are additional files which are shown in `mc` as starting with the question mark (if it is indeed a question mark and not some other random symbol being shown as a question mark).

Another, more worrying example of the same is:





This leads to `cron` not being able to start. The same is true for `wlan0` and some other files and daemons.

My system is 
	
	



```
FreeBSD XXXX 13.0-RELEASE-p6 FreeBSD 13.0-RELEASE-p6 #0 releng/13.0-b0c8bc5d9: Wed Jan 26 13:24:54 CET 2022     xxxx@xxxx:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
```

Filesystem is ZFS on SSD and `zfs scrub` did not find any errors.

I did try the following suggestions from #freebsd on Libera.chat but without success:

Regular `rm \?.viminfo` - No such file or directory
Moving everything out of `/root` directory delete the directory with `rm -rf /root` - Directory not empty
Using `rm -fv -- \?.viminfo`
Using `ls -alo` - the file does not appear in the output
Using `printf "%s" *.viminfo` gives "no matches found *.viminfo"
Tried to find inode of the file in order to delete it that way - wasn't able to find the inode.
So now I'm stuck and hope you guys are able to help me.


----------



## Alain De Vos (Feb 3, 2022)

Maybe this might help,








						Stop shell wildcard character expansion?
					

Is there any way for a compiled command-line program to tell bash or csh that it does not want any wildcard characters in its parameters expanded?  For instance, one might want a shell command like...




					stackoverflow.com


----------



## Erichans (Feb 3, 2022)

Try `rm -i *` 
rm(1) and note the "-i" flag: it is important!
You'll be asked interactively for confirmation on each individual file.


----------



## grahamperrin@ (Feb 4, 2022)

I'm curious.

`file ./.viminfo`


----------



## ralphbsz (Feb 4, 2022)

In addition to the output from "file", also do an ls -l on them. It is a little worrisome that the date (actually mtime) of the file is 1970-01-01. That seems to indicate that something has wiped out the mtime attribute of the file. What other attributes did it wipe out? How about the ownership, and permissions? How about ownership and permission of the enclosing directory? One reason the file might be not deletable is: You don't own it, or you don't have permission to modify it, or you don't own or don't have permission for the directory it is in.

It is also very bizarre that a normal ls (without the -a switch) shows a file name that starts with a dot (it should not), while the ls with the -a switch does not show it (it should).

I wonder whether there is something funny going on with the file name. Perhaps there is a space or invisible character before or after the file name? Perhaps the leading dot is not really a dot, but some other unicode character which happens to look like a dot? You can try "ls -1 | hexdump -C", then we'll see the binary representation of the file name.


----------



## potzilov (Feb 4, 2022)

grahamperrin said:


> I'm curious.
> 
> `file ./.viminfo`




```
root@XXXX:~ # file ./.viminfo 
./.viminfo: cannot open `./.viminfo' (No such file or directory), ASCII text
root@XXXX:~ #
```

When I press `TAB` while typing the command it does autocomplete `.viminfo` and the output is as above.


----------



## grahamperrin@ (Feb 4, 2022)

Thanks, so I reckon, the name of the file somehow gained a non-printing character (or something like a non-printing character).

Here, after _intentionally_ adding a non-printable character:


```
% bfs ~ -name "*viminfo*" -print
/home/grahamperrin/
                   .viminfo
^C
% pwd
/usr/home/grahamperrin
% ls -hl *viminfo
-rw-------  1 grahamperrin  grahamperrin   933B  4 Feb 00:19 ?.viminfo
%
```



potzilov said:


> … I'm in no way an expert on FreeBSD (just a regular user of it for the past 20 years or so, but was not too adventurous), …



If terminal is too adventurous in this case, you can probably use an application such as Dolphin to show hidden files, then correct the file name. (Dolphin works for me.)

I say _probably_ because, as pointed out by ralphbsz, other things are unusual in your case.


----------



## covacat (Feb 4, 2022)

try 
find /root -print0 |xargs -0 stat -f "%Op %SN"


----------



## ralphbsz (Feb 4, 2022)

potzilov said:


> ```
> root@XXXX:~ # file ./.viminfo
> ./.viminfo: cannot open `./.viminfo' (No such file or directory), ASCII text
> ```


That makes no sense (which doesn't mean that it isn't what you're seeing).

The output from file tells me two things. First, that it cannot open .viminfo, because such a file does not exist. Second, that it can read some file (we don't know which file), which contains ASCII text. But the name of the second file is not visible on the command line. Also, the output format of file is broken. Here is an example: In my directory, I have a file named bar which contains ASCII text, and there is no file foo:

```
> file foo bar
foo: cannot open `foo' (No such file or directory)
bar: ASCII text
```
See the difference: If you give the file command two file names as arguments, it prints two lines of output. Your example looks like you gave file two arguments, but it printed only one line, and that line is contradictory. Also note that you used file name completion in the shell, which should not put two arguments on the command line.

I think I need to see the output of "ls -1 | hexdump -C" in that directory. I'm beginning to suspect that we have multiple files here, some of which are invisible, or may contain special characters that screw up output formatting. For example, it is possible to create a file whose name is a single space, and that will make ls and command line completion (with tab) look really funny. It is even possible to create a file whose name is a bunch of backspace characters ... but that wouldn't backspace across the newline.

EDIT: Covecat's suggested command is also very good, accomplishes the same as my ls command.

Very weird.


----------



## potzilov (Feb 4, 2022)

ralphbsz tried to do `ls -1` the output is:

```
root@XXXX:~ # ls -1 .viminfo 
ls: .viminfo: No such file or directory
```

Your following suggestion `ls -l | hesdump C` gives (partial output):

```
00000080  0a 2e 73 75 62 76 65 72  73 69 6f 6e 0a 2e 76 69  |..subversion..vi|
00000090  6d 0a 2e 76 69 6d 69 6e  66 6f 0a 6d 79 2e 63 6e  |m..viminfo.my.cn|
000000a0  66 0a 53 79 6e 63 0a                              |f.Sync.|
000000a7
```

I cannot see ownership attributes of the file, but can see that the ownership and flags of the `/root` directory are normal and haven't changed.

Erichans tried you suggestion but .viminfo does not appear in the list of files to be deleted.


----------



## potzilov (Feb 4, 2022)

covacat said:


> try
> find /root -print0 |xargs -0 stat -f "%Op %SN"



covacat tried:

```
root@XXXX:~ # find /root -print0 | xargs -0 stat -f "%Op %SN" | grep vim
40755 /root/.vim
100644 /root/.vim/.netrwhist
stat: /root/.viminfo: stat: No such file or directory
root@XXXX:~ #
```


----------



## ralphbsz (Feb 4, 2022)

potzilov said:


> ralphbsz tried to do `ls -1` the output is:
> 
> ```
> root@XXXX:~ # ls -1 .viminfo
> ...


I presume you typed the ".viminfo" on that command line by hand? In that case, this is one good piece of information. It tells use that there is no file whose name is ".viminfo", with the usual characters that come from typing it by hand.

But now, it gets totally bizarre:


> Your following suggestion `ls -l | hesdump C` gives (partial output):
> 
> ```
> 00000080  0a 2e 73 75 62 76 65 72  73 69 6f 6e 0a 2e 76 69  |..subversion..vi|
> ...


This is bizarre. You did a "ls -l" here (the option character is ell, the letter after i and before m). I expect to see roughly this format in the output:

```
> ls -l bar blatz
-rw-r--r--  1 ralph  ralph  37 Feb  3 23:35 bar
-rw-r--r--  1 ralph  ralph   0 Feb  4 00:01 blatz
```
But instead, what you got was the output format from a normal "ls -1" command (the option character is the digit one). And that output tells us that either a file named ".viminfo" exists (look for the string ".viminfo" followed by 0a = newline in the output). Or perhaps that a file named ".viminfo\nmy.cnf" exists (that's the name with a newline in it). That would be legal but weird. And having file names with newlines in it leads to amusing (bizarre? confusing?) output from standard commands. But: A single file with newline in the name does not explain all the "no such file..." error messages you are getting.

Now let's combine that with the output you posted from the find command:

```
# find /root -print0 | xargs -0 stat -f "%Op %SN" | grep vim
...
stat: /root/.viminfo: stat: No such file or directory
```
This is where things go bad. This tells me that find found a file named /root/.viminfo (with nothing behind the name!) and passed it to stat via xargs. Given that you used -print0 and -0 on find and xargs, we can assume that the filename was passed to stat unmolested. And then stat told us that such a file does not exist.

I think I'm starting to have a bad thought: I fear your file system is damaged. If the find utility can find a file named .viminfo that exists and is in the directory, but then stat can not open the file, that's an inconsistency. That sort of matches the suspicion I had above, which is that the 1-Jan-1970 date suspiciously looks like file attributes were damaged.

Time to (a) look in dmesg and /var/log/messages for error messages, and (b) go into single user mode and run fsck. Oh wait, that won't work. You wrote above "Filesystem is ZFS on SSD and zfs scrub did not find any errors". On ZFS, there is nothing like fsck to fix problems like an inconsistency in the file system ... scrub should have done what it can, and above and beyond that, there are only specialized tools such as zdb that are not automated.

I'm out of good ideas for how to help you. My only suggestion is: cd to /root, and post the output of "ls -l" without any arguments. Also, while you are in the area, do a "ls -l -d", perhaps there is something strange about the directory itself (although I don't know of any strangeness that could cause the symptoms you are seeing). Perhaps something on other files will give us a clue.


----------



## covacat (Feb 4, 2022)

```
#include <dirent.h>
#include <stdio.h>
#include <err.h>
int main(int argc,char **argv)
{
DIR *dir;
struct dirent *dp;
if(argc != 2) {
 errx(1,"%s <file>",argv[0]);
 }
dir = opendir(argv[1]);
if (!dir) {
        err(1,"Error opening %s",argv[1]);
    }

 while ((dp = readdir(dir)) != NULL) {
    fwrite(dp,sizeof(struct dirent),1,stdout);
    fflush(stdout);
    }
    closedir(dir);
//printf("%lu\n",sizeof(struct dirent));
return 0;
}
```
compile this cc a.c -o /tmp/ad
/tmp/ad /root/.vim |hexdump -Cv
/tmp/ad /root/.vim > /tmp/out.bin
post/attach the binary output


----------



## potzilov (Feb 4, 2022)

covacat:

```
root@XXXX:~ # cc a.c -o /tmp/ad
root@XXXX:~ # /tmp/ad /root/.vim | hexdump -Cv
00000000  15 90 0d 00 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000010  20 00 04 00 01 00 00 00  2e 00 00 00 00 00 00 00  | ...............|
00000020  92 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000030  20 00 04 00 02 00 00 00  2e 2e 00 00 00 00 00 00  | ...............|
00000040  16 90 0d 00 00 00 00 00  09 0b 16 17 00 00 00 00  |................|
00000050  28 00 08 00 0a 00 00 00  2e 6e 65 74 72 77 68 69  |(........netrwhi|
00000060  73 74 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |st..............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  92 00 00 00 00 00 00 00  |................|
00000120  02 00 00 00 00 00 00 00  20 00 04 00 02 00 00 00  |........ .......|
00000130  2e 2e 00 00 00 00 00 00  16 90 0d 00 00 00 00 00  |................|
00000140  09 0b 16 17 00 00 00 00  28 00 08 00 0a 00 00 00  |........(.......|
00000150  2e 6e 65 74 72 77 68 69  73 74 00 00 00 00 00 00  |.netrwhist......|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000220  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000230  16 90 0d 00 00 00 00 00  09 0b 16 17 00 00 00 00  |................|
00000240  28 00 08 00 0a 00 00 00  2e 6e 65 74 72 77 68 69  |(........netrwhi|
00000250  73 74 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |st..............|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000270  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000300  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000310  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000320  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000330  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 00 00 00 00 00 00                           |........|
00000348
```


----------



## potzilov (Feb 4, 2022)

ralphbsz thanks for your suggestions. I tried:

```
root@XXXX:~ # ls -l
ls: .viminfo: No such file or directory
total 74
drwx------  5 root  wheel      5 Jan  6 22:18 .cache
drwxr-xr-x  3 root  wheel      7 Jan  2 18:35 .composer
drwx------  5 root  wheel      5 Jan  2 23:25 .config
-rw-r--r--  1 root  wheel   1023 Apr  9  2021 .cshrc
drwx------  3 root  wheel      3 Jan  2 23:25 .dbus
-rw-------  1 root  wheel  26162 Feb  3 22:47 .history
-rw-r--r--  1 root  wheel     80 Apr  9  2021 .k5login
-rw-------  1 root  wheel     59 Jan 29 18:11 .lesshst
drwx------  3 root  wheel      3 Dec 22 00:32 .local
-rw-r--r--  1 root  wheel    328 Apr  9  2021 .login
-rw-------  1 root  wheel   2420 Jan  1 23:39 .mysql_history
-rw-r--r--  1 root  wheel    507 Apr  9  2021 .profile
-rw-r--r--  1 root  wheel    865 Apr  9  2021 .shrc
drwx------  2 root  wheel      3 Dec 22 00:33 .ssh
drwxr-xr-x  3 root  wheel      6 Jan  1 21:57 .subversion
drwxr-xr-x  2 root  wheel      3 Jan 10 18:18 .vim
-rw-r--r--  1 root  wheel    437 Feb  4 09:22 a.c
-rw-r--r--  1 root  wheel    981 Aug  6  2019 my.cnf
drwxr-xr-x  3 root  wheel      3 Jan  2 23:25 Sync
root@XXXX:~ # ls -l -d
drwxr-xr-x  12 root  wheel  23 Feb  4 09:22 .
root@XXXX:~ #
```


----------



## ralphbsz (Feb 4, 2022)

Thank you covecat for posting the binary! This will show us exactly what is in the directory.

Potzilov: Please run the same command again, but this time with the argument being just /root (so "/tmp/ad /root |hexdump -Cv"). What the previous run showed us: A directory named /root/.vim exists. It is readable without errors. I'm still decoding what it's content is, but I think it shows three files.


----------



## ralphbsz (Feb 4, 2022)

potzilov said:


> ```
> root@XXXX:~ # ls -l
> ls: .viminfo: No such file or directory
> total 74
> ...


Your file system is definitely screwed up. The directory contains a directory entry named .viminfo, but there is no such file. I fear you have found some sort of bug, which has caused some fort of file system corruption. We'll know more about what sort of corruption when we see the hexdump from the ad program on the /root directory.


----------



## potzilov (Feb 4, 2022)

ralphbsz said:


> Thank you covecat for posting the binary! This will show us exactly what is in the directory.
> 
> Potzilov: Please run the same command again, but this time with the argument being just /root (so "/tmp/ad /root |hexdump -Cv"). What the previous run showed us: A directory named /root/.vim exists. It is readable without errors. I'm still decoding what it's content is, but I think it shows three files.


Attached.


----------



## ralphbsz (Feb 4, 2022)

Each directory entry is 280 bytes = 0x118 bytes long. Remember, we're talking about the directory /root/.vim here. Do a "man dirent", and you'll see how to decode a directory entry.



potzilov said:


> root@XXXX:~ # /tmp/ad /root/.vim | hexdump -Cv
> 00000000  15 90 0d 00 00 00 00 00  01 00 00 00 00 00 00 00  |................|
> 00000010  20 00 04 00 01 00 00 00  2e 00 00 00 00 00 00 00  | ...............|
> 00000020  92 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
> ...


At offset 0, there exists a directory entry with inode number 0x0d9015. The next entry will be at offset 1 in the directory. This directory entry is 0x0020 bytes long (and I have no idea what this means, because it is really 0x118 bytes long). This entry is of type 0x04 = DT_DIR, which means it's a directory. The name is 1 byte long. The name starts at offset 0x18, and is the string ".", zero terminated. The remainder of the 256-byte long name buffer contains some gibberish, but that's OK.


```
00000110  00 00 00 00 00 00 00 00  92 00 00 00 00 00 00 00  |................|
00000120  02 00 00 00 00 00 00 00  20 00 04 00 02 00 00 00  |........ .......|
00000130  2e 2e 00 00 00 00 00 00  16 90 0d 00 00 00 00 00  |................|
00000140  09 0b 16 17 00 00 00 00  28 00 08 00 0a 00 00 00  |........(.......|
00000150  2e 6e 65 74 72 77 68 69  73 74 00 00 00 00 00 00  |.netrwhist......|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000220  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
```
At offset 1 (0x118 in bytes), there exists a directory entry with inode number 0x92. The next entry will be at offset 2 in the directory. This directory entry is 0x0020 bytes long (still don't understand). This entry is also of type 0x04 = DT_DIR, which means it's another directory. The name is 2 bytes long. The name starts at offset 0x130, and is the string "..", zero terminated. The remainder of the 256-byte long name buffer contains some gibberish, but that's also OK.

So far, everything is great: This directory contains . and .., as expected.


```
00000230  16 90 0d 00 00 00 00 00  09 0b 16 17 00 00 00 00  |................|
00000240  28 00 08 00 0a 00 00 00  2e 6e 65 74 72 77 68 69  |(........netrwhi|
00000250  73 74 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |st..............|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000270  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000300  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000310  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000320  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000330  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 00 00 00 00 00 00                           |........|
00000348
```
At offset 2 (0x230 in bytes), there exists a directory entry with inode number 0x0d9016. The next entry will be at offset 0x17160b09 in the directory; this is irrelevant, as this is the last one. This directory entry is 0x0028 bytes long (that is very bizarre, why not 0x20?). This entry is of type 0x08 = DT_REG, which means it's a regular file. The name is 10 bytes long. The name starts at offset 0x248, and is the string ".netrwhist", zero terminated, which is indeed 10 bytes. The remainder of the 256-byte long name buffer contains some gibberish, but that's OK.

So this just shows us that we know how to decode directory entries (mostly, the length in there is a bit of a mystery to me).


----------



## sko (Feb 4, 2022)

To see if it is in fact a filesystem hiccup, you might also want to investigate the dataset with zdb(8).

zdb can only work on whole datasets or pools, so you might want to try to create a new dataset and 'cp -a' the /root directory to that dataset (and hope the error gets carried over).

`zdb -dd zroot/copy-of-root` would give you the listing of all objects with their various sizes, type and the object number. The file (s) in question might show some implausible values for their size. Examine those objects further with `zdb -ddddd zroot/copy-of-root <objectnumber>`


----------



## ralphbsz (Feb 4, 2022)

The long output one in the zip file: Tomorrow. It's after midnight here. Maybe you can try to decode it yourself? The recipe is above.


----------



## covacat (Feb 4, 2022)

ralphbsz said:


> This directory entry is 0x0028 bytes long (that is very bizarre, why not 0x20?).


any filename longer than 8 will get 0x28 longer than 16 probably 0x30 and so
struct data before filename is 24 bytes long (0x18)


----------



## covacat (Feb 4, 2022)

the data seems okish
i speculate that the inode number for the borked files is wrong
rm(1) will always lstat the file before trying anything so if the inode is problematic will fail
no idea what will happen if you unlink them directly without a stat before


----------



## ralphbsz (Feb 4, 2022)

covacat said:


> any filename longer than 8 will get 0x28 longer than 16 probably 0x30 and so
> struct data before filename is 24 bytes long (0x18)


Ah, you're exactly right! The file name length is rounded up to the nearest 8 bytes, which is why the entries for file names of length 1 and 2 were 0x20 instead of the 0x18 that I expected, and the entry for the 10-byte filename is 0x28. Thank you, now I understand.


----------



## ralphbsz (Feb 4, 2022)

covacat said:


> the data seems okish



Allow me to confirm that, with details. I will use two examples, the directory called .vim which we looked at last night, and the problematic file .viminfo.

```
000000b0  15 90 0d 00 00 00 00 00  bc 04 98 12 00 00 00 00  |................|
000000c0  20 00 04 00 04 00 00 00  2e 76 69 6d 00 00 00 00  | ........vim....|
```
Inode number is 0x0d9015; that makes sense, and was the inode number for directory entry "." inside the .vim directory. The next entry will be at offset 0x9804bc; that is completely crazy, but may be irrelevant (I don't have time right now to think through how readdir, seekdir and telldir interact; I suspect the next entry number is only relevant when reading multiple entries at once, which we are not doing here). The length of the entry is 0x20 bytes, which also makes sense (see message above): 0x18, plus a name that's rounded up to 8 bytes. The type of the entry is 4, which means directory, correct. The length of the name is 4 bytes, and the name is ".vim", correctly 4 bytes.

Finding the entry for .viminfo is a little harder, because for some bizarre reason the unused part of the dirent-length buffer is filled with irrelevant copies of other directory entries. Here is the real one (I blanked out irrelevant parts of the previous one with X):

```
000014c0  XX XX XX XX XX XX XX XX  0f 02 0d 00 00 00 00 00  |XXXXXXXX........|
000014d0  10 e7 f2 19 00 00 00 00  28 00 08 00 08 00 00 00  |........(.......|
000014e0  2e 76 69 6d 69 6e 66 6f  00 00 00 00 00 00 00 00  |.viminfo........|
```
This tells us that there is an entry with inode number 0x0d020f = 852495. The next one is number 0x19f2e710, which is again irrelevant. The length of this entry is 0x28, which is sensible. The type of this entry is 8 = regular file, and the name length is 8 characters (correct). The name is ".viminfo", padded with a zero at the end. Because the zero pad is at the byte at offset 0x20, the length of the entry had to be padded out to 0x28 bytes.

So this is a perfectly sensible and valid directory entry for a file named .viminfo. As Covecat already said above, the likely cause is that there is no file system object with inode number 852495. How can you verify that? This is a little difficult. POSIX = Unix does not have a documented/published/official system call (or API) for finding or opening files by inode number. Most file systems have internal (undocumented/unpublished/inaccessible) APIs that can do that, but I don't have time right now to dig into ZFS internals to find it. Can someone find a ZDB command for that, please?

EDITed to add: If it is really true that the directory /root has a directory entry for a file named .viminfo with inode number 852495, but such an inode number does not exist, then (a) the file system is now corrupted, and it will require deep knowledge of ZFS internals to fix, and (b) you have found a bug, although we do not know where the bug is (at least, not yet).


----------



## _martin (Feb 4, 2022)

If you already asked on irc and didn't get much attention I'd suggest opening the PR or asking on mailing list (though I'd personally prefer the former).

With only commands from base you could check the inum of the .viminfo by `ls -lai /root/` (leftmost column is the inode) and use it in conjunction with find: `find /root -xdev -type f -inum $inum -exec cat {} \;` where $inum is the actual inum of the .viminfo. Most likely you get the same error as before.

While it's annoying to be in this situation there could be a bug that could do more harm next time.

Looking as a specific dnode (zfs inode): `zdb -AA -ddddd $zfs_fileset $dnode`. For example:

```
tbsd01:(~)# ls -lai .cshrc
131270 -rw-r--r--  1 root  wheel  1089 Oct 23  2020 .cshrc
tbsd01:(~)#

tbsd01:(~)# df -m .
Filesystem      1M-blocks Used Avail Capacity  Mounted on
rpool/ROOT/13.0     23759 4672 19087    20%    /
tbsd01:(~)#

tbsd01:(~)# zdb -AA -ddddd rpool/ROOT/13.0 131270
Dataset rpool/ROOT/13.0 [ZPL], ID 134, cr_txg 2867566, 4.56G, 236469 objects, rootbp DVA[0]=<0:2147d7000:1000> DVA[1]=<0:5d1772000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=800L/800P birth=7959118L/7959118P fill=236469 cksum=bcf813f93:e50cb1bbcab:abc520bd56894:5dfc62710b949ad

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
    131270    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
                                               168   bonus  System attributes
    dnode flags: USED_BYTES USERUSED_ACCOUNTED
    dnode maxblkid: 0
    path    /root/.cshrc
    uid     0
    gid     0
    atime    Wed Apr 14 22:10:25 2021
    mtime    Fri Oct 23 07:32:42 2020
    ctime    Wed Apr 14 22:10:25 2021
    crtime    Fri Oct 23 07:32:42 2020
    gen    2867815
    mode    100644
    size    1089
    parent    399
    links    1
    pflags    40800000004
Indirect blocks:
               0 L0 0:146422000:1000 600L/600P F=1 B=2867815/2867815 cksum=571c182773:53aa538d94f5:2c446ce14b2b2b:10d0419ba49cd37e

        segment [0000000000000000, 0000000000000600) size 1.50K

tbsd01:(~)#
```
If I were you I'd really like to hear devs opinion on the situation.


----------



## grahamperrin@ (Feb 4, 2022)

potzilov said:


> ```
> root@XXXX:~ # file ./.viminfo
> ./.viminfo: cannot open `./.viminfo' (No such file or directory), ASCII text
> root@XXXX:~ #
> ...



Thanks. Here, with my intentionally mis-named file:


```
% ls -hiln
total 1
48648 -rw-------  1 1002  1002   933B  4 Feb 23:29 ?.viminfo
47897 -rw-r--r--  1 1002  1002   6.3K  4 Feb 08:41 out.bin
48035 -rw-r--r--  1 1002  1002   670B  4 Feb 23:19 out.zip
% ls -hiln *.viminfo
48648 -rw-------  1 1002  1002   933B  4 Feb 23:29 ?.viminfo
% file
^K.viminfo  out.bin     out.zip  
% file .viminfo
.viminfo: cannot open `.viminfo' (No such file or directory)
% sh
$ find . -maxdepth 1 -inum 48648 -exec mv {} .viminfo \;
$ file .viminfo
.viminfo: ASCII text
$ exit
```

potzilov please, what's the serial number of the file in your case?

`ls -hiln *.viminfo`

Postscript

Deleted a post from page 1. Now, I remember using `-ddddd` years ago, maybe with Zevo. 



Spoiler: U+000B at the beginning of the name of a file





```
root@mowa219-gjp4-8570p-freebsd:~ # zdb -ddddd august/usr/home 48648
Dataset august/usr/home [ZPL], ID 287, cr_txg 14, 139G, 860330 objects, rootbp DVA[0]=<0:582aa3a000:1000> DVA[1]=<0:5e2871f000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=2282432L/2282432P fill=860330 cksum=10966ad5ae:2e00b3d802b7:440d9e25e4c8ac:470a35eaa17697da

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
     48648    1   128K     1K     4K     512     1K  100.00  ZFS plain file
                                               168   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /grahamperrin/Documents/IT/ZFS/OpenZFS/84000/
                                                             .viminfo
        uid     1002
        gid     1002
        atime   Fri Feb  4 23:29:39 2022
        mtime   Fri Feb  4 23:29:39 2022
        ctime   Fri Feb  4 23:45:27 2022
        crtime  Fri Feb  4 23:29:39 2022
        gen     2230775
        mode    100600
        size    933
        parent  48163
        links   1
        pflags  40800000004
Indirect blocks:
               0 L0 0:32c5458000:1000 400L/400P F=1 B=2230775/2230775 cksum=43e094ba5c:2669cb7d7774:d3e9e3b55bc75:3644d9eb2782884

                segment [0000000000000000, 0000000000000400) size    1K

root@mowa219-gjp4-8570p-freebsd:~ #
```


----------



## grahamperrin@ (Feb 4, 2022)

potzilov said:


> Attached.



Anyone, please: how can I work with the out.bin file? 

Point me to something in Stack Exchange, if you like.

(I've done this type of stuff _many_ years ago, but never with a FreeBSD-based system.)


----------



## _martin (Feb 5, 2022)

grahamperrin out.bin is just an arbitrary name for an output covacat chose in his post when he was using readdir(3) to get the files from a directory. From there you can see he's saving dirent(5) structure to the file.

Deeper question is, as I'm curious myself, how can this happen on a COW FS (this being a ghost entry in directory referring to a non-existent dnode and this being an observation from forum post, not a fact).


----------



## grahamperrin@ (Feb 5, 2022)

Thanks, I mean, what command did you (or can you) run on the provided binary, to 'see' its content? 

Incidentally, I _am_ aware that my artificial example lacks the `ASCII text` aspect when file(1) does not find the file.


----------



## ralphbsz (Feb 5, 2022)

The out.bin file is nothing but a binary dump of all the "struct dirent" that are returned by a series of readdir() calls. I showed what one can find in them above: Each contains the name of a thing in the directory (where thing = typically file or subdirectory, with special cases for bizarre things like devices and FIFOs, and with "." and ".." being like subdirectories too), the type of the thing, the inode number of the thing, and that's pretty much it. The way we did this here is that the OP and Covecat ran out.bin through hexdump -Cv, and then I looked at the hexdump; instructions are above.

Word of warning: If you do "man dirent", it shows you an ancient version of the dirent structure (which has two 8-bit numbers, and then a 32-bit pad word). The version of dirent that is really in use can be found by reading /usr/include/sys/dirent.h; in there the file type and file name length are de-facto 16-bit numbers (expressed in the header as an 8-bit number and an 8-bit pad byte).


----------



## ralphbsz (Feb 5, 2022)

grahamperrin said:


> potzilov please, what's the serial number of the file in your case?
> 
> `ls -hiln *.viminfo`



Not sure that a "ls -il" command will return that, since it may run into a brick wall trying to do a stat() call on the file, since the inode does not seem to exist. A "ls -i" command might work, or might not (don't know how ls is implemented). Anyway, I posted the file's inode number above (by reading it from the hexdump of the directory content), and it is 852495.

And as martin already wrote above: I don't think that an object with that inode number can be found, since the problem does seem to be that it does not exist. But to verify that, one could do "find / -inum 852495 -print", but my hunch is that it will return nothing. Finding the inode or verifying its absence will require going into ZFS internals, perhaps looking at older version of the inode storage (in older blocks, remember ZFS doesn't overwrite things on disk for as long as possible, it appends internally).


----------



## grahamperrin@ (Feb 5, 2022)

Thanks,



ralphbsz said:


> … I showed what one can find in them above: …



Your `Code` fragments in <https://forums.freebsd.org/posts/554604> above were taken from output from what command, or utility?

Sorry for leaning towards ELI5, but this is where the dyslexic side of me _really_ needs to understand from the outset 'cause if I learn wrong, I might never properly un-learn.

(Decades ago, I used HexEdit on Macs. Probably laughable now. Not that I _require_ a GUI now, just, I need a pointer on what best to use to 'see' the content of the .bin file that was provided (zipped) on page 1.)


----------



## ralphbsz (Feb 5, 2022)

grahamperrin said:


> Your `Code` fragments in <https://forums.freebsd.org/posts/554604> above were taken from output from what command, or utility?


Take the a.c program that Covecat posted, compile it into /tmp/ad. Run /tmp/ad (with the correct argument), save the output in a file called out.bin. Then run "hexdump -Cv out.bin", and you get the hex dump that I cut and pasted from. Or eliminate the middleman of saving it into a file, and run "/tmp/ad /root | hexdump -Cv", and the hexdump is right there.

Last night at midnight I was thinking that I should take Covecat's nice program, and make it prettier, so instead of just dumping the data in binary, it formats it nicely. By that time it was after midnight, so I didn't. And tonight I have lots of work to do.


----------



## grahamperrin@ (Feb 5, 2022)

ralphbsz said:


> `hexdump -Cv out.bin`



Perfect. That's it, thanks.


----------



## Friend Of Jolly Devil (Feb 5, 2022)

Are you sure your SSD is OK?

It seems that directory entries at X have a copy at X-248 bytes, maybe your inode currently is at 852495-248?


----------



## Helmut (Feb 7, 2022)

potzilov said:


> potzilov said:
> 
> 
> > Hi guys,
> ...



Did you try 
	
	



```
rm -i \.*
```
? Had the same issue with a directory that contained unicode characters.


----------



## Andriy (Feb 7, 2022)

ralphbsz said:


> Can someone find a ZDB command for that, please?


Sure:
`zdb -dddd dataset inum`
E.g.:

```
# zdb -dddd pond/usr/devel 5876111
Dataset pond/usr/devel [ZPL], ID 13311, cr_txg 7374, 43.1G, 3248754 objects, rootbp DVA[0]=<0:224007d4000:200> DVA[1]=<0:1ac03fc7400:200> [L0 DMU objset] fletcher4 lz4 unencrypted LE contiguous unique double size=800L/200P birth=81735298L/81735298P fill=3248754 cksum=11ebe42940:5e14a8496a6:10daa419cd79d:22ac6a867e6de9

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
   5876111    1   128K    59K    26K     512    59K  100.00  ZFS plain file
                                               168   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED
        dnode maxblkid: 0
        uid     1001
        gid     0
        atime   Wed Feb  3 20:54:51 2021
        mtime   Wed Feb  3 20:54:51 2021
        ctime   Wed Feb  3 20:54:51 2021
        crtime  Wed Feb  3 20:54:51 2021
        gen     74765151
        mode    100644
        size    60219
        parent  5875861
        links   1
        pflags  40800000004
```

You can vary a number of d-s to get more or less detailed information.


----------



## _martin (Feb 7, 2022)

Andriy If only I didn't mention this already.


----------



## grahamperrin@ (Feb 26, 2022)

My own case, hopefully less peculiar. I plan to remove a directory, but not until I understand more of what's below.

For a strangely named empty directory at the root of a file system, an inode `32904` with two (2) links, is it strange that find(1) finds only one file?

`find / -xdev -inum 32904 -ls`



Spoiler: output from zdb(8)





```
root@mowa219-gjp4-8570p-freebsd:~ # zdb -ddddd august/ROOT/n253343-9835900cb95-b 32904
Dataset august/ROOT/n253343-9835900cb95-b [ZPL], ID 11281, cr_txg 2542568, 49.6G, 2701309 objects, rootbp DVA[0]=<0:d92d5dc000:1000> DVA[1]=<0:c0157c7000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=2573771L/2573771P fill=2701309 cksum=1191d9bb3f:329f1aa61da7:4e165d55c16d65:5501ef751bf3813b

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
     32904    1   128K    512      0     512    512  100.00  ZFS directory
                                               168   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /
        uid     0
        gid     0
        atime   Tue Feb 23 23:55:00 2021
        mtime   Tue Feb 23 23:55:00 2021
        ctime   Tue Feb 23 23:55:00 2021
        crtime  Tue Feb 23 23:55:00 2021
        gen     11323647
        mode    41777
        size    2
        parent  4
        links   2
        pflags  40800000144
        microzap: 512 bytes, 0 entries

Indirect blocks:
               0 L0 EMBEDDED et=0 200L/1bP B=2053

                segment [0000000000000000, 0000000000000200) size   512

root@mowa219-gjp4-8570p-freebsd:~ #
```





```
% ls -hiln / | head -n 3
total 1
  32904 drwxrwxrwt   2 0  0     2B 23 Feb  2021
      4 drwxr-xr-x   2 0  0     2B 26 Feb 04:05 august
% ls -Bhiln / | head -n 3
total 1
  32904 drwxrwxrwt   2 0  0     2B 23 Feb  2021 \011\011\011
      4 drwxr-xr-x   2 0  0     2B 26 Feb 04:05 august
% ls -hiln /
total 1
  32904 drwxrwxrwt   2 0  0     2B 23 Feb  2021 ???
      4 drwxr-xr-x   2 0  0     2B 26 Feb 04:05 august
     33 drwxr-xr-x   2 0  0    46B 23 Feb 01:45 bin
      …
```


Cross-reference: <https://old.reddit.com/r/freebsd/comments/t210w2/-/>


----------



## ralphbsz (Feb 26, 2022)

grahamperrin said:


> For a strangely named empty directory at the root of a file system, an inode `32904` with two (2) links, is it strange that find(1) finds only one file?



An empty directory usually has a link count of two: One is the link from the name of the directory, the second is the link from "." inside the directory. If the directory has subdirectories, they increase the link count, because of the ".." entries inside the subdirectory.

All the rest you show is a directory with a really weird name (three times the \011 character, which is the tab character, or Control-I), and weird permissions (the 0777 is strange enough, but then add the final "t" which I think means sticky). That directory should be perfectly usable and deletable.


----------



## grahamperrin@ (Feb 27, 2022)

ralphbsz said:


> … the second is the link from "." …



Thanks! 

I never guessed (but could have compared with the empty /bootpool directory, which is amongst the three other items to be removed). 

Removed: 


```
root@mowa219-gjp4-8570p-freebsd:~ # bfs -xdev -inum 32904 / -ls
    32904      1 drwxrwxrwt   2 root     wheel           2 Feb 23  2021 /
^C
root@mowa219-gjp4-8570p-freebsd:~ # bfs -xdev -inum 32904 / -rm
^C
root@mowa219-gjp4-8570p-freebsd:~ # bfs -xdev -inum 32904 / -ls
^C
root@mowa219-gjp4-8570p-freebsd:~ #
```


----------

