# SSH fail because /var/empty missing after removing dovecot packages



## frijsdijk (Mar 11, 2020)

Yesterday on a server I removed dovecot packages. After removing dovecot (pkg remove dovecot), pkg suggests to remove the users as well. When you rmuser those accounts (dovecot and dovenull), their homedirectories are also removed (there goes /var/empty), and SSH stops working.

Is this a bug?


----------



## VladiBG (Mar 11, 2020)

just create the /var/empty directory again if it goes deleted.
`mkdir /var/empty
chown root:wheel /var/empty
chmod 555 /var/empty`


----------



## asteriskRoss (Mar 11, 2020)

frijsdijk said:


> Is this a bug?


Since the mail/dovecot port doesn't delete /var/empty as part of the package removal, I don't think this can be considered a bug.

The man page for rmuser(8) suggests that it will ask for confirmation before deleting a home directory.  Did you answer yes?  Or invoke it as `rmuser -y dovecot` so that all confirmations were assumed yes?

As VladiBG suggested, you can happily recreate /var/empty without issue.  You should also set the immutable file flag using chflags(1) as root with the command `chflags schg /var/empty`, which will ensure the directory cannot be deleted in future without unsetting the flag.


----------



## frijsdijk (Mar 12, 2020)

My concern here mainly is the pitfall. In my case this resulted locking myself out from a physical machine and I had to go through some hoola-hoops to get access again.

The crux is that a users's homedir, which is a directory that's not exclusively owned by this user but is shared by other users (and can't even write in it) is removed, and it breaks essential services.


----------



## asteriskRoss (Mar 12, 2020)

You have my sympathy for needing to gain physical server access to resolve an issue!  I would expect the /var/empty directory to have the schg (immutable) flag set on a default FreeBSD installation to prevent even root from removing it by mistake.  The flags can be seen using the command `ls -lod /var/empty`.  Can you recreate an issue where the schg flag isn't set during the installation?  Or where the flag is ignored when there is an attempt to delete a directory marked as immutable?


----------



## ShelLuser (Mar 12, 2020)

On what FreeBSD version did this happen?


----------



## frijsdijk (Mar 13, 2020)

This is 11.3-RELEASE-p5


----------



## frijsdijk (Mar 13, 2020)

asteriskRoss said:


> Since the mail/dovecot port doesn't delete /var/empty as part of the package removal, I don't think this can be considered a bug.
> 
> The man page for rmuser(8) suggests that it will ask for confirmation before deleting a home directory.  Did you answer yes?  Or invoke it as `rmuser -y dovecot` so that all confirmations were assumed yes?
> 
> As VladiBG suggested, you can happily recreate /var/empty without issue.  You should also set the immutable file flag using chflags(1) as root with the command `chflags schg /var/empty`, which will ensure the directory cannot be deleted in future without unsetting the flag.



Yep, I know how to fix it, and the chflag will permanently fix it. Only reason I'm posting this is to suggest taking care of this in the OS itself. Removing a user after removing a package, should never remove it's homedir if it's not actually their (chown) homedir, is also shared with other applications (which are part the OS and not even ports) and services, and even breaks other services (like sshd) if removed.

I was in fact warned (just reproduced it to test this) of removing the homedir. And I should (could) have read it better (honestly I didn't), just pressed yes because the end results were the least I was expecting.


----------



## SirDice (Mar 13, 2020)

frijsdijk said:


> Removing a user after removing a package, should never remove it's homedir if it's not actually their (chown) homedir, is also shared with other applications (which are part the OS and not even ports) and services, and even breaks other services (like sshd) if removed.


It doesn't. Removing the package doesn't remove the account(s). It only _suggests_ to remove the accounts if they're not needed any more. The actual removal is left up to you to do. So it was you that should have verified if the set home directory was still required. Just as you had to verify you really didn't need those accounts any more. You basically shot yourself in the foot, and the system won't stop you from doing that. Just as it won't stop you from nuking your system by doing a `rm -rf /`.


----------



## frijsdijk (Mar 13, 2020)

Yes, of course, it was me. You're right. Let's not try to make an OS move intuitive, and take away pitt-falls. 

Dude, I love FreeBSD. I've been using it since 2.2.x something. Still love it, still use it. But it's this attitude on the forums that breaks my love for FreeBSD. It's literally ALWAYS the user that's wrong. And as I said, yes, you are right. I should have looked better. I admitted that already.

Ah, nevermind. It's useless.


----------



## SirDice (Mar 13, 2020)

frijsdijk said:


> It's literally ALWAYS the user that's wrong.


In 999 out of 1000 times, yes, it is.

Let's reverse the question, how would you suggest the system would handle this? The things you're suggesting simply aren't possible or feasible. Which is why the actual removal of the accounts is left up to the system's administrator.



frijsdijk said:


> Removing a user after removing a package


It can't tell, it doesn't know. And what if I want to remove the account a few days later? You've used rmuser(8) but what about pw(8)? And a few other ways of account management? What if NIS or LDAP is involved?



frijsdijk said:


> should never remove it's homedir if it's not actually their (chown) homedir


What if you know it's not used any more and still want to remove the home directory? 



frijsdijk said:


> is also shared with other applications (which are part the OS and not even ports) and services


How would it be able to figure this out?



frijsdijk said:


> and even breaks other services (like sshd) if removed.


So, we need to keep track of which service uses what directories? How would the application know it's used by something else?


----------



## frijsdijk (Mar 15, 2020)

I've given a few suggestions already. But you don't even read apparently.  I mean.. the fact that this directory isn't owned by the user, perhaps? The fact that this user can't even write in it?? The fact that more users 'share' this directory??? The fact that it's named "/var/empty" .. ???? Just a few suggestions.

How do I remove my account from this forum? I can't find this..  I'm done here.


----------



## asteriskRoss (Mar 17, 2020)

frijsdijk said:


> How do I remove my account from this forum? I can't find this.. I'm done here.


I'm sorry you feel that way, frijsdijk.  My experience of the forums has been that there is much to be gained from the collective knowledge of people here, the vast majority of whom are trying only to help fellow FreeBSD users.


----------



## ehanneken (Mar 17, 2020)

Should `dovecot` and other program accounts use `/nonexistent` for a home directory, instead of `/var/empty`?


----------



## ShelLuser (Mar 17, 2020)

The main problem, the way I see it, is that some people have grown to become snowflakes and they simply can't appreciate "_no_" or "_it's your own doing_" for an answer, heck; some might even feel insulted over such comments. They also often overlook the fact that they're not the only person who has encountered the issue at hand and many suggestions to "fix" things have preceded them.

Bottom line: Unix is an operating system which will do exactly what you tell it to do and it won't try to hold your hand in order to prevent you from messing up. This is also not its job, if you want something like that you should probably look into Linux or maybe Windows (though...  in this particular example even Windows would not have been of much help here I think because it too would have happily removed the home directory. Especially if you're using PowerShell which is what veteran Windows admins tend to do).

Back on topic, I also get the impression that the OP isn't telling the full story here, from rmuser(8):


> 5.   Removes the user's home directory (*if it is owned by the user*),
> including handling of symbolic links in the path to the actual home
> directory.


So either there's a bug in rmuser which made it remove the directory regardless of the owner or the OP made a mistake here. I can't reproduce any issues on 12.1. Note that I'm not trying to rub anything in here, but since the OP repeated the possible fix of checking the home directories ownership...  yah, it seems rmuser already does that:

`root@magi:/home/peter # ls -ld /var/empty
dr-xr-xr-x  2 root  wheel  2 Dec  7  2018 /var/empty
root@magi:/home/peter # rmuser testor
Matching password entry:

testor::1002:1002::0:0:The Test:/var/empty:/bin/sh

Is this the entry you wish to remove? y
Remove user's home directory (/var/empty)?
Remove user's home directory (/var/empty)? y
Removing user (testor): mailspool home passwd.
root@magi:/home/peter # cd /var/empty
root@magi:/var/empty # cd ..
root@magi:/var # ls -ld empty
dr-xr-xr-x  2 root  wheel  2 Dec  7  2018 empty
root@magi:/var #`

So the way I see it... there's nothing to fix here. Definitely not on 12.1.


----------



## frijsdijk (Mar 18, 2020)

Strange..

I created a user "testuser":

`[root@progress ~]# id testuser
uid=6060(testuser) gid=6060(testuser) groups=6060(testuser)

[root@progress ~]# grep testuser /etc/passwd  /etc/group
/etc/passwd:testuser:*:6060:6060:test user:/var/empty2:/usr/sbin/nologin
/etc/group:testuser:*:6060:

[root@progress ~]# ls -ld /var/empty2
dr-xr-xr-x  2 root  wheel  2 Mar 18 09:22 /var/empty2

[root@progress ~]# rmuser testuser
Matching password entry:

testuser:*:6060:6060::0:0:test user:/var/empty2:/usr/sbin/nologin

Is this the entry you wish to remove? y
Remove user's home directory (/var/empty2)? y
Removing user (testuser): home passwd.
[root@progress ~]# grep testuser /etc/passwd  /etc/group
[root@progress ~]# ls -ld /var/empty2
ls: /var/empty2: No such file or directory

[root@progress ~]# uname -r
11.3-RELEASE-p5`

I did the test with /var/empty2 here, because I don't want to delete /var/empty again .. but I guess the issue is clear here.


----------



## ShelLuser (Mar 18, 2020)

frijsdijk said:


> I did the test with /var/empty2 here, because I don't want to delete /var/empty again .. but I guess the issue is clear here.


I appreciate that. And for the record, despite my cynical comment above I'm on the same page as asteriskRoss on this one: I also think it would be a shame if you were to leave because of one bad experience.


----------



## frijsdijk (Mar 18, 2020)

So even if I manage to reproduce the issue, it's still all ignored?


----------



## ehanneken (Mar 18, 2020)

`rmuser(8)` is a shell script that calls `pw(8)` with `userdel -r`. According to the man page, `pw(8)` is supposed to "only remove files and directories that are actually owned by the user, or symbolic links owned by anyone under the user's home directory." However, I looked at the code, and it seems to me that the ownership check is done only on files, and not on directories.


```
void
rm_r(int rootfd, const char *path, uid_t uid)
{
    int dirfd;
    DIR *d;
    struct dirent  *e;
    struct stat     st;

    if (*path == '/')
        path++;

    dirfd = openat(rootfd, path, O_DIRECTORY);
    if (dirfd == -1) {
        return;
    }

    d = fdopendir(dirfd);
    while ((e = readdir(d)) != NULL) {
        if (strcmp(e->d_name, ".") == 0 || strcmp(e->d_name, "..") == 0)
            continue;

        if (fstatat(dirfd, e->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0)
            continue;
        if (S_ISDIR(st.st_mode))
            rm_r(dirfd, e->d_name, uid);
        else if (S_ISLNK(st.st_mode) || st.st_uid == uid)
            unlinkat(dirfd, e->d_name, 0);
    }
    closedir(d);
    if (fstatat(rootfd, path, &st, AT_SYMLINK_NOFOLLOW) != 0)
        return;
    unlinkat(rootfd, path, S_ISDIR(st.st_mode) ? AT_REMOVEDIR : 0);
}
```

Files get unlinked in the `else if` clause. Subdirectories are removed with recursive calls to `rm_r` in the preceding `if` clause. Ultimately, a directory is removed by the call to `unlinkat` at the bottom of the function, where there is no check performed. `unlinkat` will fail if a directory is non-empty, however.

The one thing I can't explain is ShelLuser's results. I need to test this myself, but I'm not near a FreeBSD machine at the moment.

UPDATE: I can explain ShelLuser's results. On my stock 12.1 system, `/var/empty` is protected with the `schg` flag:


```
root@bluejay:~ # ls -dlo /var/empty
dr-xr-xr-x  2 root  wheel  schg,uarch 2 Nov  1 00:25 /var/empty
```

I don't know if this changed after 11.3. `pw(8)` should probably be fixed, nevertheless.


----------



## frijsdijk (Mar 19, 2020)

Hmm interesting. On my Home FBSD box, /var/empty is also schg,uarch ..

Still the manpage does not reflect the source code of rmuser it seems? On top of that it appears to be clear that developers of FreeBSD have tried to protect /var/empty from being removed, but if I check a couple of boxes, results differ. I have another 11.3 box that does not protect /var/empty with chflags. It might be something wilt zfs or ufs filesystems, though both support chflags.

Or perhaps it's because the box on where it went wrong for me was installed with some root-on-zfs script I found somewhere, that didn't set the chflags. It is a 11.3 box now, but I think it came from all the way back of 9.x or 10.x something, when I think installing on zfs was not possible from the installer itself. Too long ago, can't remember.

Still, rmuser should respect the ownership of the dir it's about to remove.


----------



## ehanneken (Mar 19, 2020)

It sounds like we have a probable explanation for frijsdijk's `/var/empty` being removed. It's a combination of a bug in `pw(8)` that manifests only in singular circumstances, and a machine history unusual enough to create those conditions.

Unless someone beats me to it, I'll write my very first bug report against FreeBSD this weekend, and include a patch. Feel free to beat me to it. Whoever does it should post the tracking number in this thread. I don't expect this to be a hot issue, but at least there will be a record in the problem report database.


----------



## ShelLuser (Mar 19, 2020)

frijsdijk said:


> So even if I manage to reproduce the issue, it's still all ignored?


Keep in mind that we're all users here. You need to sent in an official bug report to get the gears moving, as ehanneken already mentioned above.



ehanneken said:


> The one thing I can't explain is ShelLuser's results. I need to test this myself, but I'm not near a FreeBSD machine at the moment.
> 
> UPDATE: I can explain ShelLuser's results. On my stock 12.1 system, `/var/empty` is protected with the `schg` flag:


Confirmed!  And nicely spotted too, here's my follow up:



> root@magi:/home/peter # ls -lod /var/empty
> dr-xr-xr-x  2 root  wheel  schg 2 Dec  7  2018 /var/empty
> root@magi:/home/peter # chflags noschg /var/empty
> root@magi:/home/peter # rmuser testor
> ...


"Not good", because testor doesn't seem to match root last I checked.

For the record: I did reproduce this on a production setup but because TransIP gives you access to the VPS console there's no reason for me to worry. Damage is already undone. It's what I advice to anyone who wants VPS hosting: get console access....


----------



## ehanneken (Mar 22, 2020)

Bug 244967


----------

