# Daemon auditdistd (receiver) does not work in a jail



## BrunolPhilippe (Apr 26, 2018)

Hello

When I configure the daemon auditdistd(8) in my jail, I obtain this issue:

```
(53174) [WARNING] [myhost] (receiver) Unable to jail to directory /var/empty: Operation not permitted.
(53174) [WARNING] [myhost] (receiver) Unable to change root directory to /var/empty: Operation not permitted.
```

In fact, the daemon "auditdistd" create a specific jail (aka sandbox) to secure connection between him and the client(s). So I don't know if it's possible to launch "auditdstd" (receiver mode) in a exiting jail.

Can we create a jail in the jail ? Are there any options to configure correctly my jail ?Thank for your support

P.S. : My configuration for auditdistd(8) runs correctly if I don't use a jail for the receiver.
Regards


----------



## ShelLuser (Apr 26, 2018)

Now, first of all a small disclaimer: although I am familiar with auditdistd I don't have extensive hands on experience with it. Keep that in mind.

Anyway, I'm quite convinced (based on what I've read about it) that it doesn't require (or set up) a full jail merely to run. It seems more likely that it's trying to set up a chroot of some sort. So I think that your problems are caused due to a certain lack of permissions within your jail.

For example... By default you're unable to use the `ping` command from within a jail, unless you set up the required permission. In this case by allowing the use of raw sockets with specifying the allow.raw_sockets permission. So I think (but I'm not 100% sure) that you're experiencing something similar.

What happens if you try to start auditdistd manually (use the -F parameter to make it start in the foreground) and check for any specific error messages? Does that specify anything more specific about why it cannot use /var/empty?

Also: do you have anything specific set up in /etc/security/auditdistd.conf?

(edit)

In the mean time I set up the following (sparse) auditdistd.conf:

```
name "psi"

receiver {
        listen "10.0.1.6"
}
```
Then I tried to start the daemon within my jail and I encounter no problems at all. My jail is pretty standard, the only specific permissions which I've set are the raw sockets which I mentioned above as well as an addition for devfs.

So I'm definitely starting to think that the cause of your problem is the way you've set up your jail. Could you perhaps share /etc/jail.conf? And, if possible, also auditdistd.conf?


----------



## BrunolPhilippe (Apr 26, 2018)

Hello,

Thank for your response.

My configuration for jail and auditdistd:


```
<jail.conf>

# Global settings applied to all jails.

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

syslog {
    host.hostname = "myreceiver";
    path = "/mypath/myreceiver";
    interface = "em0";
    ip4.addr = 10.10.10.1/255.255.254.0;
    allow.raw_sockets;
    allow.socket_af;
}
```


```
<auditdistd.conf - receiver>

name "myreceiver"

receiver {

    listen "tls://10.10.10.5:7878"
    directory "/racine/myaudit"

    certfile "/etc/security/auditdistd.cert.pem"
    keyfile "/etc/security/auditdistd.key.pem"

    host "server1" {
        remote "tls://10.10.10.100"
        password "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
}
```


```
<auditdistd.conf - sender>

name "server1"

sender {
    host "myreceiver" {
        remote "tls://10.10.10.5:7878"
        fingerprint "SHA256=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        password "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
}
```


You can see my log when I launch the daemon with debug mode.


```
<log for auditdistd - receiver>

# /usr/sbin/auditdistd -d -F
(37175) [INFO] Started successfully.
(37175) [DEBUG][1] Accepting connection to tls://10.10.10.5:7878.
(37175) [INFO] Connection from tls://10.10.10.100 to tls://10.10.10.5.
(37805) [WARNING] [TLS sandbox] (server) Unable to jail to directory /var/empty: Operation not permitted.
(37805) [DEBUG][1] [TLS sandbox] (server) Privileges successfully dropped using capsicum+chroot+setgid+setuid.
(37805) [DEBUG][1] [TLS sandbox] (server) Privileges successfully dropped.
(37175) [DEBUG][1] Version 0 negotiated with tls://10.10.10.100.
(37175) [DEBUG][1] [server1] (receiver) Challenge generated.
(37175) [DEBUG][1] [server1] (receiver) Challenge sent.
(37175) [DEBUG][1] [server1] (receiver) Response received.
(37175) [DEBUG][1] [server1] (receiver) Hash generated.
(37175) [INFO] [server1] (receiver) Sender authenticated.
(37175) [DEBUG][1] [server1] (receiver) Challenge received.
(37175) [DEBUG][1] [server1] (receiver) Response generated.
(37175) [DEBUG][1] [server1] (receiver) Response sent.
(38167) [WARNING] [server1] (receiver) Unable to jail to directory /var/empty: Operation not permitted.
(38167) [ERROR] [server1] (receiver) Unable to change root directory to /var/empty: Operation not permitted.
(37805) [DEBUG][1] [TLS sandbox] (server) Connection terminated.
(37175) [ERROR] [server1] (receiver) Worker process exited ungracefully (pid=38167, exitcode=78).
(37175) [DEBUG][1] Sandbox process exited gracefully (pid=37805).
(37175) [INFO] Termination signal received, exiting.
```


In the source code of auditdistd (file sandbox.c), I find my error:


```
...
#ifdef HAVE_JAIL
    va_start(ap, fmt);
    (void)vasprintf(&jailhost, fmt, ap);
    va_end(ap);
    if (jailhost == NULL) {
        pjdlog_error("Unable to allocate memory for jail host name.");
        goto out;
    }
    bzero(&jailst, sizeof(jailst));
    jailst.version = JAIL_API_VERSION;
    jailst.path = pw->pw_dir;
    jailst.hostname = jailhost;
    if (jail(&jailst) >= 0) {
        jailed = true;
    } else {
        jailed = false;
        pjdlog_errno(LOG_WARNING,
            "Unable to jail to directory %s", pw->pw_dir);
    }
    free(jailhost);
#else    /* !HAVE_JAIL */
    jailed = false;
#endif    /* !HAVE_JAIL */

    if (!jailed) {
        if (chroot(pw->pw_dir) == -1) {
            pjdlog_errno(LOG_ERR,
                "Unable to change root directory to %s",
                pw->pw_dir);
            goto out;
        }
}
...
```


----------



## BrunolPhilippe (Apr 26, 2018)

So /var/empty is a homedir for the user named auditdistd. If I change the homedir (ex: /home/auditdistd), I obtain the same error with the new directory.


----------



## ShelLuser (Apr 26, 2018)

First: I like your motivation and you digging into the source code to find the relevant code. You don't see that often here 

Anyway, I did some experimenting and so far I can't reproduce any issues. There's one thing which hasn't been addressed yet: what version of FreeBSD are you using?

Right now I can only think of 2 possible reasons: something related to the FreeBSD version or something related to jails and encrypted connections. Perhaps a version mismatch between your main host and the jail?

The issue looks a bit weird to me so far.


----------



## BrunolPhilippe (Apr 26, 2018)

My receiver : FreeBSD 11.1-STABLE r330242
My sender : FreeBSD 11.1-RELEASE-p9

I have analyze the syscall with truss. It's interesting...


```
# truss -faedD -o /tmp/auditdistd.out /usr/sbin/auditdistd -d -F
```


The first error comes from here:


```
# view /tmp/auditdistd.out
...
31504: 7.958045283 0.000010005 jail(0x7fffffffe560) ERR#1 'Operation not permitted'
31504: 7.958075094 0.000008521 getpid()          = 31504 (0x7b10)
31504: 7.958129648 0.000012003 write(2,"(31504) ",8) = 8 (0x8)
31504: 7.958181273 0.000010942 write(2,"[WARNING]",9) = 9 (0x9)
31504: 7.958231747 0.000011307 write(2," [server1] (receiver) ",20) = 20 (0x14)
...
```


The second error comes from here:


```
...
31190: 7.513399111 0.000010226 jail(0x7fffffffe3e0) ERR#1 'Operation not permitted'
31190: 7.513431822 0.000008291 getpid()          = 31190 (0x79d6)
31190: 7.513485401 0.000013283 write(2,"(31190) ",8) = 8 (0x8)
31190: 7.513535043 0.000010213 write(2,"[WARNING]",9) = 9 (0x9)
31190: 7.513586550 0.000011720 write(2," [TLS sandbox] (server) ",24) = 24 (0x18)
31190: 7.513637854 0.000010623 write(2,"Unable to jail to directory /var"...,38) = 38 (0x26)
31190: 7.513688833 0.000010313 write(2,": Operation not permitted.",26) = 26 (0x1a)
31190: 7.513750713 0.000010794 write(2,"\n",1)   = 1 (0x1)
31190: 7.513804978 0.000013283 chroot("/var/empty") = 0 (0x0)
31190: 7.513854813 0.000009629 chdir("/")        = 0 (0x0)
...
```


In man page jail(2)(), the jail() system call will fail if [EPERM] because it would exceed the jail's children.max limit.
Indeed, the value of children.max is 0. for the jail.

I change my configuration of jail (add children.max = 10).

The first issue of syscall is closed:


```
...
90638: 9.083131948 0.000038880 jail(0x7fffffffe3e0) = 76 (0x4c)
90638: 9.083182235 0.000009438 chdir("/")        = 0 (0x0)
90638: 9.083221466 0.000010522 setgroups(0x1,0x801c3c000) = 0 (0x0)
90638: 9.083259587 0.000009527 setgid(0x4d)      = 0 (0x0)
90638: 9.083298097 0.000010778 setuid(0x4e)      = 0 (0x0)
90638: 9.083333628 0.000009542 cap_enter()       = 0 (0x0)
...
```


But I have two news errors with the second syscall jail() and the syscall chroot():


```
...
91035: 9.511539389 0.000036516 jail(0x7fffffffe560) ERR#1 'Operation not permitted'
91035: 9.511568955 0.000008206 getpid()          = 91035 (0x1639b)
91035: 9.511625020 0.000011811 write(2,"(91035) ",8) = 8 (0x8)
91035: 9.511684787 0.000010999 write(2,"[WARNING]",9) = 9 (0x9)
91035: 9.511734936 0.000010977 write(2," [server1] (receiver) ",20) = 20 (0x14)
91035: 9.511785035 0.000010735 write(2,"Unable to jail to directory /var"...,38) = 38 (0x26)
91035: 9.512020350 0.000010665 write(2,": Operation not permitted.",26) = 26 (0x1a)
91035: 9.512067023 0.000011165 write(2,"\n",1)   = 1 (0x1)
91035: 9.512118540 0.000011075 chroot("/var/empty") ERR#1 'Operation not permitted'
91035: 9.512149230 0.000008253 getpid()          = 91035 (0x1639b)
...
```


Strange...

Regards


----------



## BrunolPhilippe (May 1, 2018)

I have upgrading my server to FreeBSD 11.2. Now I have creating a new jail for my receiver configuration. and I obtain the same issue.


```
# /usr/sbin/auditdistd -d -F
(37175) [INFO] Started successfully.
(37175) [DEBUG][1] Accepting connection to tls://10.10.10.8:7878.
(37175) [INFO] Connection from tls://10.10.10.100 to tls://10.10.10.8.
(37805) [WARNING] [TLS sandbox] (server) Unable to jail to directory /var/empty: Operation not permitted.
(37805) [DEBUG][1] [TLS sandbox] (server) Privileges successfully dropped using capsicum+chroot+setgid+setuid.
(37805) [DEBUG][1] [TLS sandbox] (server) Privileges successfully dropped.
(37175) [DEBUG][1] Version 0 negotiated with tls://10.10.10.100.
(37175) [DEBUG][1] [server1] (receiver) Challenge generated.
(37175) [DEBUG][1] [server1] (receiver) Challenge sent.
(37175) [DEBUG][1] [server1] (receiver) Response received.
(37175) [DEBUG][1] [server1] (receiver) Hash generated.
(37175) [INFO] [server1] (receiver) Sender authenticated.
(37175) [DEBUG][1] [server1] (receiver) Challenge received.
(37175) [DEBUG][1] [server1] (receiver) Response generated.
(37175) [DEBUG][1] [server1] (receiver) Response sent.
(38167) [WARNING] [server1] (receiver) Unable to jail to directory /var/empty: Operation not permitted.
(38167) [ERROR] [server1] (receiver) Unable to change root directory to /var/empty: Operation not permitted.
(37805) [DEBUG][1] [TLS sandbox] (server) Connection terminated.
(37175) [ERROR] [server1] (receiver) Worker process exited ungracefully (pid=38167, exitcode=78).
(37175) [DEBUG][1] Sandbox process exited gracefully (pid=37805).
(37175) [INFO] Termination signal received, exiting.
```


I wonder if the auditdistd receiver can run in a jail.


Regards

Bruno


----------

