# How to prevent multiple jails to share the same memory objects?



## gzbk (Mar 11, 2016)

Hello all,

I've been quite surprised to see that by default (I'm using FreeBSD 10.1) all jails share their memory objects created through `shm_open()`(2) (even when the `security.jail.sysvipc_allowed` `sysctl` value is set to 0).

How do you prevent this so a jail cannot access the shared memory objects created in another jail?

Thank you by advance!


----------



## SirDice (Mar 11, 2016)

You can't.

From jail(8):

```
allow.sysvipc
                     A process within the jail has access to System V IPC
                     primitives.  [b]In the current jail implementation, System V
                     primitives share a single namespace across the host and
                     jail environments, meaning that processes within a jail
                     would be able to communicate with (and potentially inter-
                     fere with) processes outside of the jail, and in other
                     jails.[/b]
```


----------



## gzbk (Mar 11, 2016)

The initial issue I encountered was on a system were I tried to install a second Squid proxy in a new jail. It appeared that this second Squid proxy would not be able to start until its user was given the same UID as the first Squid in the other jail, otherwise it would systematically fail with the following error:


```
2016/03/10 10:53:13| storeDirWriteCleanLogs: Starting...
2016/03/10 10:53:13|  Finished.  Wrote 0 entries.
2016/03/10 10:53:13|  Took 0.00 seconds (  0.00 entries/sec).
FATAL: Ipc::Mem::Segment::create failed to shm_open(/squid-cf__metadata.shm): (17) File exists

Squid Cache (Version 3.5.15): Terminated abnormally.
CPU Usage: 0.026 seconds = 0.026 user + 0.000 sys
Maximum Resident Size: 48640 KB
Page faults with physical i/o: 0
```

After a few researches, I finally came up to this diagnose and made the crappy test program below:


```
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>

int main(int argc, char *argv[]){
  const int SIZE = 4096;
  char *str;
  char car[1];
  int len;
  int fd;

  if (argc != 2 && argc != 3) {
    fprintf(stderr, "Usage: client </shm-path>\n");
    return 1;
  }

  if (argc == 2) {
    fd = shm_open(argv[1], O_RDONLY, 0666);
    if (fd == -1) {
      fprintf(stderr, "shm_open() failed.");
      return 1;
    }
    str = mmap(NULL, SIZE, PROT_READ, MAP_SHARED, fd, 0);
    if (str == MAP_FAILED) {
      fprintf(stderr, "mmap() failed.");
      return 1;
    }
    fprintf(stdout, "%s\n", str);
  }
  else {
    len = strlen(argv[2]) + 1;
    if (len > SIZE) {
      fprintf(stderr, "String too long.");
      return 1;
    }
    fd = shm_open(argv[1], O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
    if (fd == -1) {
      fprintf(stderr, "shm_open() failed.");
      return 1;
    }
    if (ftruncate(fd, SIZE) == -1) {
      fprintf(stderr, "ftruncate() failed.");
      return 1;
    }
    str = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (str == MAP_FAILED) {
      fprintf(stderr, "mmap() failed.");
      return 1;
    }
    memcpy(str, argv[2], len);
    sleep(5);
    close(fd);
    shm_unlink(argv[2]);
  }
  return 0;
}
```

This program takes as first parameter the path to a shared memory object, and as an optional second parameter a string:


When both parameter are given, it will create the shared memory object, put the string in it and wait for 5 seconds,
When only the path is provided, it will attempt to open it and display the content of its memory.
When two instances of this program are run in two different jails, it allows a successfull communication between them: the string passed to one instance in the first jail is successfully displayed by the second instance running in a different jail.

SysVIP are disallowed in both jails:


```
$ hostname
squid0
$ sysctl -a | grep sysvipc
security.jail.sysvipc_allowed: 0
security.jail.param.allow.sysvipc: 0
```


```
$ hostname
squid1
$ sysctl -a | grep sysvipc
security.jail.sysvipc_allowed: 0
security.jail.param.allow.sysvipc: 0
```

And just for the fun a sample of my software used (again with `hostname` output to show the different jails):


```
$ hostname
squid0
$ ./client /foobar "Anything there?"
$
```


```
$ hostname
squid1
$ ./client /foobar
Anything there?$
```

It works.


----------



## gzbk (Mar 11, 2016)

SirDice said:


> You can't.



I'm not sure how to interpret your answer, did you mean "_You can't access other jails' shared memory objects when 
 sysvipc_allowed is set to 0_", as I was initially expecting, or did you mean "_You can't prevent this_", as answer to my question (which I would feel as quite a limitation affecting FreeBSD's jail system...)?


----------



## SirDice (Mar 11, 2016)

Although shared memory is sometimes used for IPC there is a difference between SysV IPC and SysV shared memory.  As I understood it the switch only deals with SysV IPC, not shared memory.

https://www.freebsd.org/doc/en/books/arch-handbook/jail-restrictions.html


----------



## gzbk (Mar 11, 2016)

OK, so it seems there is no way to prevent a jail from accessing / altering shared memory objects from other jails... no good, no good at all  ...


----------



## SirDice (Mar 11, 2016)

Yeah, in my view an additional switch might be good to have. Something like sysvshm_allow, or something similar, which could prevent access to shared memory.


----------



## gzbk (Mar 11, 2016)

Subsidiary question: is there a way to list currently created shared memory objects? (and no, `ipcs -a` does not list them)


----------



## gzbk (Mar 11, 2016)

I'm quite worried by the idea that on one side anyone with enough privileges within a jail can tamper all SHM objects system-wide and that there is no way to estimate the potential impact of this weakness since there is no way to actually list the SHM currently used on the system (I just know for now that Squid can be reached from any jail, yepee!). I expected better from my FreeBSD boxes  ...

I've found this sentence in a discussion from last year in FreeBSD mailing list:



> On DragonFly, SHM segments are always treated as files but on FreeBSD it depends on whether or not application is inside a jail.



Having SHM object represented as files within jails would perfectly make sense to me since it should avoid different jails to access SHM object associated to path they cannot access.

Did FreeBSD actually worked like that at some point in time? Why has this behavior been altered to become what it is now?


----------



## gzbk (Mar 11, 2016)

The only way I've found yet to detect SHM objects is to look for open file descriptors using a command such as:


```
fstat | awk '{print $5}' | sort -u
```

It appears that while the jails have access to all SHM objects, they can only list their own ones (through this method at least, maybe there are other! And in all cases an attacker reaching this step will probably have no problem to scan for all common SHM objects names and their derivative in a very short amount of time).

I've tested with FreeBSD 9.3, the issue is the same, I therefore really wonder if this claim of a different behavior inside jails is justified and if it was not just like that since 7.0.

Does anyone know if there any chance that this situation will evolve in some future FreeBSD release? Or should SHM based software be avoided on FreeBSD platforms?


----------



## Purkuapas (Mar 11, 2016)

Hi *gzbk*

I've also interested in fix these issues https://forums.freebsd.org/threads/55404/ including SHM. Unfortunately, it seems to have a depressing picture with number of FreeBSD developers. The best option for you - use *bhyve* ( https://www.freebsd.org/cgi/man.cgi?query=bhyve&sektion=8 ) instead of jail. Unless of course you do not make a patch to improve jail by himself.


----------



## wblock@ (Mar 11, 2016)

Please send this question to the freebsd-jails mailing list.  There are more jail users there.


----------



## gzbk (Mar 12, 2016)

wblock@ said:


> Please send this question to the freebsd-jails mailing list.  There are more jail users there.



Done.


----------



## robroy (Mar 15, 2016)

gzbk, I listened to Bryan Cantrill's recent Papers We Love:  Jails and Zones yesterday, and this specific Jail characteristic was discussed, among other things.


----------



## gzbk (Mar 17, 2016)

robroy said:


> gzbk, I listened to Bryan Cantrill's recent Papers We Love:  Jails and Zones yesterday, and this specific Jail characteristic was discussed, among other things.



Hi Robroy,

Thanks for the reference. The issue I describe is actually even worse than the SysV IPC issue described in this video. SysV IPC are disabled by default in Jail, and I think you can enable them on a jail-per-jail basis. What Bryan Cantrill denounces is that there is an implicit mutual trust between SysV IPC enable jail which should not exist.

But in the current issue, the fact is that there is _*no*_ way to disable SHM objects sharing and accesses between jail. Unlike SysV IPC, you cannot tell this Jail needs SHM objects so I will enable the feature for this one, and this other jail is Internet facing and therefore less trustable so I will disable SHM objects for it. _*Each and every jail*_ can create, read and modify SHM objects from the whole system and there is just no way to prevent it:


Each jail can create an uncontrollable and unblockable communication channel with each other jail,
If you are running a Squid proxy, each and every jail can read and modify Squid's SHM objects,
Etc.

By the way, following James Gritton's suggestion in the mailing list discussion, I've filled the bug 208082 to follow this issue. At this time when Linux is catching up in the realm of lightweight system partitioning, I sincerely hope that FreeBSD will make some progress on such issues. I'm a FreeBSD enthusiast (freshly BSDA certified !), I want to keep some diversity on my systems and use a FreeBSD wherever I can, but not if this means opening long standing security holes while an equivalent and free solution provides the same feature with better security and compatibility.


----------

