# Setting up a (Debian) Linux jail on FreeBSD



## ShelLuser (Dec 3, 2018)

Hi gang!

_Prerequisites_: I am assuming that you know how to install software on FreeBSD and also have some basic understanding about FreeBSD jails.

*Editorial*

I'm going to be very honest here: I started disliking Linux for a while, and I've worked with it for a long time. For various reasons, but one of them being that it has turned into a commercial company driven project (you _do_ realize that RedHat and Canonical (= company behind Ubuntu) are enterprise sized companies, right?). Arguments? Simple: systemd. A piece of shit software service which goes directly against the Unix philosophy and worse yet: makes Linux pretty much incompatible with others (you'll see later on). Isn't it annoying that because of projects like systemd (and several other changes) many people can't keep up with those developments anymore and need extra study or training? How convenient that RedHat also happens to be one of the bigger companies which provides Linux training! And no: not for free of course  There's a good reason why IBM bought them.

_However..._

Despite my dislike I'm also the first to acknowledge all the effort that goes into those projects. And although I definitely have my (bias) ideas about stuff such as systemd (and the distributions using it) fact of the matter is that many don't have a choice. Also: many projects are still the inclusive distributions they always were. Debian still gives life to many derivatives for example. And well, even Linux can be somewhat fun 

*1 - Linux compatibility*
FreeBSD actually has a Linux compatibility layer which allows it to run Linux (ELF) binaries without a problem. Keep in mind though that this isn't full proof, but many things can be made to work without issues. You're going to need 2 things: support in the kernel and a userland to provide optionally required libraries (the latter isn't needed for the jail but I'd still install it anyway in case you're also going to use this setup for more purposes).

*Kernel modules*
If you run a GENERIC kernel then you already have everything you need. If you run a more customized kernel (like I do) then you'll need to make sure that it supports these kernel modules:

linux.ko & linux64.ko; Speaks for itself, right?
linux_common.ko; This one should also be an obvious one.
linprocfs.ko; Although FreeBSD doesn't use /proc filesystem by default you can set this up if you want to. Linux on the other hand _needs_ /proc to be present and to make matters worse it'll also work somewhat differently than FreeBSD does things. This module will take care of that.
linsysfs.ko; When you build a FreeBSD base environment you'll end up with /usr/obj/usr/src which basically contains the binary structure which you build. On FreeBSD you can remove this if you want but Linux actually keeps ties into those binaries & libraries whenever you build the kernel. On Linux /sys often links to /usr/src/linux/sys (from the top of my head, I could be slightly off but the argument is fully correct). So how to cope with that? Well, that's what this module is for, it'll simulate the effect.
linuxkpi.ko; As far as I know this one simulates the *K*ernel *P*rogramming *I*nterface, but I have no idea what this exactly does. However, when I check /usr/src/sys/modules/linuxkpi/Makefile then my theory quickly becomes that this module provides access to the Linux hardware layer. So if a program tries to access USB, PCI or even a Linux kernel module then this is the FreeBSD kernel module which handles all that.
fdescfs.ko; A file descriptor provides a method for a program to communicate with the OS. Good examples are stdin, stdout and stderr. And you guessed it: there is a difference between Linux & FreeBSD which is what this kernel module will solve.
tmpfs.ko; Chances are high that your system already uses this but because it's still a requirement I'm listing it nonetheless. tmpfs is basically what used to be a ramdrive on DOS & Windows: a space in memory which is reserved to be used as a temporary filesystem. This is often used to provide /tmp and/or /var/tmp.
You don't have to worry about loading all these modules yourself, just let the system handle that by adding:

```
linux_enable="YES"
```
... to your /etc/rc.conf file. And there's also the issue of FreeBSD being able to dynamically load kernel modules whenever it needs one.

*Userland*
Linux binaries often have specific dependencies on libraries (and maybe other binaries) in which case you'll need some kind of userland which is what emulators/linux_base-c7 can provide. This will install a CentOS userland within the /compat/linux directory structure. Don't worry, it won't be a resource hog; on my system the (uncompressed) ZFS filesystem uses up around 254Mb.

_Congratulations, you are now Linux compatible! _

*2 - Adding a full Linux userland*
But which Linux? See, there is a problem...

As you may (or should!) know a Jail is nothing more but a FreeBSD userland which gets started by the kernel. You can even make it 'do' stuff by having the kernel initialize your rc.d structure: `sh /etc/rc` (see also ports(7)). On Linux we'd normally have /etc/init.d/rc at our disposal but guess what? systemd didn't only take over the init process, _nooo_ that wasn't good enough: it's also spreading its tentacles into other areas such as the booting structure and mounting options itself. On most Linux environments /etc/fstab is simply a systemd emulation.

And although the FreeBSD Linux compatibility layer can do _a lot_, it's not perfect. Crapola like systemd doesn't properly run on it for example (which I actually consider a pro ).

Fortunately there are still plenty of people who grasp and still honor the ideology which Linux once stood for and one of those projects is Devuan. What's Devuan? It's a Debian derivative which does not include systemd so we'll be fully able to use this as a Jail just fine.

So why all this interest for Debian even though we already have a CentOS userland you ask? Two reasons: Although /compat/linux does indeed provide a userland it's not complete. It was set up to emulate, not fully simulate. So don't expect an init.d structure.

But the second reason is much cooler...  See: Debian's package system is, in my humble opinion, superior by design (within the context of Linux!). It's really neatly set up and unlike RPM it's even doable (though still a bit of a drag) to maintain your own packages. Something which you might want to do if you prefer to build your own software. Debian's packages are provided as both binaries and source, how convenient is _that_?

And they didn't stop there..  Eventually they created debootstrap which does just what the name implies: it will grab all the packages needed for a base system and set that up. So to get to the finale of this exciting build up: it has even been included in the FreeBSD ports collection as: sysutils/debootstrap, now _that_ is cool I think. So quickly install this critter because we're going to need it!

*Bootstrapping Devuan*
If you're using ZFS then I _strongly_ suggest to set up a dedicated filesystem. At the very least this will help you to keep track of the space your jail(s) are consuming:

```
zfpeter@zefiris:/home/peter $ zfs list -r zroot/opt/jails
NAME                     USED  AVAIL  REFER  MOUNTPOINT
zroot/opt/jails         1.32G  88.4M   162M  /opt/jails
zroot/opt/jails/devuan   262M  28.9G   262M  /opt/jails/devuan
zroot/opt/jails/psi      928M  28.9G   928M  /opt/jails/psi
```
This can also help for security measures, but that's beyond the scope of this guide. So, as seen above we'll be using /opt/jails/devuan in my examples.

Devuan is deviously clever   See; every Debian distribution has a specific name. Debian's latest (at the time of writing!) is stretch whereas Devuan is called ascii. Now, the "problem" is that debootstrap uses scripts of the same name to help it separate between the distributions. And ascii isn't the same as stretch. Fortunately for us Devuan also honors the Debian standard distribution names: stable, testing and unstable. And unlike on FreeBSD Stable is honestly just that 

_Sidenote:_ Yes, I am aware that you can also specify an individual script when using debootstrap, but I like to keep things as simple as possible.

*Important:* We're going to perform the bootstrap process in 2 stages. Why? debootstrap can set up the hierarchy, but it _won't_ be able to utilize things such as linprocfs or linsysfs, so we're going to have to set that up ourselves.

```
root@zefiris:/home/peter # debootstrap --foreign --arch=amd64 stable /opt/jails/devuan http://deb.devuan.org/merged/
W: Cannot check Release signature; keyring file not available /usr/share/keyrings/debian-archive-keyring.gpg
I: Retrieving InRelease
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
```
C'mon, doesn't this make your geekish powerlevels raise to beyond Super Saiyan levels?  I dub this the devu-ascii-ha, but don't worry: yelling is not required 

Anyway, this will process quite a list. What is happening is that debootstrap is retrieving all the packages that make up the Devuan base system and places those in our jail directory after which it'll extract them.

*Jail preparations*
Depending on your system this is probably going to take a while so now would be a good time to start preparing our upcoming jail by setting up our upcoming special Linux filesystems. Create a file called /etc/fstab.devuan and add the following:

```
$ cat /etc/fstab.devuan                                                         
## Mountpoint(s) for the Devuan jail
# Dev   Mountpoint      FS              Options         Dump / Check

linprocfs       /opt/jails/devuan/proc  linprocfs       rw,late 0 0
linsysfs        /opt/jails/devuan/sys   linsysfs        rw,late 0 0
tmpfs           /opt/jails/devuan/tmp   tmpfs   rw,late,mode=1777 0 0
```
If you're using ZFS like I do then you're going to need late because otherwise these filesystems will get mounted before ZFS is ready (in which case only your root filesystem would be available). Otherwise you obviously don't have to worry.

Around this time the first stage of debootstrap should be ready and I urge you to look around in your new Devuan environment. Points of interest:

./debootstrap; this is where the native 'binary' got placed, but also where you'll find debootstrap.log (as created by 'our' version) which shows you exactly what the system did.
./etc/init.d/rc; no systemd crapola for us! This is the key to booting our upcoming jail.
./root; I share a lot of criticism towards Linux so I also think it's important to give credit where credit's due: setting /root to 700 is in my opinion a solid option. No intruders, especially not during these early stages.
./var/cache/apt/archives; and this is where our new software collection resides 
*The finishing touch*
Since we're here anyway we can now mount some required directories for the next stage:

```
root@zefiris:/opt/jails/devuan # mount -F /etc/fstab.devuan `pwd`/sys
root@zefiris:/opt/jails/devuan # mount -F /etc/fstab.devuan `pwd`/proc
root@zefiris:/opt/jails/devuan # mount -F /etc/fstab.devuan `pwd`/tmp
root@zefiris:/opt/jails/devuan # mount -t devfs none dev
root@zefiris:/opt/jails/devuan # chroot . /bin/bash
I have no name!@zefiris:/#
```
So now we've "started" what I'd like to call the "shadow jail". Unfortunately we can't use debootstrap again to perform the second stage because it'll try to create devices (= assumption on my end) which isn't supported. And don't be fooled: despite having a bit of a userland we actually got nothing, the only thing debootstrap did was extract some packages. But we _also_ want those packages to get registered so that we can fully utilize dpkg:

```
I have no name!@zefiris:/# dpkg --force-depends -i /var/cache/apt/archives/*.deb
Selecting previously unselected package adduser.
(Reading database ... 0 files and directories currently installed.)
Preparing to unpack .../archives/adduser_3.115_all.deb ...
Unpacking adduser (3.115) ...
Selecting previously unselected package apt-utils.
Preparing to unpack .../apt-utils_1.4.8_amd64.deb ...
Unpacking apt-utils (1.4.8) ...
```
This will take a while and you'll see plenty of warnings pass your screen, you can safely ignore those for now. Another advantage here is that packages don't only get installed, they get _configured_ as well. Get ready to specify your timezone.

*Inconsistent (but usable) state*
Now, you're probably going to end up with a few error messages. As I mentioned earlier Linux compatibility isn't perfect:

```
Setting up sysvinit-core (2.88dsf-59.9+devuan2) ...
cp: preserving permissions for '/etc/inittab': No data available
dpkg: error processing package sysvinit-core (--install):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
base-files
debianutils
libc-bin
readline-common
sysvinit-core
```
But keep well in mind: just because there were errors doesn't mean the package didn't got installed:

```
I have no name!@zefiris:/# dpkg -l base-files
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
iF  base-files     9.9+devuan2. all          Devuan base system miscellaneous
I have no name!@zefiris:/#
```
This package is somewhat alright, but make sure to run: `# dpkg --configure --pending`, this will sort out other left over issues. Now normally you'll only end up with 2 errors and there's little we can do about them:

```
Setting up sysvinit-core (2.88dsf-59.9+devuan2) ...
sysvinit: creating /run/initctl
mv: cannot move '/dev/initctl.new' to '/dev/initctl': Operation not supported
dpkg: error processing package sysvinit-core (--configure):
subprocess installed post-installation script returned error exit status 1
Setting up base-files (9.9+devuan2.5) ...
cp: preserving permissions for '/root/.profile': No data available
dpkg: error processing package base-files (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
sysvinit-core
base-files
```
There really isn't much which we can do here but all in all the 'damage' isn't _too_ bad:

```
I have no name!@zefiris:~# dpkg -l | grep -v ii
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version                            Architecture Description
+++-=============================-==================================-============-========================================================================
iF  base-files                    9.9+devuan2.5                      all          Devuan base system miscellaneous files
rF  sysvinit-core                 2.88dsf-59.9+devuan2               amd64        System-V-like init utilities
```
See? Only 2 packages which have a problem. However: the r status on sysvinit isn't good, try and force a re-install if you see this: `# dpkg --force-all -i /var/cache/apt/archives/sysvinit-core*`. The status you want for _both_ packages is iF. Because once you have that then you'll still run into warnings whenever you install another package (during its configure stage) but your installation _won't_ fail because of it.

Speaking of which... do you know why a FreeBSD base system is far superior than a Devuan Linux base system?

```
I have no name!@zefiris:~# man dpkg
bash: man: command not found
```
However, you can fix this by running: `# apt-get install man-db`. This will also be a good test to see if your system is in a usable status. Keep in mind that building the database of manualpages is going to take a while. Another con of Linux in my opinion: it creates the database immediately when you install the package instead of leaving it up to the user to decide when (I forgot about this detail and now it messes up the timing for my guide ).

*3 - Setting up the jail*
So now that we have our Devuan environment setup it's time to actually use it within FreeBSD. Edit /etc/jail.conf and add this section:

```
devuan {
        host.hostname = "devuan.jail";
        interface = lo0;
        ip4.addr = 127.0.0.5;
        path = /opt/jails/devuan;
        exec.start = "/etc/init.d/rc 3";
        exec.stop = "/etc/init.d/rc 0";
        persist;

        mount.devfs;
        mount.fstab = /etc/fstab.devuan;

        allow.mount;
        allow.mount.devfs;
}
```
You can do this in two ways. Instead of lo0 you could also use your public network interface and assign a 'real' IP address, I used this approach with my Psi jail because it made things easier on me. However, I don't trust Linux anymore (especially after all those details which surfaced about Ubuntu adding numerous "phone home" options, all opt-out obviously), so _no way_ that I'll allow it network access "just like that". Ergo: setup on localhost which means that you'll need to set up a NAT solution on your firewall which will allow your Linux jail access to your network.

My reasoning is simple: if you really need quick access then you can always rely on chroot for now, just as I did earlier.

When this is done then all you have to do is fire up the jail: `# jail -c devuan`, and to get onto the console: `# jexec devuan /bin/bash`.

*4 - Summing up*

Set up Linux compatibility. Usually you only have to add: `linux_enable="YES"` to /etc/rc.conf.
Although optional it is advisable to install emulators/linux_base-c7 as well.

Optionally set up a dedicated section for your jail. When using ZFS then creating a new dedicated filesystem is definitely a good idea.
Install sysutils/debootstrap.
Bootstrap your system: `# debootstrap --foreign --arch=amd64 stable /path/to/jail http://deb.devuan.org/merged/`
Obviously replace amd64 for i386 on 32bit machines!

Mount linux file systems, for best results add these to a dedicated "jail fstab" like /etc/fstab.devuan.
`# mount -t linprocfs none /path/to/jail/proc`
`# mount -t linsysfs none /path/to/jail/sys`
`# mount -t tmpfs none /path/to/jail/tmp`

Enter your new system to finish up:
`# chroot /path/to/jail /bin/bash`
`# dpkg --force-depends -i /var/cache/apt/archives/*.deb`
`# dpkg --configure --pending`

Make sure that only base-files and sysvinit-core are partially configured: `# dpkg -l | grep -v ^ii`.
Their status should be iF.
If there is a problem try to forcefully reinstall the package: `# dpkg --force-all -i /path/to/package`.

Check that everything works by installing a package: `# apt-get install man-db`.
Set up your jail by adding the following to /etc/jail.conf:


```
devuan {
        host.hostname = "devuan.jail";
        interface = lo0;
        ip4.addr = 127.0.0.5;
        path = /path/to/jail;
        exec.start = "/etc/init.d/rc 3";
        exec.stop = "/etc/init.d/rc 0";
        persist;

        mount.devfs;
        mount.fstab = /etc/fstab.devuan;

        allow.mount;
        allow.mount.devfs;
}
```

Start your new jail using: `# jail -c devuan`.
Access the console using `# jexec devuan /bin/bash`.
Enjoy!
_And there you have it...._

Best of both worlds, what's there not to like?


----------



## humphrayLegare (Feb 3, 2019)

Thank you very much ! this tutorial is 100% exactly what i was looking for !


----------



## heraldo (Feb 13, 2019)

Thank you very much!!


----------



## zirias@ (Apr 21, 2019)

Two things I found out:

1. to allow the linux userland to allocate PTYs, load the pty(4) driver (needed e.g. for running an ssh daemon in the jail)

2. some scripts fail because `cp -p` isn't working inside the jail. Quick and dirty workaround: replace /bin/cp in the jail with /rescue/cp from the FreeBSD base.

With these changes, I managed to fully configure all devuan packages


----------



## ercdude (Apr 23, 2019)

Hey all. 

Thanks for this thread, it helps me to understand better how chroot and jails works, but unfortunately I couldn't set it up. With Devian, the problem is when I tries to use apt-get, it always throws me an error about cache:

`E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-Start. Current value: 25165824. (man 5 apt.conf)`

I'm able to use `apt-get ` under chroot? Or should I use only at jails? (I made it before with lots of warnings, but works. don't know why I cant anymore :/)

I've tried the same with debian. It runs well, i reach almost the same step, but at the end it does not find `/etc/init.d/rc`. Hoping that anyone could help me. Anyone knows how to proceed?

tnks!


----------



## zirias@ (Apr 23, 2019)

ercdude said:


> With Devian, the problem is when I tries to use apt-get, it always throws me an error about cache:
> 
> `E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-Start. Current value: 25165824. (man 5 apt.conf)`


Devian? Uhm, Debian or Devuan? Anyways, `apt` running out of cache space is probably not related to it running in a chroot or jail. The messsage shows your cache size starts (!) at 25MB, which should be more than enough for a "normal" list of repositories. So, did you add a lot of stuff to your /etc/apt/sources.list? If so, just follow the advice of the error message (you might have to create an apt.conf file yourself). If not, something on your system might be corrupted, and you might for example try to clean out /var/lib/apt/lists/ and do `apt-get update` to fetch the lists again. Most probably this has all nothing to do with FreeBSD.



ercdude said:


> I'm able to use `apt-get ` under chroot? Or should I use only at jails?


Shouldn't matter either. It's just safer to do stuff in a jail than chroot, chroot only protects from accessing files outside the tree, nothing else. E.g. I accidentally shut down my machine running some command in a chroot with a devuan userland -- this wouldn't be possible when running in a jail.


----------



## ercdude (May 2, 2019)

Zirias said:


> Devian? Uhm, Debian or Devuan?



Debian, lol. The last try, at least.



Zirias said:


> So, did you add a lot of stuff to your /etc/apt/sources.list? If so, just follow the advice of the error message (you might have to create an apt.conf file yourself).



I didn't see that `man 5 apt.conf` at the end. Anw, I changed it like you say and it's moving on... Now my problem is with apt-get, seems that it doesn't have a signed key (?) and I can't install openssh-server for some reason... anw I'll try to solve that, but everything seems to work good now.

Thanks )


----------



## zirias@ (May 3, 2019)

ercdude said:


> my problem is with apt-get, seems that it doesn't have a signed key (?)


Install the appropriate keyring package (e.g. `apt-get install devuan-keyring --allow-unauthenticated` on devuan) to solve this. Again, this has nothing to do with FreeBSD, just basic usage of signed APT repositories.


----------



## Lamia (Jun 26, 2019)

Thanks Zirias.
I have followed your guide and another I found online.
I can't use the jail. Here is the error I get:

```
# dpkg -l | grep -v ^ii                                        
Desired=Unknown/Install/Remove/Purge/Hold                                     
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)                    
||/ Name                          Version                            Architecture Description
+++-=============================-==================================-============-========================================================================    
iF  base-files                    9.9+devuan2.5                      all      
  Devuan base system miscellaneous files
```

Edited: Not technically an error. But that is why my installation hangs. I have tried several options but no luck.


----------



## Lamia (Jun 27, 2019)

This command "debootstrap --foreign --arch=amd64 stable /opt/jails/devuan http://deb.devuan.org/merged/" ends here:

```
I: Validating tzdata 2019a-0+deb9u1
I: Validating util-linux 2.29.2-1+devuan2.1
I: Validating vim-common 2:8.0.0197-4+deb9u1
I: Validating vim-tiny 2:8.0.0197-4+deb9u1
I: Validating wget 1.18-5+deb9u3
I: Validating whiptail 0.52.19-1+b1
I: Validating xxd 2:8.0.0197-4+deb9u1
I: Validating zlib1g 1:1.2.8.dfsg-5
I: Chosen extractor for .deb packages: ar
```


----------



## Lamia (Jun 28, 2019)

Lamia said:


> This command "debootstrap --foreign --arch=amd64 stable /opt/jails/devuan http://deb.devuan.org/merged/" ends here:
> 
> ```
> I: Validating tzdata 2019a-0+deb9u1
> ...


Problem with file system(zfs). I have fixed it; so simple.


----------



## Lamia (Jun 29, 2019)

Can anyone please tell me how to start the jail?

```
# jail -c devuan
devuan: created
jail: devuan: getpwnam: No such file or directory
jail: devuan: /etc/init.d/rc 3: failed
devuan: removed

# ezjail-admin start devuan
Starting jails:/etc/rc.d/jail: WARNING: /var/run/jail.devuan.conf is created and used for jail devuan.
 cannot start jail  "devuan":
10
jail: devuan: getpwnam root: No such file or directory
jail: devuan: /etc/init.d/rc 3: failed
.
/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables  is obsolete.  Please consider migrating to /etc/jail.conf.
Error: Could not start servdevuan.
  You need to start it by hand.
```
I prefer using ezjail for now though.


----------



## JohnnySorocil (Aug 12, 2019)

Lamia said:


> Can anyone please tell me how to start the jail?
> 
> ```
> # jail -c devuan
> ...



I had the same issue. My understanding is that passwd database is missing.
Using command from https://blog.protocolsyntax.com/2017/06/09/debian-7-wheezy-installation-in-freebsd-10-jail/ fixed problem for me.

```
cat /usr/jails/linux/etc/passwd | sed -r 's/(:[x|*]:)([0-9]+:[0-9]+:)/:*:\2:0:0:/g' > /usr/jails/linux/etc/master.passwd
pwd_mkdb -p -d /usr/jails/linux/etc /usr/jails/linux/etc/master.passwd
```


----------



## twllnbrck (Aug 13, 2019)

I use sysutils/iocage and have no problems to start the devuan jail.


----------



## Lamia (Aug 14, 2019)

I can start and get into it now but now shows this error:

```
Get:11 http://pkgmaster.devuan.org/merged ascii/main amd64 libglib2.0-data all 2.50.3-2 [2517 kB]
Get:12 http://pkgmaster.devuan.org/merged ascii/main amd64 shared-mime-info amd64 1.8-1+deb9u1 [731 kB]
Get:13 http://pkgmaster.devuan.org/merged ascii/main amd64 xdg-user-dirs amd64 0.15-2+b1 [52.2 kB]
Get:14 http://pkgmaster.devuan.org/merged ascii/main amd64 xml-core all 0.17 [23.2 kB]
Fetched 15.9 MB in 20s (763 kB/s)     
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
Setting up base-files (9.9+devuan2.5) ...
rmdir: failed to remove '/var/run': Directory not empty
dpkg: error processing package base-files (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 base-files
E: Sub-process /usr/bin/dpkg returned an error code (1)
```

It comes up for every apt-get upgrade/install.

I have this fstab config:

```
$ cat /etc/fstab.devuan                                                         
## Mountpoint(s) for the Devuan jail
# Dev   Mountpoint      FS              Options         Dump / Check

linprocfs       /opt/jails/devuan/proc  linprocfs       rw,late 0 0
linsysfs        /opt/jails/devuan/sys   linsysfs        rw,late 0 0
tmpfs           /opt/jails/devuan/tmp   tmpfs   rw,late,mode=1777 0 0
```

I do not see so much about this error 
	
	



```
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
Setting up base-files (9.9+devuan2.5) ...
rmdir: failed to remove '/var/run': Directory not empty
```
 online.


----------



## Alain De Vos (Sep 4, 2019)

debootstrap --foreign --arch=i386 stable /opt/jails/devuan http://deb.devuan.org/merged/
First steps are fine,
# dpkg -l  | grep -v ii
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version                            Architecture Description
+++-=============================-==================================-============-========================================================================
iF  base-files                    9.9+devuan2.5                      all          Devuan base system miscellaneous files
iF  sysvinit-core                 2.88dsf-59.9+devuan2               amd64        System-V-like init utilities

>But then,
# apt-get install man-db
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package man-db

Starting the jail and logging in works but no networking.
But i cannot ping it.
ping: socket: Protocol not supported

Yet in sysctl i have,
security.jail.allow_raw_sockets=1

First of all i need to get ping local address work.
ifconfig lo1
lo1: flags=4169<UP,LOOPBACK,RUNNING,MULTICAST>  mtu 16384
        inet 192.168.2.77  netmask 255.255.255.255

netstat -rn
/proc/net/route: No such file or directory
INET (IPv4) not configured in this system.

route add default gw 192.168.1.1 lo1
SIOCADDRT: Invalid argument

Bummer!


----------



## Alain De Vos (Sep 5, 2019)

I seem to have better results with oldstable :
debootstrap --foreign --arch=i386 oldstable /myjail http://deb.devuan.org/merged/
Yet still no networking


----------



## userxbw (Sep 7, 2019)

that's intense, but have you tired slackware, no systemD. void no systemD...


----------



## Alain De Vos (Sep 7, 2019)

I might try other "2.6.32 kernel" distributions. I took the centos DVD and extracted all packages, now  /compat/linux/usr/bin contains 5688 files. wget works fine. ping does not work because of "linux capabilities".
iocage should also allow to install a linux distribution but it depends on zfs.
Someone should do something with ezjail for linux ...


----------



## ShelLuser (Sep 7, 2019)

Keep in mind that the kernel has 0 influence here.


----------



## ko56 (Sep 20, 2019)

ShelLuser said:


> So now we've "started" what I'd like to call the "shadow jail". Unfortunately we can't use debootstrap again to perform the second stage because it'll try to create devices (= assumption on my end) which isn't supported. And don't be fooled: despite having a bit of a userland we actually got nothing, the only thing debootstrap did was extract some packages. But we _also_ want those packages to get registered so that we can fully utilize dpkg:



Thanks for this very informative post.  I am in the middle of an installation of Devuan. Has anyone actually tried to use phase 2 of debootstrap at this point?


----------



## Alain De Vos (Sep 20, 2019)

void linux could also  be interesting because it also does not have systemd. But I don't know if it is maintained.


----------



## userxbw (Sep 21, 2019)

Last time I head that dude that developed it, he came back from wherever he disappeared to. I too think he was a *BSD developer  and that Void is created off some of that idealism. From what the write up on it says. I have used it, and its a nice distro. 

There support is on FreeNode... (hexchat) mostly.


----------



## Vadim_Mkk (Sep 30, 2019)

I tried to install Devuan  Stable (ASCII = Debian 9.9) and Devuan Testing (Beowulf = Debian 10) to my FreeBSD 12-p10
Devuan Stable (ASCII) installed and started  without problem, how I understand SysVinit Stable (ASCII) compatible with Linux kernel 2.6.
Devuan Stable (ASCII) installed  without problem, but didn't starting and write error message "Kernel to old" how I understand   Devuan Testing compatible with Linux kernel 2.8.
The earlier Linuxator version contained uname command, which show Linux kernel version for Linuxator, but  this command dissaperared.
But other hand Doclker and other Linux container virtualization system  haven't newest images for Debian (for example versions Debian - 7, 8) and others linux distros.


----------



## Vadim_Mkk (Sep 30, 2019)

My previous post has one error ( mistake ctl+c and ctl+v) - Devuan Testing Beowulf installed but didn't started - error message Kernel to old


----------



## aragats (Nov 5, 2019)

How to deal with /dev/pts?
When using `jexec` in Linux jail I have:
	
	



```
# ll /dev/pts/
total 1
dr-xr-xr-x 2 root root    512 Nov  5 14:01 .
dr-xr-xr-x 9 root root    512 Nov  5 14:01 ..
crw--w---- 1 1001 adm  136, 3 Nov  5 15:18 3
```
So, the 3 corresponds to my current terminal, but when logging in with `ssh` no PTY can be allocated:
	
	



```
% ssh -t mylinuxjail
PTY allocation request failed on channel 0
```


----------



## cabriofahrer (Dec 11, 2019)

Does this devuan installed in a jail provide graphics/2D/3D-Acceleration and sound? I am asking if this could be a solution for watching Netflix and for playing games through the linux-steam-client?


----------



## shkhln (Dec 11, 2019)

What you don't like about our current Steam solution?


----------



## cabriofahrer (Dec 12, 2019)

shkhln said:


> What you don't like about our current Steam solution?



I just asked a generic question, also including the Netflix-Issue. I did not want to say that there is something about the current steam solution that I don't like. I have had no time to do further experiments since the last time I posted on the other thread, but the last status for me was that not even all source-games I tried would work. But I do appreciate and support very much all possible solutions and developments on the matter, of course.
But I also asked the question because independently from steam, I would just like to know if 2D/3D works in a jail, for I have never worked with jails and I am just curious.


----------



## shkhln (Dec 12, 2019)

Well, and the answer is "No". Linuxulator provides a very different environment from the Linux kernel, which requires quite a bit of package customization to work properly. Trying to run a standard Linux distribution in a jail makes everything harder, not easier.

It also pays to remember that 100% Linux compatibility is _not_ the goal for Linuxulator, since that pretty much requires converting FreeBSD into a Linux reimplementation with all their architectural choices and whatnot.


----------



## cabriofahrer (Dec 12, 2019)

Thank you then, that's all I wanted to know!


----------



## shkhln (Dec 12, 2019)

ShelLuser said:


> linuxkpi.ko; As far as I know this one simulates the *K*ernel *P*rogramming *I*nterface, but I have no idea what this exactly does. However, when I check /usr/src/sys/modules/linuxkpi/Makefile then my theory quickly becomes that this module provides access to the Linux hardware layer. So if a program tries to access USB, PCI or even a Linux kernel module then this is the FreeBSD kernel module which handles all that.



You might want to correct that part. Linuxkpi simply provides some struct/constant definitions and helper functions to ease the process of porting the kernel drivers from Linux. Specifically intel/amd graphics and, I think, some network drivers. Linuxkpi doesn't expose any APIs to userspace applications. It is also fully independent from the Linux emulation code.

As for the USB support, see linux_libusb. I haven't tried it myself (yet), but anything else definitely won't work.


----------



## UrsusDominatus (Dec 17, 2019)

Hi all.
Can you help me with Devuan jail + vnet setup. I have several ordinary FreeBSD jails configured with vnet, bridge and epair. Following through tutorial I ended up with functional jail, but I cannot setup network in it. ifconfig and ip commands inside jail produce output like:

```
root@devuan:/# ifconfig epair15b 192.168.10.15 netmask 255.255.255.0
SIOCSIFADDR: Invalid argument
SIOCSIFFLAGS: Invalid argument
SIOCSIFNETMASK: Invalid argument

root@devuan:/# ifconfig eth0 192.168.10.15 netmask 255.255.255.0
SIOCSIFADDR: Invalid argument
SIOCSIFFLAGS: Invalid argument
SIOCSIFNETMASK: Invalid argument
```


```
root@devuan:/# ifconfig -a
eth0: flags=4162<BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 02:d3:4b:71:3c:0b  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo0: flags=4104<LOOPBACK,MULTICAST>  mtu 16384
        loop  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```


```
root@devuan:/# ip a
Cannot open netlink socket: Address family not supported by protocol
```

Jail config is:

```
devuan {
    host.hostname = "devuan.example.com";
    $ip = 15;
    mount.fstab = "/usr/local/jails/devuan.fstab";

    exec.prestart = "ifconfig epair$ip create";
    exec.prestart += "ifconfig bridge0 addm epair${ip}a";
    exec.prestart += "ifconfig epair${ip}a up";

    exec.poststop = "ifconfig bridge0 deletem epair${ip}a";
    exec.poststop += "ifconfig epair${ip}a destroy";

    exec.start = "/etc/init.d/rc 3";
    exec.stop = "/etc/init.d/rc 0";
    exec.clean;

    vnet.interface = "epair${ip}b";
    vnet = "new";

    persist;
    mount.devfs;
    allow.mount;
    allow.mount.devfs;
}
```
Probably I am doing some stupid configuration mistake, but I am zero in linux, and slightly more than that in FreeBSD.


----------



## chris123 (Dec 21, 2019)

UrsusDominatus said:


> Hi all.
> Can you help me with Devuan jail + vnet setup. I have several ordinary FreeBSD jails configured with vnet, bridge and epair. Following through tutorial I ended up with functional jail, but I cannot setup network in it. ifconfig and ip commands inside jail produce output like:
> 
> ```
> ...


I'm struggling with the same thing: https://www.ixsystems.com/community/threads/debian-devuan-linux-jail-network-access-problems.80920 . Using Iocage and FreeNAS.


----------



## dch (Dec 31, 2019)

shkhln said:


> What you don't like about our current Steam solution?



what steam solution would this be? I'd like to see if I can get XCom2 running on FreeBSD (with nvidia gpu) as the little linux box I'm currently using gets nastily hot.


----------



## shkhln (Dec 31, 2019)

dch said:


> what steam solution would this be? I'd like to see if I can get XCom2 running on FreeBSD (with nvidia gpu) as the little linux box I'm currently using gets nastily hot.



See Thread 72140.


----------



## macosxgeek (Jan 15, 2020)

Very good post. Thank you ShelLuser!

OK. So I did all the steps and I'm trying to start the jail now with:

`jail -c devuan`

Unfortunately I'm getting the following error message:



```
jail: devuan: mount.devfs: /path/to/jail/dev: No such file or directory
```

Any ideas what can be possibly wrong?


----------



## gilby (Jan 15, 2020)

I am wondering if this allows the use of hardware that is not supported in FreeBSD. I have an Nvidia GPU to use with Tensorflow, and instead of putting Linux as the OS on this server, I would love to get it running on FreeBSD, with a Linux jail just for this.


----------



## shkhln (Jan 15, 2020)

gilby said:


> I am wondering if this allows the use of hardware that is not supported in FreeBSD.



It doesn't. Jails are not magic.


----------



## gilby (Jan 15, 2020)

I guess we'll have to learn how to use Linux.


----------



## SirDice (Jan 16, 2020)

gilby said:


> I am wondering if this allows the use of hardware that is not supported in FreeBSD.


Jails run on the host's kernel. You may have a Linux userland, it still runs on top of a FreeBSD kernel. So, no.


----------



## Vadim_Mkk (Jan 16, 2020)

Tried setup Devuan Jail on FreeBSD 12.1 with Linix base 7.7 1908
Linuxator "uname" command  shows kernel version 2.6.XX and after executes command "dpkg --force-depends -i /var/cache/apt/archives/*.deb" appeares message  "libc6 2.24-11+deb9u4 requre kernel above 3.2" - after this messages I canceled to set up this Jail because that follow form the official description "glibc - the core libraries for the GNU system and GNU/Linux systems, as well as many other systems that use Linux as the kernel. These libraries provide critical APIs including ISO C11, POSIX.1-2008, BSD, OS-specific APIs and more. These APIs include such foundational facilities as open, read, write, malloc, printf, getaddrinfo, dlopen, pthread_create, crypt, login, exit and more" that mean what this jail must have system mistake on basic level. May be this Devuan Jail witn kernel 2.6.XX may be correct work with Devuan Jessy/oldstable libc6 2.19-18+deb8u10 - but I don't have enough free time for adventurous and creative experiments with software of 2014 year  I know that older wine, rum or cognac - that more more refined and sophictical its taste and aroma.  But software isn't rich alcohol. 
Void Linux randomly  and suddenly freezes - solve only reboot. Devuan stable  older  than Debian stable  minimum two years - for example zfs on Devuan 0.6.5 native and  0.7.12 backpots  vs Deiban native  0.7.12 and  backpots 0.8.2. Gentoo is very time consumption compilation from sources for ephemeral performance 1% - 2%  burst. Where to go to the poor peasant? Where looking for disto with out systemd that will correctly work with Anaconda Distribution?
How wrote one poet -  The plastic world won, the dummy was stronger 
Bazar whops  cathedral 
P.S. But Devuan without systemd correct works with removable media and network manager


----------



## shkhln (Jan 16, 2020)

There is a reason Linux jails are completely unsupported and it's not ease of use…


----------



## Vadim_Mkk (Jan 16, 2020)

Only true believer Debian Stable  help me! 
In the near days I will shove  the Anaconda Distribution into the Debian LXC container.
ZFS 0.8.2 works perferct in Debian 10 buster, well compresses data without lost performance IO.
It's poorly that Anaconda doesn't start in the FreeBSD - but what me to do?


----------



## Vadim_Mkk (Jan 27, 2020)

Vadim_Mkk said:


> "libc6 2.24-11+deb9u4 requre kernel above 3.2"



Really,  If something doesn’t work correctly - read mindfully man, the devil in the details  and everything new is the very well forgotten old 


> Only if installing Squeeze, change the compatibility level declared 2.6.18.
> 
> ```
> root@morsa:/root #  echo 'compat.linux.osrelease=2.6.18' >> /etc/sysctl.conf
> ```



In additon devildetail clears  sysvinit in the Jail


> 9. Inside the jail, delete the configuration files sysvinit_*.
> 
> ```
> root@morsa:/jailz/etc # rm /jailz/deb-master/var/cache/apt/archives/sysvinit_*
> ```


and rebuilding base system with the different command

```
I have no name!@morsa:/# dpkg --force-depends -Ei /var/cache/apt/archives/*.deb
```

I'll try to upgrade the kernel version above  3.2  and see what happens...


----------



## SirDice (Jan 27, 2020)

Vadim_Mkk said:


> I'll try to upgrade the kernel version above 3.2 and see what happens...


Jails don't have a kernel, not a FreeBSD kernel and certainly not a Linux kernel.


----------



## Vadim_Mkk (Jan 27, 2020)

How I think when downloading  application from  debian repo to devuan/debian jail debbootstrap translates in jail defailt "linux kernel  version" that show "uname" from liunuxator directory...
Why  linux() contains this words?


> The following sysctl(8) tunable variables are available:
> compat.linux.osname       Linux kernel operating system name.
> compat.linux.osrelease    Linux kernel operating system release.  Changing this    to something else is discouraged on non-development systems, because  it may change the way Linux programs work.  Recent versions of GNU libc are known to  use different syscalls depending on the   value of this sysctl.


----------



## Vadim_Mkk (Feb 12, 2020)

ShelLuser said:


> Best of both worlds, what's there not to like?


ShelLuser which FreeBSD version you use?
I tried run Devuan  in the FreeBSD  11.3 jail.
I configure chroot environment  before configure jail
`mvg@freebsd:/opt/jails/devuan # mount -F /etc/fstab.devuan `pwd`/s
mvg@freebsd:/opt/jails/devuan # mount -F /etc/fstab.devuan `pwd`/proc
mvg@freebsd:/opt/jails/devuan # mount -F /etc/fstab.devuan `pwd`/tmp
mvg@freebsd:/opt/jails/devuan # mount -t devfs none dev
mvg@freebsd:/opt/jails/devuan # chroot . /bin/bash`
When I entered chroot and run
`dpkg --force-depends -i /var/cache/apt/archives/*.deb`
I got next error

```
Setting up sysvinit-core (2.88dsf-59.9+devuan2) ...
sysvinit: creating /run/initctl
mv: cannot move '/dev/initctl.new' to '/dev/initctl': Operation not supported
dpkg: error processing package sysvinit-core (--configure):
subprocess installed post-installation script returned error exit status 1
```
after this errors I while stayed chroot can find `apt search` install packages with  `apt install` with the permanent error about sysvinit-core when I went out chroot and started Devuan Jail I couldn't run `apt search` or `apt install` because  I had permanent network error. Didn't help adding to the jail.conf  *allow.raw_sockets* - network error stayed. If I didn't go to the chroot - then when I run jail `dpkg --force-depends -i /var/cache/apt/archives/*.deb` and I didn't get any errors after executing `dpkg -l | grep -v ii` but I had permanent network error and couldn't start  `apt search` or `apt install`.
In both cases, the result is one - deadlock 
FreeNAS has similar problem


----------



## Vadim_Mkk (Feb 12, 2020)

SirDice said:


> Jails don't have a kernel, not a FreeBSD kernel and certainly not a Linux kernel.


I understand it, but when I changed version Linux for 4.19 - disappeared error about libc6 2.24-11+deb9u - seem when run `dpkg --force-depends -i /var/cache/apt/archives/*.deb` it query formal version Linux kernel.
When I add to jail.conf *linux - *I get correct version Devuan kernel 4.9.хх


----------



## Vadim_Mkk (Feb 13, 2020)

If you need mostly to use  Linux software that doesn't ported FreeBSD - easy way to use native  Debian Linux without any software crooked nails, braces and the big time consumption.
I don't want use the   linux-c7-7.7.1908 port - because I agree with M.W Lucas  -  "Be warned, though: once you install something outside of the Ports Collection, you’ll need to maintain it by hand"©
Offensively that Devuan doesn't take off in the FreeBSD Jail - I haven't a free time  for experiments with not predictable results. Good luck to everybody, I gone to learn Debian and Anaconda Distribution.
I hope that FreeBSD sooner or later  to make working jails with Linux - for example Devuan or Void.
Buy,,,


----------



## shkhln (Feb 13, 2020)

gpb said:


> Doing this they translate system calls from one API/ABI to another and emulate the Linux system call table.



I hate to disappoint you, but that is how Linuxulator works in the first place.



Vadim_Mkk said:


> If you need mostly to use  Linux software that doesn't ported FreeBSD - easy way to use native  Debian Linux without any software crooked nails, braces and the big time consumption.



That's just wrong.


----------



## Vadim_Mkk (Feb 13, 2020)

shkhln said:


> That's just wrong.


Why I need to jump with a tambourine and spend a lot of time on it is not clear and not predictable results   if  I can run needed me applications on another OS without  those problems?
Time is the very expensive and non-renewable resource.
The hell with systemd , 20M lines code - but I have what I need. I work with laptop, no heavy loaded server. Something like this


----------



## shkhln (Feb 13, 2020)

Oh, I thought you were talking about Debian _jail_…


----------



## Vadim_Mkk (Feb 13, 2020)

My target was to run Anaconda distribution on FreeBSD in the  Jevuan jail.  This attempts  failed and  this experiments over.


----------



## Vadim_Mkk (Feb 13, 2020)

shkhln said:


> about Debian _jail_…


Debian buster (10)   and  bullseye (11) have  sysvinit-core and sysvinit-utils packages together with systemd


----------



## shkhln (Feb 13, 2020)

Can you describe the difference in non-FUD language please?


----------



## shkhln (Feb 14, 2020)

Looks this was copy-pasted from the (obsolete) Oracle documentation, otherwise the text would mention ability to run 64-bit Linux binaries, which I presume Illumos supports. In any case, this mostly describes container management and not emulation itself.


----------



## shkhln (Feb 14, 2020)

Why are we going through this exercise anyway? I'm just curious how many posts it would take you to admit that there exist only one (!) way of emulating the Linux kernel. That is, implementing it's public API, which means implementing Linux syscalls.


----------



## shkhln (Feb 14, 2020)

Nobody ever claimed FreeBSD jails have the same management capabilities, that's a straw man argument. The Linux emulation, however, is quite comparable. I wouldn't be surprised if Linuxulator runs circles around Illumos' implementation with regard to the completeness/compatibility.



gpb said:


> But it clearly explains exactly what branded zones are, which is what you asked for.



I didn't.


----------



## Vadim_Mkk (Feb 21, 2020)

A little off topic for the respect community  because I don't want make new thread..
How I understand hasn't insurance that Debian Jail will work correct with Linux application.
The little question - will work Nvida CUDA  with the Linux applicatons in the current linuxator with driver's for  FreeBSD version higher than  4XX,XX ?
Because in the description linux-nvidia-libs-440.31 I read one sententious here


> This makes amd64 Linux OpenGL programs work on FreeBSD 12.0 and recent 11-STABLE.  This does not enable CUDA.


For example can I start TensorFlow 2 on Linuxator? Or if I tried to setup TensorFlow 2 in Linuxator I spent for nothing my time?
P.S I want install Spyder in FreeBSD  through default pip python3.7 - this operation  finished a huge quantity of errors.


----------



## shkhln (Feb 21, 2020)

CUDA is still broken. Nobody is actively working on fixing it, so it will stay broken for the foreseeable future.


----------



## Vadim_Mkk (Feb 21, 2020)

Sorry, then I  have to staying on the  Debian...
Good luck everybody, I switch to reading  mode..But I take with me the ZFS - I liked it very much.
Biggest RedHat, Oracle and other IT giants with bllianary R&D   budgets (i begin suspect that they sawing money and simulation innovation)  can't do it better that ZFS file systems over last 14 years. Although also ZFS isn't full  ideal - but full ideal  never reachable.Bye bye all..


----------



## shkhln (Feb 22, 2020)

Vadim_Mkk said:


> Sorry, then I  have to staying on the  Debian...



Am I supposed to feel bad about it or something? Hire a kernel developer to work on CUDA or convince the foundation to sponsor such project.


----------



## Vadim_Mkk (Feb 22, 2020)

Why such this sounds? Let's get along Dostoevsky passions in this case?
Is task -is tools for its solution.


----------



## Aruns (Jun 15, 2020)

I closely followed all step to set up a Linux jail on FreeBSD 11.3. The final step results this,

root@freebsd-3M5C:/opt/jails/devuan # jail -c devuan
devuan: created
ELF binary type "0" not known.
ELF binary type "0" not known.
jail: devuan: exec /etc/init.d/rc: Exec format error
jail: devuan: /etc/init.d/rc 3: failed
devuan: removed

am looking for a fix.


----------



## mast07 (Jun 15, 2020)

A similar error was solved in this thread:








						Solved - java: Exec format error
					

[~]$ java -version ELF binary type "0" not known. exec: /usr/local/linux-oracle-jdk1.8.0/bin/java: Exec format error  [~]$ kldstat Id Refs Address            Size     Name  1   35 0xffffffff80200000 1fa7c38  kernel  2    1 0xffffffff821a9000 30aec0   zfs.ko  3    2 0xffffffff824b4000 adc0...




					forums.freebsd.org
				




Perhaps it provides also some help here...


----------



## Lamia (Jun 15, 2020)

You will easily install Devuan via vm-bhyve using the debian template.


----------



## Aruns (Jun 16, 2020)

mast07 said:


> A similar error was solved in this thread:
> 
> 
> 
> ...



 Thanks it's worked.


----------



## Deleted member 63539 (Aug 10, 2020)

Here is my fstab.antix, please note I mounted more file systems than you for my AntiX to work happily:


```
linprocfs       /antix/proc  linprocfs       rw,late 0 0
linsysfs        /antix/sys   linsysfs        rw,late 0 0
tmpfs           /antix/tmp   tmpfs   rw,late,mode=1777 0 0
tmpfs           /antix/run   tmpfs   rw,late,mode=1777 0 0
/home/username  /antix/home/username nullfs rw,late 0 0
```

Note: /antix is my AntiX jail's root. I created inside a jail a user with the same name, id and gid as a user I have on the host to use nullfs() to share the user's home dir.


----------



## ziomario (Nov 21, 2021)

```
I have no name!@marietto:/# apt-get update

Get:1 http://deb.devuan.org/merged stable InRelease [33.9 kB]
Get:2 http://deb.devuan.org/merged stable/main amd64 Packages [8309 kB]
Get:3 http://deb.devuan.org/merged stable/main Translation-en [6482 kB]
Fetched 14.8 MB in 1min 3s (236 kB/s)                                   
E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-St
art. Current value: 25165824. (man 5 apt.conf)
Reading package lists... Error!
E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-St
art. Current value: 25165824. (man 5 apt.conf)
E: Error occurred while processing php-symfony-messenger (NewVersion2)
E: Problem with MergeList /var/lib/apt/lists/deb.devuan.org_merged_dists_s
table_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.
```


----------



## TrOuBLe (Nov 25, 2021)

ziomario said:


> ```
> I have no name!@marietto:/# apt-get update
> 
> Get:1 http://deb.devuan.org/merged stable InRelease [33.9 kB]
> ...




```
echo "APT::Cache-Start 251658240;" > path/to/the/jail/etc/apt/apt.conf.d/00aptitude
```

Solved my issue


----------



## ziomario (Nov 25, 2021)

TrOuBLe said:


> ```
> echo "APT::Cache-Start 251658240;" > path/to/the/jail/etc/apt/apt.conf.d/00aptitude
> ```
> 
> Solved my issue



very thanks,it worked like a charm.


----------



## ThisIsMask (Feb 5, 2022)

I'm trying this and other flavors (Ubuntu, Alpine, Arch) however, all of them don't have /etc/init.d/rc script to be used in jail configuration (including Debian):


```
exec.start = "/etc/init.d/rc 3";
    exec.stop = "/etc/init.d/rc 0";
```

Errors:


```
Starting jails: cannot start jail  "Test-Jail":
4
jail: Test-Jail: exec /etc/init.d/rc: No such file or directory
jail: Test-Jail: /etc/init.d/rc 3: failed
```

I think there was similar ask in beginning of this post but still don't have solution. Would anyone please share your recommendation or did I miss certain step? TIA


----------



## ziomario (Feb 5, 2022)

BaoNT said:


> I'm trying this and other flavors (Ubuntu, Alpine, Arch) however, all of them don't have /etc/init.d/rc script to be used in jail configuration (including Debian):
> 
> 
> ```
> ...



If I don't get wrong,you should use Devuan.


----------



## Lamia (Feb 5, 2022)

This should do the trick - https://wiki.freebsd.org/LinuxJails . It works for debian variants - devuan, ubuntu - and centos  (Red Hat and its variants).


----------



## Lamia (Feb 5, 2022)

Lamia said:


> This should do the trick - https://wiki.freebsd.org/LinuxJails . It works for debian variants - devuan, ubuntu - and centos  (Red Hat and its variants).


You will need make slight changes depending on your choice variant e.g. for repos.


----------



## TrOuBLe (Feb 5, 2022)

BaoNT said:


> I'm trying this and other flavors (Ubuntu, Alpine, Arch) however, all of them don't have /etc/init.d/rc script to be used in jail configuration (including Debian):
> 
> 
> ```
> ...



Don't expect too much from Linux over FreeBSD, it has a large gap on the Network level.
I was able to install Ubuntu 18 and lower version and Debian as well in jail.
To boot up the jail you need to call the system to load in a different manner below my jail.conf

```
bionic {
    path = /jails/${name}/base;
    host.hostname = "${name}.jail";
        interface = lo0;
        ip4.addr = 127.0.0.5;
    exec.clean;
    exec.system_user = "root";
    exec.jail_user = "root";
    exec.consolelog = /jails/${name}/.jail_cfg/console.log;
    exec.prestart = "cp /etc/resolv.conf $path/etc";
    exec.poststop = "rm $path/etc/resolv.conf";
    exec.start = "/bin/true";
    exec.stop = "/bin/true";
        persist;
        mount.devfs;
        mount.fstab = "/jails/${name}/.jail_cfg/fstab";
        allow.mount;
        allow.mount.devfs;
    allow.chflags;
        allow.raw_sockets;
};
```

Following the documentations:
https://wiki.freebsd.org/LinuxJails


----------



## ziomario (Feb 5, 2022)

you can also give a look at CBSD : https://www.bsdstore.ru/en/about.html


----------



## TrOuBLe (Feb 5, 2022)

If you will face user login issue you must do the following


```
# cd /my/jail/path/etc
# echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
 # pwd_mkdb -d ./ -p master.passwd
pwd_mkdb: warning, unknown root shell
```

In case you are not able to use *apt *do the following after booting the Linux jail, inside the deb jail:

```
sudo adduser _apt --force-badname
```

Mainly this is what I have faced like issues with Linux Over FreeBSD...
Let us know if you have generated different issues, I am interested to update my docs in that topic.


----------



## ThisIsMask (Feb 6, 2022)

This gets me one step nearer. I could start/stop/restart jail but I'm having issue with network.


```
/ # ip a
ip: socket(AF_NETLINK,3,0): Address family not supported by protocol
```

I'm still researching around to understand more about jail networking. In normal jail (bsdinstall) alias just works but not in this case.



TrOuBLe said:


> Don't expect too much from Linux over FreeBSD, it has a large gap on the Network level.
> I was able to install Ubuntu 18 and lower version and Debian as well in jail.
> To boot up the jail you need to call the system to load in a different manner below my jail.conf
> 
> ...


----------



## ziomario (Feb 6, 2022)

Im trying to do the same,but with bastille,following this tutorial :






						Create an Ubuntu Linux jail on FreeBSD 12.2
					

Create a jailed Ubuntu Linux with Bastille on FreeBSD 12.2 Make sure Bastille is installed and configured. click here This is not offically supported, so don’t use it in production! update 2021/02/11: Auto Installer You could try the following POC for automated jail creation: DO NOT USE THIS IN...



					hackacad.net
				




but I'm failing :


```
@marietto:/usr/home/marietto/Desktop/Scripts/bastille # jexec ubuntu /bin/bash


@ubuntu:/# apt update


Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree    
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolvi
ng 'archive.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.


@ubuntu:/# ping google.it
ping: socket: Protocol not supported

@ubuntu:/# exitundefined
```

linux jails works well 1/10. This method is not good at all. Since I'm not a developer,I can do only easy tasks. So I'm thinking to do something like this :


a small linux vm virtualized with bhyve that can communicate with the freebsd host. this vm will have the minimum features requested by docker and nothing else.
this vm will be detached from the background,just like the virtualbox driver,that it is installed,hidden,but u can't do anything with it,at least graphically speaking
some components of docker will run inside the vm,some others on freebsd and they communicate sharing files and features as much as they can.
this minimum vm can be used also for a lot of other tasks and it should give to us the linux calls that are missing in the linuxulator.


----------



## Lamia (Feb 9, 2022)

I am also recommending this guide - https://wiki.freebsd.org/JailingGUIApplications.


----------



## Lamia (Feb 9, 2022)

Looks like I have already mentioned that guide before. My bad!


----------



## ziomario (Feb 9, 2022)

I'm not using ZFS. do u have the same guide but for ufs ? I want to try to run google chrome inside the jail (I never been able to make it work)


----------



## SirDice (Feb 9, 2022)

ziomario said:


> I'm not using ZFS. do u have the same guide but for ufs ?


There's very little in this howto that actually depends on ZFS.


----------



## ziomario (Feb 9, 2022)

ok but if I can't issue the ZFS commands,I'm not able to complete it.


----------



## SirDice (Feb 9, 2022)

ziomario said:


> ok but if I can't issue the ZFS commands,I'm not able to complete it.


What ZFS commands? There are no ZFS commands in this howto.

And instead of blindly copy/pasting commands, actually think about and understand what each step does. Then you can just as easily apply this howto to UFS.


----------



## ziomario (Feb 9, 2022)

Its full of zfs commmands :


```
The commands given are all executed as root, on the host, unless explicitly stated differently.

Toggle line numbers    1 zfs create -o compress=lz4 -o atime=off zroot/jails
   2 zfs create zroot/jails/basejail
   3 bsdinstall jail /zroot/jails/basejail


Here we used the bsdinstall method for convenience. Otherwise downloading and unpacking base.txz and configuring it should suffice. At this point we'd configure the basejail for pkg, like the location of your Poudriere repo if you have it. No other configuration is required for the base as the jails will basically run a single process.

So we snapshot it and create our Firefox jail filesystem:

Toggle line numbers    1 zfs snapshot zroot/jails/basejail@latest
   2 zfs create zroot/jails/firefox
   3 zfs clone zroot/jails/basejail@latest zroot/jails/firefox/root
   4 zfs create zroot/jails/firefox/var
   5 zfs create zroot/jails/firefox/tmp
   6 zfs create zroot/jails/firefox/home
   7 rsync -a /zroot/jails/firefox/root/var/ /zroot/jails/firefox/var/
   8 zfs set mountpoint=/zroot/jails/firefox/root/var zroot/jails/firefox/var
   9 zfs set mountpoint=/zroot/jails/firefox/root/tmp zroot/jails/firefox/tmp
  10 zfs set mountpoint=/zroot/jails/firefox/root/usr/home zroot/jails/firefox/home


For extra security we want our jail to run with minimum require privilege, so we set some properties on these datasets, which should make obvious why we separated them like this. Of course, these rules are not applicable to every application, as some, unfortunately would like to write or execute to/from paths they shouldn't. For firefox, these suffice, tho'.

Toggle line numbers

   1 zfs set setuid=off exec=off zroot/jails/firefox/var
   2 zfs set setuid=off exec=off zroot/jails/firefox/tmp
   3 zfs set setuid=off exec=off zroot/jails/firefox/home


At this point it's worth observing that when base is to be update, all we need to do is update the basejail and create a new snapshot for cloning. With that, and separate var/home/tmp dirs, it's trivial to update the jails' bases, just zfs destroy root dataset and re-clone it from basejail. This will require unmounting and re-mounting the other datasets, but it can all be easily scripted for simple maintenance.

Next, with the filesystem in place, we install the packages. xauth and firefox are the base minimum, while liberation-fonts-ttf is recommended addition for some nice fonts in Firefox.

Toggle line numbers    1 pkg -c /zroot/jails/firefox/root install firefox xauth liberation-fonts-ttf
```

don't worry,that I dont copy and paste the commands blindly. I learn what I need,but I don't want to learn everything. If inside a tutorial there are parts that I want to learn later or one day,I will skip them. I don't want to be overloaded with too much informations. Otherwise it stops to be an hobby and it becomes a job. But I'm not tailored for this as a job. It can be only an hobby. In this specific case,Its hard for me to skip the zfs commands because If I do it,I don't know or it becomes hard to  replace the skipped commands with the ufs commands. When I started learning FreeBSD I made a choice : to start with UFS,to make things easier,at the beginning. I've thought that if I had chosen zfs I would have overloaded myself with useful informations for sure,but not so essential,at least for a newbie,at the beginning.


----------



## SirDice (Feb 9, 2022)

That's from a completely different howto?


----------



## ziomario (Feb 9, 2022)

SirDice said:


> That's from a completely different howto?



what do u mean ? I'm referring to Lamia tutorial,because I haven't been able to configure correctly the linux jail using this tutorial :






						Create an Ubuntu Linux jail on FreeBSD 12.2
					

Create a jailed Ubuntu Linux with Bastille on FreeBSD 12.2 Make sure Bastille is installed and configured. click here This is not offically supported, so don’t use it in production! update 2021/02/11: Auto Installer You could try the following POC for automated jail creation: DO NOT USE THIS IN...



					hackacad.net
				




so I'm open to try new tutorials.


----------



## SirDice (Feb 9, 2022)

ziomario said:


> I'm referring to @Lamia  tutorial,


You were asking it in a thread of a different howto.


That's it. I'm getting tired of howto's being hijacked with unrelated questions. I'm going to remove any and all responses from all howtos, and lock those threads down so only the original howto author can respond.


----------



## ziomario (Feb 10, 2022)

SirDice said:


> You were asking it in a thread of a different howto.
> 
> 
> That's it. I'm getting tired of howto's being hijacked with unrelated questions. I'm going to remove any and all responses from all howtos, and lock those threads down so only the original howto author can respond.



that's a bad choice because usually happens that the original author does not respond anymore maybe because some time is passed and he forgot or because he has no time to reproduce the situation because he is involved in a totally different job and task. Your choice is good to stop the circulation of experiments and ideas.


----------



## Lamia (Feb 10, 2022)

I did not get LinuxGUI-Jailed using one guide. And many other people would say the same. And contents of the wiki.freebsd.org can be difficult to "decipher" as several developers writing articles on similar or related topics. I shared links of such related topic here - GUI Apps in a FreeBSD Jail, LinuxGUIJailed & Linuxator. I may not be able to provide the links here now. But they are very related. I can only point at these to you and several others written outside FreeBSD platforms - forum, wiki, bugzilla, etc. 

Some may be dated yet valuable enough to provide the fix. Being a moderated forum, the onus is on the admins to decide what is approved and posted.


----------



## TrOuBLe (Feb 16, 2022)

I was away from your discussions...
In short, what I have learned from Linux jail and from different documentation is that we can not CHANGE or MODIFY anything on the network level because LinuxJail is just using an emulator to run over Freebsd Kernel which is totally different from Linux kernel. The ethernet interface and its configuration are just inherited from the Freebsd HOST.
I would appreciate any experienced Freebsd admin to correct my info.


----------



## ziomario (Feb 16, 2022)

TrOuBLe said:


> I was away from your discussions...
> In short, what I have learned from Linux jail and from different documentation is that we can not CHANGE or MODIFY anything on the network level because LinuxJail is just using an emulator to run over Freebsd Kernel which is totally different from Linux kernel. The ethernet interface and its configuration are just inherited from the Freebsd HOST.
> I would appreciate any experienced Freebsd admin to correct my info.



I know this. But I see that there are a lot of gui applications that needs a network stack. I suppose that they want a linux network stack and since they find a freebsd network stack,they don't work. But I read somewhere that there is a method to create a network stack inside a real linux  chroot. The method maybe could be applied also inside the linux jail.


----------



## TrOuBLe (Feb 17, 2022)

ziomario said:


> I know this. But I see that there are a lot of gui applications that needs a network stack. I suppose that they want a linux network stack and since they find a freebsd network stack,they don't work. But I read somewhere that there is a method to create a network stack inside a real linux  chroot. The method maybe could be applied also inside the linux jail.


Well the only way to make those apps works is to create a Virtual Network Interface inside the Jail.
I have switched to another strategy with Linux, the better way is to build the Linux app in FreeBSD.


----------



## ziomario (Feb 17, 2022)

TrOuBLe said:


> Well the only way to make those apps works is to create a Virtual Network Interface inside the Jail.
> I have switched to another strategy with Linux, the better way is to build the Linux app in FreeBSD.



give me some documentation. But if it is too much technical I can't do that. I suppose I should follow the more abstract method to build a virtual network interface inside the jail,because I'm not a programmer.


----------



## ziomario (Feb 19, 2022)

I don't understand why I always get the error "*Can't start system message bus - /proc is not mounted ... failed!*" because :

1) I have added the argument :


```
linprocfs       /compat/devuan/proc  linprocfs    rw            0 0
```

to the file /etc/fstab.devuan on FreeBSD

2) I've mounted proc with this command :


```
mount -t linprocfs none /compat/devuan/proc
```

I don't get errors,but when I create the devuan jail, I always see the error that proc is not mounted :


```
@marietto:/usr/home/marietto # jail -c devuan

 devuan: created
Using makefile-style concurrent boot in runlevel 3.
dmesg: read kernel buffer failed: Operation not permitted
Starting enhanced syslogd: rsyslogdmknod: /dev/xconsole: Operation not supported
chown: cannot access '/dev/xconsole': No such file or directory
rsyslogd: cannot create '/dev/log': Operation not supported [v8.2102.0 try https://www.rsyslog.com/e
/2176 ]
rsyslogd: imuxsock does not run because we could not aquire any socket  [v8.2102.0]
rsyslogd: activation of module imuxsock failed [v8.2102.0]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): No such file or directory.
rsyslogd: activation of module imklog failed [v8.2102.0 try https://www.rsyslog.com/e/2145 ]
```


*----> Can't start system message bus - /proc is not mounted ... failed!*


```
Starting periodic command scheduler: cron.
Starting session management daemon: elogind.
```


----------



## sody (Feb 19, 2022)

I have the same errors all the time
the downside is /var/log/messages never populated because of this.
has anyone found a solution / alternative to this rsyslogd service?

tried to add to /etc/default/devfs.rules under jail ruleset (4) the following, reboot with no luck:
add path log unhide

might it be related to allow.socket_af ? i tried to set on /boot/loader.conf
security.jail.param.allow.socket_af=1

but it still "0" for some reason, maybe should give a try to jail params but not sure this would work.

Edit: tried jail -mr allow.socket_af=1but that didnt change it either.. has anyone found a solution to this?

Sody


----------



## ziomario (Feb 20, 2022)

sody said:


> I have the same errors all the time
> the downside is /var/log/messages never populated because of this.
> has anyone found a solution / alternative to this rsyslogd service?
> 
> ...



you also get the errors below ?


```
/compat/devuan/bin/chrome


[21224:102845:0218/234112.132650:ERROR:file_path_watcher_linux.cc(321)] inotify_init() failed: Function not implemented (38)

(chrome:21224): Gtk-WARNING **: 23:41:13.905: Unknown key gtk-applications-prefer-dark-theme in /root/.config/gtk-3.0/settings.ini

Gtk-Message: 23:41:14.184: Failed to load module "colorreload-gtk-module"

Gtk-Message: 23:41:14.185: Failed to load module "window-decorations-gtk-module"

[21224:102854:0218/234114.947955:ERROR:bus.cc(397)] Failed to connect to the bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

[21224:102854:0218/234114.963025:ERROR:bus.cc(397)] Failed to connect to the bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

[21224:102846:0218/234115.770664:ERROR:address_tracker_linux.cc(196)] Could not create NETLINK socket: Address family not supported by protocol (97)

[21224:102853:0218/234115.795018:ERROR:bus.cc(397)] Failed to connect to the bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

[21224:102853:0218/234115.795076:ERROR:bus.cc(397)] Failed to connect to the bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

[21224:102846:0218/234116.707297:ERROR:udev_watcher.cc(52)] Failed to initialize a udev monitor.

[21233:102862:0218/234117.094025:ERROR:address_tracker_linux.cc(196)] Could not create NETLINK socket: Address family not supported by protocol (97)

[21233:102864:0218/234117.094035:ERROR:file_path_watcher_linux.cc(321)] inotify_init() failed: Function not implemented (38)

[21224:21224:0218/234117.188257:ERROR:process_singleton_posix.cc(1100)] Failed to bind() /tmp/.com.google.Chrome.Pbohb3/SingletonSocket: No such file or directory (2)

[21224:21224:0218/234117.216008:ERROR:chrome_browser_main.cc(1305)] Failed to create a ProcessSingleton for your profile directory. This means that running multiple instances would start multiple browser processes rather than opening a new window in the existing process. Aborting now to avoid profile corruption.
```


----------



## sody (Feb 20, 2022)

ziomario said:


> you also get the errors below ?
> 
> 
> ```
> ...


I dont use jail for chrome so never tried interactive apps rather than server ones


----------



## ziomario (Feb 20, 2022)

sody said:


> I dont use jail for chrome so never tried interactive apps rather than server ones



which method do u use to run chrome ?


----------



## sody (Feb 22, 2022)

ziomario said:


> which method do u use to run chrome ?


I don't 

I mainly want to run apache httpd, php, mysql, exim, dovecot, bind,pure-ftpd services
so far all work but dovecot, bind,pure-ftpd. I sent email about bind (named) to the jail lists I hope someone wpuld be able to help.

Sami


----------

