# Running Google Chrome in a dedicated Linux-Jail



## Holger (Jun 12, 2022)

*Introduction and motivation*

There are great articles ([1] and [2]) by *patovm04* here on the forum explaining how to run Chrome and Brave in a Linux chroot environment (usually `/compat/linux` or `/compat/ubuntu`).

These approaches work great. However, I am a big fan of FreeBSD's jails and it has always bothered me that these Linux compatibility layers were some kind of “sort-of” jail, but not real ones. By “real one” I mean a lightweight virtualization environment that makes use of FreeBSD's jail infrastructure (`jexec`, `jls`, `/etc/jail.conf`, etc.).

I am hoping that the approach I am presenting here will illustrate the following benefits:

Leverage standard jail infrastructure to uniform handling of Linux and FreeBSD jails.
Create as many Linux jails as you want (not just one in `/compat/linux`, respectively `/compat/ubuntu`).
There is at least one negative side to putting the Linux emulation into a separate jail: It is less convenient, because you won't be able to execute Linux programs as if they were part of the host system. However, you will get a clean separation between the FreeBSD world and the Linux world.

This tutorial was tested on FreeBSD-13.1-RELEASE.

*Basic setup*

The setup I am starting with is the following:

I am using ZFS, but do not rely on it in this tutorial. I have added remarks when appropriate on how to do the same thing on UFS.
I am using “bare metal” jails, no management utility like `sysutils/bastille` or `sysutils/iocage`. The use of them could possibly simplify some aspects of this tutorial, but I like the down-to-earth approach when using jails directly.
My jails are attached to a loopback device `lo1` and I use `pf` to NAT to forward network traffic to my physical network. This way I can assign static IP addresses to my jails and keep using DHCP on my physical network. If you are interested in the details of this setup, just give me a call and I post them here.
I use Xorg, not Wayland, which I am unfamiliar with. If you know Wayland and a simple way to modify this tutorial to support both, please let me know!
Ok, let's go.

*Enabling Linux support

Why we probably don't need /etc/rc.d/linux or similar*

Normally, the service `/etc/rc.d/linux` takes care of everything that is needed to run Linux binaries. The service basically does the following:

Load the required kernel module(s).
Configure how to handle unbranded ELF executables.
Mount Linux system directories like `/dev`, `/proc`, etc., under the path specified by `compat.linux.emul_path`.
Loading the kernel modules can be done in `rc.conf`. Mounting the Linux system directories will be dealt with by FreeBSD's Jail infractructure.

So, besides the issue with the unbranded ELF executables, there is not much need for starting the service. Therefore, I omit it in this tutorial.

*Loading the modules*

Ensure that Linux support is activated:

```
# kldload linux64
```

To make it permanent, add the module in `/etc/rc.conf` to the `kld_list` variable, but ensure that your other modules remain there:

```
kld_list="... linux64 ..."
```

E.g., my `kld_list` now looks like this:

```
kld_list="i915kms linux64"
```

*Creating the jail*

I like to put each of my jails into a separate ZFS dataset. This, for example, makes it easier to create snapshots or to transfer the jail to another computer. The zpool I am using for this is called `scratch`. You'll have to adjust that name to your particular configuration (e.g. `zroot` or similar):

```
# zfs create scratch/jail/ubuntu
# zfs set mountpoint=/jail/ubuntu scratch/jail/ubuntu
```
If you are using UFS instead of ZFS, a simple `mkdir -p /jail/ubuntu` will do.

Next, we'll need to prepare the system directories:

```
# for DIR in /dev/fd /dev/shm /tmp /proc /sys; do mkdir -p /jail/ubuntu/${DIR}; done
```

Now we can populate the jail with an Ubuntu root filesystem. For this, we'll use `sysutils/debootstrap`:

```
# pkg install debootstrap
# debootstrap --arch=amd64 --no-check-gpg focal /jail/ubuntu
```

Before we can actually start the jail, we need to specify what has to be mounted inside the jail. FreeBSD's jail mechanism can parse an `fstab`-like file that does exactly that. The location of this file is irrelevant, but for simplicity, we put it right into the jail itself. So, create it using an editor:

```
# vi /jail/ubuntu/etc/fstab
```
and put the following content into it:

```
devfs           /jail/ubuntu/dev      devfs           rw                      0       0
tmpfs           /jail/ubuntu/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0
fdescfs         /jail/ubuntu/dev/fd   fdescfs         rw,linrdlnk             0       0
linprocfs       /jail/ubuntu/proc     linprocfs       rw                      0       0
linsysfs        /jail/ubuntu/sys      linsysfs        rw                      0       0
/tmp            /jail/ubuntu/tmp      nullfs          rw                      0       0
```
Note that, with this configuration, the jail will share the host's `/tmp` directory. This makes it easier to share audio and X11 sockets with the jail, but, for security reasons, you might want a different configuration sometime later when everything is up and running. –

Next, we need to configure the jail. Open the file `/etc/jail.conf` and append the following content:

```
ubuntu {
    host.hostname="ubuntu.schattenwelt.org";
    ip4.addr="lo1|10.10.0.5/24";
    path="/jail/ubuntu";
    allow.raw_sockets=1;
    exec.start='/bin/true';
    exec.stop='/bin/true';
    persist;
    mount.fstab="/jail/ubuntu/etc/fstab";
}
```

You will probably need to adjust the following:

The domain name (mine is `schattenwelt.org`, but your's will be different).
The IP address. In my configuration I use a loopback device (`lo1`) and assign static IP address for each jail manually. If you wonder how I get internet access inside the jail: I use the packet filter `pf` for that, redirecting network traffic from and to the jail to my physical network.
We are now ready to start the jail:

```
# service jail onestart ubuntu
```

To start the jail at each boot, put the following into your `/etc/rc.conf`:

```
jail_enable="YES"
```

You can check if the jail is properly running by doing

```
# jls
```

which should give the following output:

```
JID  IP Address      Hostname                      Path
     ...
     4  10.10.0.5       ubuntu.schattenwelt.org       /jail/ubuntu
```

*Setting up the Ubuntu jail*

Now that the Ubuntu jail is up and running, we can set up the Linux system. First, enter the jail:

```
# jexec ubuntu /bin/bash
```

Configure locales and timezone:

```
# dpkg-reconfigure locales
# dpkg-reconfigure tzdata
```

Install necessary packages (for running Google Chrome later on):

```
# printf "deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse" > /etc/apt/sources.list
# apt update
# apt install curl gnupg pulseaudio fonts-symbola ttf-mscorefonts-installer
```

Install GPG key enabling the Google Chrome DEB repository:

```
# curl -s https://dl.google.com/linux/linux_signing_key.pub | apt-key --keyring /etc/apt/trusted.gpg.d/google-chrome-stable.gpg add -
```

Install Google Chrome:

```
# printf "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
# apt update
# apt install google-chrome-stable
```

Create a wrapper script for launching Chrome: Create a file `/opt/google/chrome/chrome-wrapper` and add the following content:

```
#!/bin/bash
#
# Copyright (c) 2011 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

HERE="/opt/google/chrome"

export CHROME_VERSION_EXTRA="stable"

# We don't want bug-buddy intercepting our crashes. http://crbug.com/24120
export GNOME_DISABLE_CRASH_DIALOG=SET_BY_GOOGLE_CHROME

# Sanitize std{in,out,err} because they'll be shared with untrusted child
# processes (http://crbug.com/376567).
exec < /dev/null
exec > >(exec cat)
exec 2> >(exec cat >&2)

export DISPLAY=:0
export LIBGL_DRI3_DISABLE=1

"$HERE/chrome" --in-process-gpu --no-sandbox --no-zygote --test-type --enable-features=UseOzonePlatform --ozone-platform=x11 --v=0 "$@" || true
```

Make it executable:

```
# chmod +x /opt/google/chrome/chrome-wrapper
```

Finally, add a non-privileged user that will be able to run Google Chrome (in my case this is `hsebert`, myself):

```
# adduser hsebert
```

Add this point, you should be able to run Chrome. There will be no sound yet, though (see a later chapter to fix that). To run it, first change to our non-privileged user created above:

```
# su - hsebert
```

Then run Chrome:

```
$ /opt/google/chrome/chrome-wrapper
```

When everything is allright, quit Chrome, leave the `su`-environment and ultimatlely the jail:

```
$ exit
# exit
```

*Troubleshooting*

You may need to allow the Ubuntu jail to connect to your Xorg server on the host. Do this by using the following command (on the host!):

```
$ xhost +
```
To make this change permanent, place it, e.g. in `~/.xinitrc`.


*Allow non-root execution*

You can now start Chrome from outside the jail using `jexec` as root:

```
# jexec ubuntu /bin/bash -c "su -c /opt/google/chrome/chrome-wrapper - hsebert"
```
Note that we use `su` inside the jail to drop privileges and use our non-privileged user inside the jail (`hsebert`).

Having to be root in order to launch Chrome is a little bit inconvenient. FreeBSD doesn't allow non-privileged jail-execution for good reasons, though. So we will use a different approach using `security/sudo`:

```
# pkg install sudo
```

First, we'll create a launcher script on our host and call it `/usr/local/bin/linux-chrome`:

```
#!/bin/sh

if [ $# -lt 1 ] ; then
    echo "usage: ${0} <user>"
    exit 1
fi

if [ "${1}" = "root" ] ; then
    echo "Cowardly refusing running Chrome as root"
    exit 1
fi

jexec ubuntu /bin/bash -c "su -c /opt/google/chrome/chrome-wrapper - ${1}"
```

Make it executable:

```
# chmod +x /usr/local/bin/linux-chrome
```

Next, we allow a particular user on the host system to execute `linux-chrome` using `sudo`, but without requiring a passwort _for this particular command_ (this is important!). For this, create a dedicated `sudoers` file as follows:

```
# visudo -f /usr/local/etc/sudoers.d/linux-chrome
```
and put a line into it having the following format:

```
<user> <host> = (root) NOPASSWD: <cmd>
```

The user (`<user>`) on my host system who is to be allowed to run `linux-chrome` is `hsebert` (same user name as inside the Linux jail, but this is not a necessity). My host name (`<host>`) is `thinkle`. The command to be run (`<cmd>`)is `/usr/local/bin/linux-chrome` (full path!). So, the resulting line looks like this:

```
hsebert thinkle = (root) NOPASSWD: /usr/local/bin/linux-chrome
```

After saving and closing `vi`, your regular user should be able to run Chrome as follows:

```
$ sudo linux-chrome $(whoami)
```

You can put this command inside a Desktop launcher, for example.

*Enabling sound*

Google Chrome on Linux relies on Pulseaudio, so we have to deal with it. In this guide I chose to run Pulseaudio as a system-wide service, which has pros and cons:

Pro: It can be used by multiple jails at the same time, and even by Bhyve virtual machines, if remote access is configured (see below).
Con: Even the Pulseaudio developers discourage the use as a system-wide service because of security issues, see here. Pick your poison. –
First, we need to install the package:

```
# pkg install pulseaudio
```

Pulseaudio aware applications can communicate with a Pulseaudio server either by TCP or a Unix-Domain-Socket (similar to Xorg). We configure this feature explicitly by editing `/usr/local/etc/pulse/system.pa` and add the following two lines:

```
load-module module-native-protocol-tcp auth-anonymous=1 auth-ip-acl=127.0.0.1;192.168.178.0/24
load-module module-native-protocol-unix auth-anonymous=1 socket=/tmp/pulse-native
```
The second line is most important for us: It tells us how the Pulseaudio socket will be named (`/tmp/pulse-native`). We'll need that information later when we configure Pulseaudio in the Linux jail.

_Important:_ If you do not want remote connections over the network, delete the first line containing `module-native-protocol-tcp`. If remote connections are o.k. for you (e.g. because you would like to play audio in your Bhyve virtual machines), be sure to set the correct subnet mask (mine is in this case: 192.168.178.0/24).

Now that we have configured the Pulseaudio server, we need to enable it. The package `audio/pulseaudio` does not come with an `rc` script to start it up at boot time. Therefore, we have to create one: Create the file `/usr/local/etc/rc.d/pulseaudio` and let it have the following content:

```
#!/bin/sh

# PROVIDE: pulseaudio
# REQUIRE: DAEMON FILESYSTEMS
# KEYWORD: nojail shutdown

. /etc/rc.subr

name="pulseaudio"
desc="Start the Pulseaudio server"
rcvar="pulseaudio_enable"
pulseaudio_bin="/usr/local/bin/${name}"
pulseaudio_pidfile="/var/run/pulse/pid"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
load_rc_config "${name}"

pulseaudio_start()
{
    ${pulseaudio_bin} --system --disallow-module-loading &
}

pulseaudio_stop()
{
    if [ -f "${pulseaudio_pidfile}" ]
    then
        kill $(cat "${pulseaudio_pidfile}")
    fi
}

run_rc_command "$1"
```

Next, enable the service in `/etc/rc.conf`:

```
pulseaudio_enable="YES"
```

Finally, start the service so the we do not have to reboot the machine:

```
# service pulseaudio start
```

We are almost there. The one thing left is to let Chrome in the Linux jail know which socket to use when talking to our Pulseaudio server. We could put this information directly into the `chrome-wrapper` script introduced further up, but I think it's better to make it a system-wide default for our Linux jail. Therefore, create the file `/jail/ubuntu/etc/profile.d/05-pulseaudio.sh` and add the following line:

```
export PULSE_SERVER=unix:/tmp/pulse-native
```

_Note:_ If you would like to configure remote access, simply replace to line above with

```
export PULSE_SERVER="<Host-IP-address>"
```
where "Host-IP-address" is the addres of the machine running the actual Pulseaudio server.

*Debugging sound*

If there is no sound, ensure that the Pulseaudio connection is working: Enter the Linux jail and run

```
$ pactl list
```

If everything works, you should see a list of available sources and sinks. If you get something like this:

```
Connection failure: Connection refused
pa_context_connect() failed: Connection refused
```
then ensure the following:

Is the `PULSE_SERVER` variable set correctly?
Is the socket `/tmp/pulse-native` available, i.e. is `/tmp` properly mounted?
Is the Pulseaudio server running on the host: `ps -a | grep pulseaudio`?
Have you tried turning it off and on again?

*References*

[Linuxulator] How to install Brave (Linux app) on FreeBSD 13.0+
[Linuxulator] How to run Google Chrome (linux-binary) on FreeBSD
Create an Ubuntu Linux jail on FreeBSD 12.2
fstab in FreeBSD jails


----------



## hp550c (Aug 17, 2022)

Hi Holger.  Great write-up!  I'm close to getting it to work, but ran into an issue with running Chrome.  I'm getting this when running the wrapper script:


```
./chrome-wrapper: line 17: /dev/fd/62: Operation not supported
./chrome-wrapper: line 18: /dev/fd/62: Operation not supported
[15782:110082:0817/142226.801489:ERROR:file_path_watcher_inotify.cc(329)] inotify_init() failed: Function not implemented (38)
[15782:110088:0817/142226.857930:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[15782:110088:0817/142226.858034:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[15782:110085:0817/142226.860094:ERROR:address_tracker_linux.cc(190)] Could not create NETLINK socket: Address family not supported by protocol (97)
[15782:110087:0817/142226.860371:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[15782:110087:0817/142226.860434:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[15782:15782:0817/142226.861194:ERROR:platform_shared_memory_region_posix.cc(217)] Creating shared memory in /dev/shm/.com.google.Chrome.yZEuZA failed: No such file or directory (2)
[15782:15782:0817/142226.861215:ERROR:platform_shared_memory_region_posix.cc(220)] Unable to access(W_OK|X_OK) /dev/shm: No such file or directory (2)
[15782:15782:0817/142226.861222:FATAL:platform_shared_memory_region_posix.cc(222)] This is frequently caused by incorrect permissions on /dev/shm.  Try 'sudo chmod 1777 /dev/shm' to fix.
[0817/142226.906687:ERROR:ptracer.cc(43)] ptrace: Invalid argument (22)
[0817/142226.906767:WARNING:process_reader_linux.cc(379)] Couldn't initialize main thread.
[0817/142226.906794:ERROR:proc_task_reader.cc(46)] format error
[0817/142226.906807:WARNING:exception_snapshot_linux.cc(349)] thread ID 15782 not found in process
[0817/142226.906851:ERROR:process_snapshot_linux.cc(129)] thread not found 15782
[0817/142226.907065:ERROR:proc_task_reader.cc(46)] format error
./chrome-wrapper: line 23: 15782 Trace/breakpoint trap   (core dumped) "$HERE/chrome" --in-process-gpu --no-sandbox --no-zygote --test-type --enable-features=UseOzonePlatform --ozone-platform=x11 --v=0 "$@"
```

My mounts look like this:


```
zroot/usr/jails/ubuntu on / type zfs (rw,noatime)
devfs on /dev type devfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
fdescfs on /dev/fd type fdescfs (rw)
proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
/tmp on /tmp type nullfs (rw,nosuid,noatime)
devfs on /dev type devfs (rw)
```

Odd that /dev is mounted twice.  It looks like /dev/shm is mounted as well (one of the errors).  The other errors during Chrome startup are related to dbus.  Did you have to run dbus inside the Ubuntu jail to get it to work?  Or what am I missing?

Thanks again!


----------



## Holger (Aug 17, 2022)

Hi! I'm glad you find this tutorial useful! –

Are the mounts inside the jail properly configured?

Could you please post the output of `mount` on the FreeBSD host while your Linux jail is running?


----------



## hp550c (Aug 17, 2022)

It's fixed! 

The issue was in fact because /dev was being mounted twice.  It was being mounted once via `mount.devfs` in `jail.conf`, and again in the jail's /etc/fstab (via `mount.fstab`).  Once I removed `mount.devfs` it only mounted /dev once and now there are a LOT more devices listed now (apparently the ones that make Chrome work ).  Launching Chrome via the wrapper script brings it right up!

Now to continue with your guide and getting sound working and launching it outside the jail on the FreeBSD host.

Thanks again!


----------



## Holger (Aug 18, 2022)

Great, you fixed it!

Could the problem have come from the fact that use are using the `linux` service? I.e. do you have in your `rc.conf` something like:

```
linux_enable="YES"
```

If yes, this might have cased the double-mounts. In such a case, I should update the tutorial to explicitly mention that the `linux` should not be enabled. I.e. that the above line should be removed from `rc.conf`.


----------



## hp550c (Aug 19, 2022)

I don't think `linux_enable="YES"` caused it.  It had /dev mounted in /compat/linux/dev, and /dev for my jail was in /usr/jails/ubuntu/dev, so they weren't colliding.


----------



## hp550c (Sep 2, 2022)

Hey Holger.  I finally got the time to implement the rest of your guide (running Chrome from the FreeBSD host as a regular user, setting up audio, etc.).  It's all working great now -- amazing!  One thing I did have to fix was the pulseaudio startup script you have mentioned.  You need to add `load_rc_config "${name}"` before the `pulseaudio_start()` function, otherwise I was getting an error that `pulseaudio_enable="YES"` wasn't set in `/etc/rc.conf` when it actually was (because the startup script wasn't actually importing the variable from rc.conf).


----------



## Holger (Sep 10, 2022)

hp550c said:


> Hey Holger.  I finally got the time to implement the rest of your guide (running Chrome from the FreeBSD host as a regular user, setting up audio, etc.).  It's all working great now -- amazing!  One thing I did have to fix was the pulseaudio startup script you have mentioned.  You need to add `load_rc_config "${name}"` before the `pulseaudio_start()` function, otherwise I was getting an error that `pulseaudio_enable="YES"` wasn't set in `/etc/rc.conf` when it actually was (because the startup script wasn't actually importing the variable from rc.conf).


Thanks for the fix! I will update the tutorial ASAP.

*Edit: *Tutorial has been updated.


----------

