# docker and linux compatability issues



## Andrew D (Jan 27, 2018)

Hi there,
I am running

```
FreeBSD 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Sun Jan 7 21:42:48 AEDT 2018
```
with

```
Id Refs Address Size Name
1 35 0xffffffff80200000 1fe5bd0 kernel
2 1 0xffffffff82419000 2018ed zfs.ko
7 3 0xffffffff82642000 7b0f linux_common.ko
9 1 0xffffffff8264d000 42864 linux.ko
10 1 0xffffffff82690000 3c93f linux64.ko
```
Got a bit of an issue with docker and I was hoping someone might be able to help.
I have all the Linux compatibility kernel modules and stuff installed.
I get the following when I try and do anything with Linux images. BSD images work fine.

```
root@vps2:~ # docker run alpine ls
jail: exec ls: Exec format error
jail: ls: failed
```
_or _

```
docker run oznu/unms ls
jail: exec /init: Exec format error
jail: /init ls: failed
```
Does anyone have any ideas on what may be going wrong?

Thanks in advance for your time.

Andrew


----------



## drhowarddrfine (Jan 27, 2018)

I don't participate here anymore but why the hell would anyone running FreeBSD remotely fathom, even once, using Linux software and Docker when there are infinitely far better FreeBSD alternatives?


----------



## Andrew D (Jan 27, 2018)

Hi, I am quite aware of the alternative jail implementations, however my Employer wants UMNS, which will only run in a docker container from what I've read.
If I can't get it working I'll have to move this host to  linux which I really don't wish to do.


----------



## Preetpal (Jan 27, 2018)

drhowarddrfine said:


> I don't participate here anymore but why the hell would anyone running FreeBSD remotely fathom, even once, using Linux software and Docker when there are infinitely far better FreeBSD alternatives?



It's easy to use (low barrier to entry) and it allows you to work on projects (in either a Windows, FreeBSD, macOS, or Linux environment) that will be deployed on (FreeBSD, Linux, and/or Windows) using the production environment.


----------



## Preetpal (Jan 27, 2018)

Andrew D said:


> Hi there,
> I am running
> FreeBSD 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Sun Jan 7 21:42:48 AEDT 2018
> with
> ...



Did you follow the instructions here ( https://wiki.freebsd.org/Docker )?


----------



## Preetpal (Jan 27, 2018)

I followed the steps and I got an Ubuntu docker image working (quite honestly I was pleasantly surprised since I did not know Docker worked on FreeBSD; I encountered issues pulling CentOS though). I also encountered issues running Alpine Linux.


```
/usr/home/ps ruby@2.5.0 $ docker run -i -t alpine /bin/ls
ELF binary type "0" not known.
ELF binary type "0" not known.
jail: exec /bin/ls: Exec format error
jail: /bin/ls: failed
```

Maybe it has something to do with the fact that Alpine Linux uses *musl* instead of *glibc*.


----------



## Andrew D (Jan 28, 2018)

ubuntu does indeed run. interesting. But none of the UNMS containers work, so I guess none of them run on glibc.
linux it is  Thanks for your help.


----------



## Preetpal (Jan 28, 2018)

Maybe something can be done with _brandelf_?


----------



## Preetpal (Jan 28, 2018)

Andrew D said:


> ubuntu does indeed run. interesting. But none of the UNMS containers work, so I guess none of them run on glibc.
> linux it is  Thanks for your help.



Check this out: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213809


----------



## herrbischoff (Jan 29, 2018)

The only reliable way to run Docker on FreeBSD is to virtualize Linux in bhyve and run it there. Docker has and always will be Linux-only because the underlying technology is Linux-only. You can however run the client binary (and only the client binary) as a native FreeBSD application, no Linux emulation layer required. This way, you can control the virtualized Docker instance directly from FreeBSD. Why anyone would do that instead of just using Linux to begin (if Docker is a hard requirement) with is beyond me.

BTW: Docker as a product will be gone in all but 5 years, being obviated by Kubernetes and rkt.


----------



## Preetpal (Jan 29, 2018)

herrbischoff said:


> The only reliable way to run Docker on FreeBSD is to virtualize Linux in bhyve and run it there. Docker has and always will be Linux-only because the underlying technology is Linux-only. You can however run the client binary (and only the client binary) as a native FreeBSD application, no Linux emulation layer required. This way, you can control the virtualized Docker instance directly from FreeBSD. Why anyone would do that instead of just using Linux to begin (if Docker is a hard requirement) with is beyond me.



Have you heard of SmartOS (https://www.joyent.com/smartos)? There are also Windows server containers (https://blog.docker.com/2016/09/build-your-first-docker-windows-server-container/) and there's even a FreeBSD docker image. So the technology isn't Linux only IMO.

The thing is that you can run any Linux Docker image (or even an application consisting of multiple Linux Docker containers) quite easily without minimal Linux knowledge even on Windows, so why would someone not want to do the same on FreeBSD?


----------



## herrbischoff (Jan 29, 2018)

Preetpal said:


> Have you heard of SmartOS



It's a question of priorities. That's a commercial platform. The vendor primarily wants to *sell* their product to enterprises and can therefore invest heavily into customization.



Preetpal said:


> even on Windows



This is because Microsoft has put serious development manpower behind this because they realized Windows as a development platform was losing to Linux and (to a smaller but still significant degree) macOS. FreeBSD development neither has this incentive or interest nor the time or resources to spare to port some functionality that would not be maintained upstream anyway. What you're asking for is a serious commitment with very little payoff. Besides, the Docker container itself can only ever run Linux (except on Windows where it can run Windows as well). What motivation should there be for the FreeBSD community to invest significant time and effort into a Linux-based container format when jails are a comparable native and fully-tested mechanism?

You may be interested in a FreeBSD-native implementation of the App Container Specification: https://github.com/3ofcoins/jetpack (Prototype)

It hasn't been in active development for over a year now.


----------



## tingo (Jan 29, 2018)

FWIW, Docker on Windows runs in a (Linux) vm. Just sayin'...


----------



## Preetpal (Jan 30, 2018)

herrbischoff said:


> This is because Microsoft has put serious development manpower behind this because they realized Windows as a development platform was losing to Linux and (to a smaller but still significant degree) macOS. FreeBSD development neither has this incentive or interest nor the time or resources to spare to port some functionality that would not be maintained upstream anyway. What you're asking for is a serious commitment with very little payoff. Besides, the Docker container itself can only ever run Linux (except on Windows where it can run Windows as well). What motivation should there be for the FreeBSD community to invest significant time and effort into a Linux-based container format when jails are a comparable native and fully-tested mechanism?



I am not asking for anything for the record and I would not want anyone to waste their time on a futile effort. Docker on FreeBSD exists because someone wanted it and is usable even if it is experimental. Doesn't Docker on FreeBSD use jails anyway, so are you against the Linux ABI emulation in FreeBSD?

In the sentence where I said "even on Windows", I wanted to emphasize how it makes it possible for end users to use pre-configured software without having to install anything directly on the host system and without having to acquire the technical knowledge of how to configure the software themselves. See (https://github.com/jupyter/docker-stacks) and (https://github.com/sagemath/docker-images) for examples of what I mean.


----------



## Preetpal (Jan 30, 2018)

tingo said:


> FWIW, Docker on Windows runs in a (Linux) vm. Just sayin'...



https://www.hanselman.com/blog/Dock...indowsWithOrWithoutHyperVVirtualMachines.aspx


----------



## herrbischoff (Jan 30, 2018)

Preetpal said:


> usable even if it is experimental



For all intents and purposes it's not working. When using something like containers, you want production-level stable, not experimental. Your definition of "working" may be very different from mine though.



Preetpal said:


> without having to acquire the technical knowledge



Yes, it definitely is.



Preetpal said:


> so are you against the Linux ABI emulation in FreeBSD?



I'm indifferent. It's just not meant for something as deeply reliant on the Linux kernel as Docker is. If you want Docker, use Linux. If you want jails, use FreeBSD. If you're serious about containers, use CoreOS or SmartOS or any of the other (commercial) strategic platforms trying to corner the market. Your needs define your tech stack, not the other way around.


----------



## Preetpal (Jan 30, 2018)

herrbischoff said:


> For all intents and purposes it's not working. When using something like containers, you want production-level stable, not experimental. Your definition of "working" may be very different from mine though.



Everything is experimental for a period of time before becoming "production-level stable" (give the developers some slack; it is labelled experimental for a reason). Maybe it's a stretch to say it's usable (I have only installed Ubuntu and ran bash myself), but the potential is there. Here is a session from inside a running Ubuntu Docker container on FreeBSD. 


```
root@freebsd-gateway:~ # uname -a
FreeBSD freebsd-gateway 11.1-RELEASE-p4 FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14 06:12:40 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
root@freebsd-gateway:~ # docker run -it ubuntu /bin/bash
root@:/# cd ~
root@:/root# uptime
 11:17:17 up  4:56,  0 users,  load average: 0.38, 0.50, 0.55
root@:/root# perl -e 'print "Hello, World!\n";'
Hello, World!
root@:/root# echo 'Perl works.'
Perl works.
root@:/root# ls /
.dockerenv   .dockerinit  bin/         boot/        dev/         etc/         home/        lib/         lib64/       media/       mnt/         opt/         proc/        root/        run/         sbin/        srv/         sys/         tmp/         usr/         var/         
root@:/root# ls /b
bin/  boot/
root@:/root# ls /bin/
bash   chown  dd     dnsdomainname  false    gunzip    journalctl  loginctl  mknod   mountpoint     pidof  readlink   sed         stty       systemd               systemd-machine-id-setup        tailf     true        vdir          zcat    zfgrep  zmore
cat    cp     df     domainname     fgrep    gzexe     kill        ls        mktemp  mv             ps     rm         sh          su         systemd-ask-password  systemd-notify                  tar       umount      wdctl         zcmp    zforce  znew
chgrp  dash   dir    echo           findmnt  gzip      ln          lsblk     more    networkctl     pwd    rmdir      sh.distrib  sync       systemd-escape        systemd-tmpfiles                tempfile  uname       which         zdiff   zgrep
chmod  date   dmesg  egrep          grep     hostname  login       mkdir     mount   nisdomainname  rbash  run-parts  sleep       systemctl  systemd-inhibit       systemd-tty-ask-password-agent  touch     uncompress  ypdomainname  zegrep  zless
root@:/root# ls /usr/bin/
[           base64     chsh                  debconf-communicate     dpkg-divert              faillog    gpg-zip      install   link       lslogins          nl       perl5.22.1  pwdx          sdiff             sha256sum  sum                   tac          tput                 uptime    whoami
addpart     basename   cksum                 debconf-copydb          dpkg-maintscript-helper  fallocate  gpgsplit     ionice    linux32    lspgpot           nohup    pg          realpath      select-editor     sha384sum  systemd-analyze       tail         tr                   users     x86_64
apt         bashbug    clear                 debconf-escape          dpkg-query               find       gpgv         ipcmk     linux64    mawk              nproc    pgrep       rename.ul     sensible-browser  sha512sum  systemd-cat           taskset      truncate             utmpdump  xargs
apt-cache   bootctl    clear_console         debconf-set-selections  dpkg-split               flock      groups       ipcrm     locale     mcookie           nsenter  pinky       renice        sensible-editor   shred      systemd-cgls          tee          tset                 vmstat    yes
apt-cdrom   busctl     cmp                   debconf-show            dpkg-statoverride        fmt        head         ipcs      localectl  md5sum            numfmt   pkill       reset         sensible-pager    shuf       systemd-cgtop         test         tsort                w         zdump
apt-config  captoinfo  comm                  delpart                 dpkg-trigger             fold       hostid       ischroot  localedef  md5sum.textutils  od       pldd        resizepart    seq               skill      systemd-delta         tic          tty                  w.procps
apt-get     catchsegv  csplit                diff                    du                       free       hostnamectl  join      logger     mesg              pager    pmap        rev           setarch           slabtop    systemd-detect-virt   timedatectl  tzselect             wall
apt-key     chage      cut                   diff3                   env                      getconf    i386         last      logname    mkfifo            partx    pr          rgrep         setsid            snice      systemd-path          timeout      unexpand             watch
apt-mark    chattr     deb-systemd-helper    dircolors               expand                   getent     iconv        lastb     lsattr     namei             passwd   printenv    runcon        setterm           sort       systemd-resolve       tload        uniq                 wc
arch        chcon      deb-systemd-invoke    dirname                 expiry                   getopt     id           lastlog   lscpu      nawk              paste    printf      savelog       sg                split      systemd-run           toe          unlink               whereis
awk         chfn       debconf               dpkg                    expr                     gpasswd    infocmp      ldd       lsipc      newgrp            pathchk  prlimit     script        sha1sum           stat       systemd-stdio-bridge  top          unshare              which
base32      chrt       debconf-apt-progress  dpkg-deb                factor                   gpg        infotocap    line      lslocks    nice              perl     ptx         scriptreplay  sha224sum         stdbuf     tabs                  touch        update-alternatives  who
root@:/root# cat .
./        ../       .bashrc   .profile  
root@:/root# cat /proc/cpuinfo | head -n 7 | grep Intel
vendor_id   : GenuineIntel
model name   : Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
root@:/root# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
root@:/root# echo "Did you even try Docker on FreeBSD???"
Did you even try Docker on FreeBSD???
root@:/root# exit
exit
root@freebsd-gateway:~ # docker version
Client version: 1.7.0-dev
Client API version: 1.19
Go version (client): go1.9.2
Git commit (client): 582db78
OS/Arch (client): freebsd/amd64
Server version: 1.7.0-dev
Server API version: 1.19
Go version (server): go1.9.2
Git commit (server): 582db78
OS/Arch (server): freebsd/amd64
root@freebsd-gateway:~ #
```


----------



## herrbischoff (Jan 30, 2018)

We appear to have very different views on this and you seem to be intent on misunderstanding me. Best of luck in using Docker on FreeBSD. If it works for you, that's great.


----------



## Preetpal (Jan 30, 2018)

herrbischoff said:


> We appear to have very different views on this and you seem to be intent on misunderstanding me.



TBH I felt the same way in this exchange.


----------



## sko (Jan 30, 2018)

herrbischoff said:


> Preetpal said:
> 
> 
> > Have you heard of SmartOS
> ...



SmartOS is released under CDDL; Triton, manta and containerpilot are realeased under Mozilla PL. 
What Joyent sells is cloud-services on their infrastructure which is built on top of smartOS/Triton/manta.

We are using smartOS on 2 hosts now (2 more to come) and I'm also experimenting with smartOS as an hypervisor for redundant/failover FreeBSD gateways. Crossbow is really amazing for virtualized networks.

We don't use docker, but for some linux-only/legacy software there are several LX-zones with Alpine linux running without any issues.

The Joyent stack might be overkill for small installations, but there are projects like e.g. Danubecloud or Project FiFo which also offer direct docker integration.


----------



## herrbischoff (Jan 31, 2018)

Some recent interesting development regarding CoreOS:

https://www.redhat.com/en/about/pre...ding-its-kubernetes-and-containers-leadership


----------



## sko (Jan 31, 2018)

CoreOS _heavily_ relies on and makes use of systemd and provides no secure multi-tenancy as it only leverages cgroups and namespaces and lots of wallpaper and duct tape (called e.g. docker or LXC) over all the air gaps inbetween...
Most importantly, systemd can't be even remotely considered production ready (just 3 random examples that popped up first and/or came to mind...), and secondly, cgroups and namespaces (combined with all the docker/LXC duct tape) might be a convenient toolset for development and offer some great features for this use case, but all 3 were never meant to provide secure isolation for containers; so they shouldn't be used in production if you want/need secure isolation and multi-tenancy (which IMHO you should always want in a production environment).

SmartOS uses zones, respectively LX-zones, for deployment of docker containers. So each container actually has his own full network stack and is safely contained within a zone. This provides essentially the same level of isolation as running a separate KVM VM for each container (which seems to be the default solution in the linux/docker world today) - but zones run on bare-metal without all the VM and additional kernel/OS/filesystem overhead the fully-fledged KVM VM drags along.



If all you want to do is running some docker containers for development or personal use/learning/amusement, just go for one of the ready-to-use Linux-based solutions like CoreOS and do what you actually wanted to achieve with these containers.
If you want to do your development with docker on FreeBSD and are willing to get your hands dirty on the way, you can/should continue trying to get docker containers properly working inside jails, document every issue you had to work around and try to fix or at least report them and commit your bug fixes upstream, so "jailed docker containers" might become a properly robust and production-ready solution one day.
If you are in the unfortunate position and have to run docker containers in a production- or multi-tenancy environment, you just _have_ to go the extra steps involved in using a properly secure and production-safe/-ready solution like e.g. smartOS.


----------



## Preetpal (Jan 31, 2018)

sko said:


> Most importantly, systemd can't be even remotely considered production ready (just 3 random examples that popped up first and/or came to mind...)



You cannot avoid systemd in the Linux world, it's on all the distributions that you would use on servers (Debian, RHEL).


----------



## herrbischoff (Jan 31, 2018)

sko said:


> so "jailed docker containers" might become a properly robust and production-ready solution one day.



Never going to happen. The App Container Specification is going to take hold before that — and it does not appear to have any serious traction as of late but I could be wrong about that. If all communities were willing to properly talk to each other *and* strategic interests of the big players aligned, this could be an ideal outcome. Which henceforth would also be known as the day hell froze over. It has been argued several times over the last couple of years that ZFS along with the isolation mechanisms of jails could form a superior foundation for container technology. ZFS has been transplanted to the Linux ecosystem already. In the end, they're just going to come up with some convoluted solution promising proper isolation at some point, calling it a revolution and finally have any possible lasting benefits sacrificed on the altar of systemd anyway. At least that's my experience with Linux in the past couple of years.


----------



## sko (Feb 1, 2018)

Preetpal said:


> You cannot avoid systemd in the Linux world, it's on all the distributions that you would use on servers (Debian, RHEL).



We have used devuan during the migration to FreeBSD and smartOS on all systems that were running debian. Alpine, which uses openRC, also has gained quite some traction especially as a base for containers or on small VMS/VPS. All our LX-Zones are also running Alpine.



herrbischoff said:


> If all communities were willing to properly talk to each other *and* strategic interests of the big players aligned, this could be an ideal outcome.


With docker completely breaking its API and toolchain every few versions it will never become a stable cross-platform solution, hence my view of docker as a development thingy, not a production safe solution. App Container definitely looks like a sane and stable solution, but in the linux ecosystem everyone seems to be obsessed with the "not invented here" syndrome and tries to come up with its own solution. Also it seems to be the norm nowadays to constantly re-invent everything and completely break things during the process. I think I missed the point when dumpster fires became a normal way of operations, but I always keep a bag of popcorn handy to watch the fallout when finally everything blows up over there....



> Which henceforth would also be known as the day hell froze over.


There's so much oil getting poured into the fire, I think we will have a warm and cozy climate down there for a long time...


----------



## Preetpal (Feb 1, 2018)

sko said:


> Alpine, which uses openRC, also has gained quite some traction especially as a base for containers or on small VMS/VPS. All our LX-Zones are also running Alpine.



What programming languages are the applications that you're running on Alpine Linux written in? I tried using Ruby in an Alpine Linux Docker container and it managed to crash reading an environment variable, consequently, I never tried Alpine Linux again (hopefully things are better, but I bet its compatibility isn't great).


----------



## sko (Feb 1, 2018)

2 are running Java applications, one of them combined with lots of BASIS/BBj stuff. Both were set up by the vendors/contractors; none of them complained about problems. Another one is some pre-packaged-god-knows-what behemoth with precompiled binaries written in unknown languages and lots of shell scripts. Never bothered opening that can of worms by looking closer at it (if I did I'd have to support it in-house because "I know it").

Others are running mainly off-the-shelf software from the package repository or customized and self-compiled old versions of e.g. bind9 (IIRC 9.6 with some fixes for views and DNSSEC). They were only set up for quick migration of openVZ containers with services for our legacy infrastructure. Mostly the pre-compiled packages were just splat into the new zones with "make install" - thanks to the linux ABI being stable (stagnant) for years this worked very well as long as the libc was compatible. Those containers were/are bound to be retired anyways, so it really was only a matter of spending as little time as possible for the migration and focusing on building the new infrastructure instead. TBH they have been running far longer now than I intended/wished, but "the end is nigh" 

We once tried running Zoneminder in an LX-zone (the FreeBSD port was non-functional back then IIRC), thanks to massive problems getting it working in the first place (Zoneminder is the archdemon of dependency hell...) and Zoneminder then being heavily unreliable, this idea was canned and we opted for a dedicated appliance. In retrospect I think the problems were mainly because Zoneminder needs a lot of its dependencies in exactly one specific version and often uses weird specific behaviour of that single version. I ended up compiling nearly everything from sources and the whole thing was extremely fragile. I think in such cases you can't really blame the underlying distro and its choice of software/package versions...
I've stumbled over some blog entry a while ago where someone actually documented installing Zoneminder in an LX-zone, so it seems to be "possible" (don't know how stable/reliable though)


----------



## herrbischoff (Feb 1, 2018)

sko said:


> docker as a development thingy



That’s how I think about Docker as well. It’s an interesting toy, not terribly well executed but a working proof of concept nonetheless. Now, if only the direct development-to-production workflow wouldn’t be one of the major selling point of Docker… That’s why I don’t see any particular value in Docker for FreeBSD.



sko said:


> I think I missed the point when dumpster fires became a normal way of operations



Sadly, it’s at least been this way since Facebook declared “run fast and break things” to be their development strategy. Then, the JavaScript kiddies came and to this day try to invent everything anew, only worse this time.


----------

