# FreeBSD Jails vs Docker - A Scientific Approach



## tanis (Jul 23, 2022)

Last week someone told me they wanna start to use Docker in a enterprise environment and I justed didn't asked for further details, because my mind was off doing something else at that time and I went on with my business. Today, for whatever reason, that chat came back to me and itched me the whole day. So I started a bit researcher Docker as I'm not familiar with the technology, sure I read about it in the news and I heard people talk about it, but I use jails so I never gave it 2nd thought till today.
My google search: freebsds jail vs docker

3rd result:
Jails vs Docker - A performance comparison of different container technologies - Christian Ryding & Rickard Johansson

So I read that, well because I thought two young eager computer engineers willing to travel to uncharted land. To my surprise this paper is rather "thin", skinny comes to mind, regarding technical details.

I'm thinking about recreating that paper, BUT with a proper tweaked FreeBSD to make it a fair street fight and collect all the technical details here, just for the fun of it.


----------



## kpedersen (Jul 23, 2022)

Actually quite a good paper.

The conclusion was generally sound too. "Jails and Docker are very similar but due to hype generated benefits; go for Docker" 

Figure 8 was useful however; especially with the whole point behind fast container spinup.


----------



## jbo (Jul 23, 2022)

If you want to have more fun: Look (deep) into security related aspects.
I haven't looked at docker the last two years. Surely the situation somewhat changed but maybe also not...

Also, lets not forget another aspect that is important to many: Docker images are somewhat "cross platform portable".
This is of course not important to many of us here, but it certainly is to some docker users.


----------



## kpedersen (Jul 23, 2022)

jbodenmann said:


> Also, lets not forget another aspect that is important to many: Docker images are somewhat "cross platform portable".


This is the bit that I strongly question. Docker images only work on a single platform. Linux (99% amd64).

Needing to provide an amd64 Linux environment to run the image via virtualization is completely the opposite to cross platform / portable.

But we all know that because it becomes painfully obvious when we use FreeBSD. But explaining this to the masses is unproductive.


----------



## jbo (Jul 23, 2022)

kpedersen said:


> This is the bit that I strongly question. Docker images only work on a single platform. Linux (99% amd64).


Hmm... yeah. As mentioned: I don't know much about docker.
I know that one of our customers was really into that. But I think they also used some 3rd-party "enterprise" tool. I wouldn't be surprised if that was "just" docker "launcher" with an integrated VM so the same image could be ran on Windows and MacOS.

Don't listen to me on docker related aspects


----------



## Jake0162 (Jul 25, 2022)

I was curious about this paper as well, I came across it today and then saw this post appear on the forms.

I found figure 6 to be the most concerning to me as the disk read performance dropped off so quickly. I can't help but wonder what the reason for this could be, maybe an un-tuned ZFS setup. Also the script that they used for their testing link here ionugget.py for anyone who wants to read through it.

From what I can see it would seem that they use a 1024Kb for writes size and 512Kb for reads, but no info about what the zpool and dataset had for settings in the paper or the author's git repos.


----------



## zirias@ (Jul 26, 2022)

jbodenmann said:


> Hmm... yeah. As mentioned: I don't know much about docker.
> I know that one of our customers was really into that. But I think they also used some 3rd-party "enterprise" tool. I wouldn't be surprised if that was "just" docker "launcher" with an integrated VM so the same image could be ran on Windows and MacOS.


IIRC, docker already contains the ability to run a "foreign" image virtualized, so it's more or less transparent to the user. Except of course if your host is already a VM and you run into the typical "nested virtualization" issues for that reason. 

I'm personally not too fond of such hidden complexity. If you need virtualization for your usecase, this should be implemented explicitly, by a different tool. But maybe I'm getting old...


----------



## hardworkingnewbie (Jul 26, 2022)

tanis said:


> 3rd result:
> Jails vs Docker - A performance comparison of different container technologies - Christian Ryding & Rickard Johansson


Well, it's a benchmark, so always a little bit synthetic. 

The driving force why many people do use Docker is the promise of "doing more in less time", also that it doesn't matter on which Linux distribution you are running the Docker image, because the infrastructure is the same across all platforms and the image is standardized. It's using a standard base system (like Ubuntu) and that's it. 

A good example for Docker's use is Discourse. The open source version of that forum engine is only offered as Docker image, because for the makers it really simplifies support questions a lot. That image even comes with builtin update. You still can install the source and all components by yourself, but this takes much more time and with updates/support then you're on your own. 

This is basically how to install its docker image: 


```
sudo -s
git clone https://github.com/discourse/discourse_docker.git /var/discourse
cd /var/discourse
chmod 700 containers
./discourse-setup
```

...and then some web based stuff. 

This is how to install that thing manually: 








						Install Discourse Forum on Ubuntu 18.04 Without Docker
					

This tutorial is going to show you how to install Discourse on Ubuntu 18.04 server without docker on a 1GB virtual private server (VPS).




					www.linuxbabe.com
				




So much more complexity. And this is why Docker became such a big thing, with all good and bad consequences involved.


----------



## drhowarddrfine (Jul 26, 2022)

Zirias Cool kids do what cool kids do cause social media told them to. No thinking involved. Thinking involves work which they are loathe to do


----------



## forquare (Jul 26, 2022)

Zirias said:


> IIRC, docker already contains the ability to run a "foreign" image virtualized, so it's more or less transparent to the user. Except of course if your host is already a VM and you run into the typical "nested virtualization" issues for that reason.
> 
> I'm personally not too fond of such hidden complexity. If you need virtualization for your usecase, this should be implemented explicitly, by a different tool. But maybe I'm getting old...


Docker Desktop is a package that does it all for you - the name perhaps suggesting that it's aimed at Desktop folk who don't want to get down into the weeds.  It will create a VM (which you can tweek various parameters in the Docker Desktop settings), and IIRC will install some of the Docker commands for you.

The actual sysutils/docker tool itself doesn't have the functionality to create a VM.  It expects you to configure it to talk to a unix socket/tcp port where a compatible daemon is listening and will facilitate running the image in whatever runtime and environment is configured there.

So it depends what route you take as to whether you get "magic" or not, but even if you choose Docker Desktop, I'd argue you've got to be pretty ignorant to not see that there's a VM running - or maybe it's only me that pokes through the settings of a new application when I install it...


----------



## drhowarddrfine (Jul 26, 2022)

Further comment. It would bother me a lot seeing all the "Docker is Dead" articles around for many years but now people on FreeBSD are wanting it. Seems Google wants everyone to use Kubernetes so all the cool kids start using that cause Google told them to. 

A lot of this seems more relevant to those with large systems and I know some here have large or large-ish systems they manager but I think most people who start using these things don't need to.

I stopped worrying about such things about three years ago so I probably don't know what I'm talking about.


----------



## mer (Jul 26, 2022)

sysutils/docker-machine port says "tool to create docker hosts"
A way that we've found docker useful at work is "build environments".  You know how we all have preferences in what makes the perfect workstation?  Well if you are working on a project with a bunch of writing software you can run into "well it built and ran fine on my system,not sure why it crashes on yours".
So create a docker container that has the build environment (should also be close to or the same as the runtime environment) that everyone uses to actually compile test in before checking into the SCM tool.  Gets rid of a lot of headaches.  It also can serve as the basis for git CI builds and if you do it right, also lets you create a debug environment when you have to look at core files for a released image.

Is Docker really any better or any worse than other virtualization techniques?  Maybe, Maybe not.  Depends on exactly what your requirements are, what is available.


----------



## kpedersen (Jul 26, 2022)

drhowarddrfine said:


> but I think most people who start using these things don't need to.


I find it isn't really Docker that people care about; it is DockerHub that they really want. Nice clickable packages that hide complexity.

The question is; are people lazy / careless so they can just spin up a "service" with _who-knows-what_ default access settings or is it that people would be unable to set it all up themselves anyway*?

* I am not sneering at those who can't setup big services. There are numerous occasions where I have had to give up on some. Usually due to the sheer number of dependencies and other mess that developers drag in these days.


----------



## forquare (Jul 26, 2022)

drhowarddrfine said:


> Further comment. It would bother me a lot seeing all the "Docker is Dead" articles around for many years but now people on FreeBSD are wanting it. Seems Google wants everyone to use Kubernetes so all the cool kids start using that cause Google told them to.
> 
> A lot of this seems more relevant to those with large systems and I know some here have large or large-ish systems they manager but I think most people who start using these things don't need to.
> 
> I stopped worrying about such things about three years ago so I probably don't know what I'm talking about.


Docker might be dead (or dying), but that doesn't mean Linux containerisation is.  A lot of the work that Docker did was given to the Cloud Native Foundation (whose parent is the Linux Foundation), and so much of the methodology is now a "standard" and there are multiple tools implementing it, of which Docker is but one.

The latest version of Kubernetes removes support for the Docker runtime.  At work we host multiple Kubernetes environments and in theory Docker isn't involved at all (some engineers still use the docker cli tool (sysutils/docker)).

I think I'd agree with you about size.  For example, we hosted Jenkins CI with Kubernetes worker nodes because it was fast, scalable, and easy, allowing us to go from ~500 engineers to several thousand with basically no configuration changes.  If you're just hosting a website then it's massively overkill.


----------



## hardworkingnewbie (Jul 26, 2022)

kpedersen said:


> The question is; are people lazy / careless so they can just spin up a "service" with _who-knows-what_ default access settings or is it that people would be unable to set it all up themselves anyway*?


Of course they are, otherwise Apple would have never become so big!


----------



## drhowarddrfine (Jul 26, 2022)

The same thinkning that let tiktok, emojis, reddit, and instagram take off.


----------



## Crivens (Jul 26, 2022)

drhowarddrfine You mean the saying about flies and turds?


----------



## ziomario (Jul 26, 2022)

jbodenmann said:


> Also, lets not forget another aspect that is important to many: Docker images are somewhat "cross platform portable".
> This is of course not important to many of us here, but it certainly is to some docker users.



why its not important to many of you here ? don't you want to expand the number of tools available here ? don't you want always more tools to integrate even better more operating systems together ? I don't want to think that the freebsd users are part of a "religious" closed circle. I'm an hobbyist and I do easy tasks,but I want that a lot of these tasks help me to use different os on the same machine using a lot of tecniques and tools.


----------



## kpedersen (Jul 26, 2022)

ziomario said:


> why its not important to many of you here ? don't you want to expand the number of tools available here ? don't you want always more tools to integrate even better more operating systems together ?


In theory that sounds good but in practice tools like this just make a mess. And bugs, questions and security issues that will arise from this mess will need to be addressed by the community and developers. This all takes time from the things that matter more.


----------



## rootbert (Jul 26, 2022)

kpedersen said:


> The question is; are people lazy / careless so they can just spin up a "service" with _who-knows-what_ default access settings or is it that people would be unable to set it all up themselves anyway*?


I am a heavy oci container user. I am not lazy, but my days only have 24hours like everybody else's and I need to push our software out to customers on time. However, when my contractors want to use a certain software stack the following consideration arise:

if we use the software we have to trust the vendor - the vendor knows best how to install, configure and lock down that software. why should I invest hours over hours just for installing it - this is wasted time and error prone. Furthermore I can omit setting up a whole infrastructure for building my reproducible builds because they are provided by the manufacturer - no problem because my library/compiler etc. does not match the previous build. If we want to really go into details we simply inspect the container, if we find anything suspicous in that container it is probably worth to investigate further and rethink if we can trust that vendor.


----------



## kpedersen (Jul 26, 2022)

rootbert said:


> and lock down that software


That is fair but this part; from all the docker images I have seen, they are all quite open; to ease use and interoperability.

One of the difficulties of precanned docker images is you need to deploy a little bit of guesswork to close all the holes because you weren't the one that set it up.

Docker images tend to feel more like demo VMs rather than something I would actively choose to put in production.


----------



## Jake0162 (Jul 27, 2022)

ziomario said:


> why its not important to many of you here ? don't you want to expand the number of tools available here ? don't you want always more tools to integrate even better more operating systems together ? I don't want to think that the freebsd users are part of a "religious" closed circle. I'm an hobbist and I do easy tasks,but I want that a lot of these tasks help me to use different os on the same machine using a lot of tecniques and tools.


I don't think I would use docker for my person projects or my servers, but it might be nice to have it for FreeBSD. If nothing else it would allow people with very little knowledge to start playing around with the OS without spending a month reading documentation/books on jails and PF before they could have a usable system.


----------



## ziomario (Jul 27, 2022)

Jake0162 said:


> I don't think I would use docker for my person projects or my servers, but it might be nice to have it for FreeBSD. If nothing else it would allow people with very little knowledge to start playing around with the OS without spending a month reading documentation/books on jails and PF before they could have a usable system.



I don't know how to define this kind of behavior. I see in a lot of FreeBSD experienced users. It could be defined as "we are enough for ourselves. We have jails,it works good,we don't need dockers. We don't need a lot of different tools,since we have similar and better tools within the FreeBSD ecosystem". This approach is partially great,because while in the world the system admins want to use tools widely used they grow,they have the chance to grab money from companies and this will attract more developers. But if every FreeBSD sys admin think to don't have the needing to know different tools,methods,even different OSes,the FreeBSD devs will always remain few, at least until they open up to other ecosystems. Is being few people bad ? In part I think so, because the system grows little if the developers are few and if there is little money and the improvements will come later,so we should deal with a lot of bugs and lack of tools that may help to work better.


----------



## tanis (Jul 27, 2022)

kpedersen said:


> Actually quite a good paper.
> 
> The conclusion was generally sound too. "Jails and Docker are very similar but due to hype generated benefits; go for Docker"
> 
> Figure 8 was useful however; especially with the whole point behind fast container spinup.



Hmm is that so , I miss the following points:
- CPU choice: FreeBSD has been targeting Intel x86 since the beginning, AMD is fairly “new”
- memory: no word about possible differences in the memory management system, or perhaps they are the same, anyway u wanna compare them u have to describe them 
- OS specific: standard installation, applied customization, what has been done to compensate for two different operating systems, or had been done anything at all? 
- choice of file system: during the writing of the paper ZFSoL had been stable and production ready the issue was and has always  been the license and in particular the Kernel interface, hasn’t it been 2019 they changed the api which broken the zfs module and Linus came out saying people shouldn’t use it anyway. That being said, the paper goes with two filesystems which couldn’t  be more apart. Either it’s FreeBSD ZFS vs ZFSoL, or it’s UFS2 vs. Ext4. Taking ZFS is at least a questionable choice here.

Then there is the paper discussion in particular the question:

What unexplored benefits Docker and Jails can have by implementing each other’s unique features?

The answer according to the paper can be summarized in:

One size fits all is better then the right tool for the right job.

That’s more an opinion then anything else. 

If that paper would appear on my desk, that are the points I would wanna discuss first. After all what I want as an Engineer is the best outcome, best as in (cost, performance and time, the order is alphabetical ).

And just for the record the time frame for that paper was 10 weeks, I would expect at least a little bit more depth then what has been provided here. That’s work of two weeks, perhaps three.


----------



## hardworkingnewbie (Jul 27, 2022)

forquare said:


> The actual sysutils/docker tool itself doesn't have the functionality to create a VM.  It expects you to configure it to talk to a unix socket/tcp port where a compatible daemon is listening and will facilitate running the image in whatever runtime and environment is configured there.


No wonder given the fact that development on the FreeBSD port of Docker stopped around 7 years ago. Yes, there was once an effort made trying to get Docker run on FreeBSD. 

The problem with that is that Docker relies on Linux only kernel features like name spaces (were supposed to be introduced in FreeBSD 12, did that happen?) or control groups. So in order to get it running you've got to rewrite these portions of docker to work differently, but do the same.


----------



## forquare (Jul 27, 2022)

hardworkingnewbie said:


> The problem with that is that Docker relies on Linux only kernel features like name spaces (were supposed to be introduced in FreeBSD 12, did that happen?) or control groups. So in order to get it running you've got to rewrite these portions of docker to work differently, but do the same.


It's only a problem if you want it to run "natively" without any Linux components, which would render most of the ecosystem useless to you - it is my impression that the porters were originally trying port not just the CLI tool but also the container runtime.  Every other non-Linux platform with Docker support doesn't port the runtime, it ports the CLI tool and runs the runtime in a VM - this may be because the effort was done by the folks behind Docker...

The Docker CLI tool (i.e. sysutils/docker, which is (aside from patches) the same Docker CLI tool I have on my work Mac and my personal Linux desktop) does not require Linux only kernel features like name spaces.  It just requires the configuration to talk to the Docker daemon which could be running on a local Linux VM or a remote Linux host.

A while ago I mostly got colima running on FreeBSD, it's basically some wrapper around qemu that creates a VM that runs the Docker daemon. The bit that stopped me going any further was how old sysutils/docker was and how much patching was required. If sysutils/docker was up-to-date and worked then I think those that wanted it could have Docker on FreeBSD in the same way that macOS and Windows has Docker.

And I keep saying Docker but mean to say "Linux container", because Docker is just a particular implementation these days.

Slightly off topic, but personally I'm quite excited to see where the runj project goes.


----------



## freezr (Jul 27, 2022)

I am least person authorized to speak about such topic but I couldn't find on which Linux distro the research was made.

Did I miss this info?

Thanks,

F.


----------



## rootbert (Jul 27, 2022)

they used Ubuntu 19.10


----------



## freezr (Jul 27, 2022)

rootbert said:


> they used Ubuntu 19.10



I missed paragraph 7.2... 

Ubuntu has the worst default ever I would argue that by default FreeBSD is not tuned for any specific task  but...


----------



## tanis (Jul 27, 2022)

That’s the actual question, isn’t it?!  I mean we can sit here an talk the pro’s an con’s or we just try to contradict that paper. Can’t be that hard. 

Vultr offers Intel and FreeBSD for a fair price, so hardware will not be an issue.

Filesystem I would stick with UFS2.

So what’s the Docker distribution in the business, we can pick a fight with? 

Which settings are needed in sysctl.conf to get FreeBSD ready for the fight? 

An in the end we put that all in an ansible playbook to keep it documented and reproducible, an then it’s on I guess!


----------



## tanis (Aug 2, 2022)

Looks like I wasn’t up to date.There is actually some technology around since 2018 which aims to play in the same ballpark like Docker and it’s successor Kubernetes:

BastilleBSD

Just stumbled on it by accident. 

Still haven’t found the time though, to actually spend some time on this.


----------



## jbo (Aug 2, 2022)

tanis said:


> BastilleBSD


Another great tool providing "all" the goodies is sysutils/cbsd by Ole.
What I specifically like about cbsd is that it's one tool capable of not only managing jails but also VMs - and those even for/with different hypervisors.


----------



## Jake0162 (Aug 4, 2022)

tanis said:


> Looks like I wasn’t up to date.There is actually some technology around since 2018 which aims to play in the same ballpark like Docker and it’s successor Kubernetes:
> 
> BastilleBSD
> 
> ...


Bastille is really good, I've been using it for my homelab and my vps's to handle separation of different servers. My main gripe with it would be getting it to play nice with my pf rules but that's mostly just on me to fix; and networking while trying to keep the entire setup confined to the jails through Bastille becomes a bit of a burden. It's definatly worth a look if you have a lot of jails to deal with, and makes setting them all up with templates easy. Saves time mostly though.


----------



## fraxamo (Aug 6, 2022)

Jake0162 said:


> My main gripe with it would be getting it to play nice with my pf rules but that's mostly just on me to fix;


Have you seen point #8 at this link?


----------



## Jake0162 (Aug 7, 2022)

fraxamo said:


> Have you seen point #8 at this link?


Yup, I've read the documentation, my issues stem from trying to redirect stuff between jails with nginx acting as a reverse web proxy. Besides that though it might be a bit till I can even tinker with the project. I bricked my server during a bios upgrade so now I'm waiting on a programmer and a bunch of windbond chips.


----------



## jbo (Aug 7, 2022)

Jake0162 said:


> Yup, I've read the documentation, my issues stem from trying to redirect stuff between jails with nginx acting as a reverse web proxy. Besides that though it might be a bit till I can even tinker with the project. I bricked my server during a bios upgrade so now I'm waiting on a programmer and a bunch of windbond chips.


I would suggest opening a separate thread/topic about this in the corresponding category (and providing more details).


----------



## jordanch (Sep 13, 2022)

tanis said:


> Last week someone told me they wanna start to use Docker in a enterprise environment and I justed didn't asked for further details, because my mind was off doing something else at that time and I went on with my business. Today, for whatever reason, that chat came back to me and itched me the whole day. So I started a bit researcher Docker as I'm not familiar with the technology, sure I read about it in the news and I heard people talk about it, but I use jails so I never gave it 2nd thought till today.
> My google search: freebsds jail vs docker
> 
> 3rd result:
> ...


How / why should FreeBSD be tweaked ?


----------



## jordanch (Sep 13, 2022)

mer said:


> sysutils/docker-machine port says "tool to create docker hosts"
> A way that we've found docker useful at work is "build environments".  You know how we all have preferences in what makes the perfect workstation?  Well if you are working on a project with a bunch of writing software you can run into "well it built and ran fine on my system,not sure why it crashes on yours".
> So create a docker container that has the build environment (should also be close to or the same as the runtime environment) that everyone uses to actually compile test in before checking into the SCM tool.  Gets rid of a lot of headaches.  It also can serve as the basis for git CI builds and if you do it right, also lets you create a debug environment when you have to look at core files for a released image.
> 
> Is Docker really any better or any worse than other virtualization techniques?  Maybe, Maybe not.  Depends on exactly what your requirements are, what is available.


The issue with this approach is that it obfuscates the dependencies of your software. "I don't know how/why it worked, so I'll just package the whole system". The convenience / laziness this affords is a benefit, but this isn't an approach I'm in favor of.


----------



## jordanch (Sep 13, 2022)

tanis said:


> Looks like I wasn’t up to date.There is actually some technology around since 2018 which aims to play in the same ballpark like Docker and it’s successor Kubernetes:
> 
> BastilleBSD
> 
> ...


Try CBSD also. Used it in practice on a small scale and it is good. 

What matters a lot with these tools is the active user base and the active contributor base. This is where jail tools fall behind.


----------



## nordkamp (Sep 13, 2022)

I personally think the main issues with this paper come down to the amount of uncontrolled variables there are.
Two completely different operating systems, running completely different filesystems with different defaults, one might have security mitigations, etc. and you expect to be able to compare virtualisation systems, which again are virtualising two completely different operating systems.

Much better would be to take the delta in performance between the host and the guest for jails and docker on their respective setups, and compare those as percentages. That way you aren't just plotting apples and oranges and are actually comparing the relative performance of the technology.


----------



## mer (Sep 13, 2022)

jordanch said:


> The issue with this approach is that it *obfuscates the dependencies of your softwar*e. "I don't know how/why it worked, so I'll just package the whole system". The convenience / laziness this affords is a benefit, but this isn't an approach I'm in favor of.


No it does not.  It actually defines them.  The bigger reason is it allows users to develop on their system of choice and ensure whatever code they write will build compile and run on the intended target.  They can also test in the actual deployment environment before merging in code.  Much cheaper to download a docker container than to send physical hardware around the world for developers.  Say the deployment is on specific versions of a specific OS with specific versions of support packages, create a container then a developer can write code on a Mac, Windows, Linux, BSD, Atari 800, as long as there is support for docker containers.  The rest of the team has a level of confidence that nobody breaks the build.
Did you gloss over the part about having it tied into git with the Continuous Integration bits?  Consistent build envirionments, you don't need to find dedicated hardware with that exact set of software, you need to add a new deployment, simply create a new container and add it to the CI stuff.
If your deployment is on say small embedded devices, the company never has enough real hardware for everyone, so a container of some sort is the easiest way for everyone to contribute.


----------



## bl4ckr00t (Dec 14, 2022)

Hey folks, such nice points throwed over here ! 
Yesterday I discovered cbsd project, as well got some contact with bhyve and microbhyve, as I'm conducting some research on container orchestration in BSD platforms. BastilleBSD is a great tool, didn't try it yet, but will. 

I have a good interesting in contesting this research paper presented, as evaluating containers performance, way of working, packaging and providing development environments and production systems with virtualised technologies varies in so many ways, that comparing all of that in such raw means, can lead to almost infinity misconcepted understandings as well conclusions. 

As I'm entering the jails world as of now, I don't know how is the developer experience and keeping them working in production, but I would love to contribute to get this better and better. Yeah, I know won't must not follow the caddle and do everything for only the fame and the glory, but improving the way people ship code, build stuff and get more productive as well massive scale being played in background and at the corners, is a huge win for the whole ecosystem. The work behind cbsd and runj is such demonstration of all of those potential.

tanis any thoughs on how can we conduct those experiments ? I think we can start by pointing all of the misleadings done in the paper, and pointing the "right way" to get, at least, the most optimal metrics for a fair fight.


----------



## bl4ckr00t (Dec 14, 2022)

mer said:


> sysutils/docker-machine port says "tool to create docker hosts"
> A way that we've found docker useful at work is "build environments".  You know how we all have preferences in what makes the perfect workstation?  Well if you are working on a project with a bunch of writing software you can run into "well it built and ran fine on my system,not sure why it crashes on yours".
> So create a docker container that has the build environment (should also be close to or the same as the runtime environment) that everyone uses to actually compile test in before checking into the SCM tool.  Gets rid of a lot of headaches.  It also can serve as the basis for git CI builds and if you do it right, also lets you create a debug environment when you have to look at core files for a released image.
> 
> Is Docker really any better or any worse than other virtualization techniques?  Maybe, Maybe not.  Depends on exactly what your requirements are, what is available.


Regarding CI stuff and developer experience, also from the "DevOps" side, how can jails provide such "flexbility" ? I'm asking as a newcomer to jails world.
Can we show the world that jails has the same ease-of-deployment scale ?

I have a serious interest in leveraging jails to where it deserves to be : as simple as possible for managing and using in production for whatever is possible.

Just to clarify, I'm setting a small cloud provider infrastructure, for my company and offer those services to my customers. So, basically evaluating approaches and technologies. Jails, CBSD, runj and bhyve seems to be pretty solid to us, the thing now is how to offer this to people so they can ship their systems with ease (we'll provide some config style like a YAML manifest in their SCM of choice, the rest we take care off  ).


----------



## patmaddox (Dec 15, 2022)

Jails are awesome from a developer standpoint. The best part is, they’re lightweight, well-designed, and fully integrated into the OS.

That means, for me, I don’t need to use a jail manager. I’ve checked out Bastille, and it’s cool, but my opinions/taste differ from Bastille. That’s part of the beauty of jails - I just have a few simple scripts to set things up how I like. There‘s a learning curve, but it seems to have clicked for me and there’s a way smaller surface area than there is with Docker and its ecosystem.

As for deployment, I haven’t used them for that yet - we’re still in R&D phase of that at my company. I am currently developing a process that uses poudriere to build ZFS jail images, and uses jectl to upgrade to new versions similar to how boot environments work.

I encourage you to experiment with jails... a big bulk of the dev world is going in a different direction obviously, but in my experience popularity does not necessarily equate to effectiveness.


----------



## patmaddox (Dec 15, 2022)

As an example, I am evaluating monitoring and alerting tools. So today I created seven or eight jails - a couple Nginx jails, and one each for the tools I wanted to try out.

I run a script `jlfs` which makes new ZFS datasets how I like them, and extracts base.txz. My default jail.conf calls a script to set up an epair, add it to the bridge, modify rc.conf in the jail to assign the IP I set as a jail var. Each item in my jail.conf is the jail name, and an IP var, that’s it. `service jail onestart mynewjail` and I’m up and running.

Thats another thing - because jails are basically just a chroot+process namespace+network stack, you can interact with their files directly from the host if you want. They’re just files on the file system, not a shared mount, or having to run remote commands or whatever. `sysrc -f /path/to/jail/etc/rc.conf` lets the host manage certain things on the jail, e.g. setting ifconfig.

Jails are killer.


----------



## jbo (Dec 15, 2022)

patmaddox said:


> Jails are awesome from a developer standpoint.


Also from a deployment/production standpoint!



patmaddox said:


> I’ve checked out Bastille, and it’s cool, but my opinions/taste differ from Bastille.


There are _plenty_ of alternatives (which may or may not match your taste better).


----------



## alexseitsinger (Dec 17, 2022)

It looks like that paper was focused on comparing performance. I'm not sure that it's terribly useful to anyone except a business since CPU time and memory are easy to come by. I think if you're already using FreeBSD, jails are hard to refuse. However, on something else, Docker probably makes more sense.


----------

