# IoT & Freebsd?



## zader (Oct 25, 2021)

Hi All,


I have a group of linux developers who are really kind of set on "docker" this and "docker" that.. and who’s afraid of a git repository big black bat .. They manage a small (5000) fleet of iot devices and their current work flow is basically git > repos > hosted on docker images that are hosted on Balena.

the problem is .. the scale..
What tools, or methods would you use to accomplish the same thing using FreeBSD/Jails/ZFS and such ..

some questions ...
* best way to update 5k devices, I really wouldn’t want to hit a public repo with all that traffic..
* how do you push out non "packaged/compiled" code like python scripts and configuration?
* how do you manage version control, updates and roll backs?
* how do you repair, maintain and monitor the fleet?
are there any good corporate examples of people using freebsd in the wild already? I’m sure there has to be..?


My simple brain want to think saltstack with (padre / git), iocage and zfs replication? with a graphana / Prometheus and your 90% there .. right?

It has to be harder than that ... doesn’t it? ... does that scale to the 50/100k/500k devices?

thanks..


----------



## SirDice (Oct 25, 2021)

zader said:


> What tools, or methods would you use to accomplish the same thing using FreeBSD/Jails/ZFS and such ..


I'm a Puppet fan but Ansible will do just fine too.


zader said:


> best way to update 5k devices, I really wouldn’t want to hit a public repo with all that traffic..


Set up your own package repos. Update using Puppet/Ansible.


zader said:


> how do you push out non "packaged/compiled" code like python scripts and configuration?


Git and/or custom ports/packages. Puppet/Ansible takes care of installation and configuration.


zader said:


> how do you manage version control, updates and roll backs?


Git? Subversion? Heck, even good old CVS would do. 



zader said:


> how do you repair, maintain and monitor the fleet?


Big fan of Zabbix. Works like a charm on FreeBSD. Takes a bit of fiddling to get some good templates and scripts but once it's running it works awesome.


----------



## Alain De Vos (Oct 25, 2021)

With the combination of "plain jails" and "ansible" to configure you do alot.
To monitor i use : "timescaledb / netdata / prometheus / grafana"
To rollback : "incremental zfs snapshots"


----------



## kpedersen (Oct 25, 2021)

Worst case scenario, if their workflow is entirely Linux centric and they are not wanting to learn new things or change workflow, you can do what Docker on macOS officially does and just run Linux inside of a VM like Bhyve. Actually, Docker for FreeBSD would be pretty much exactly this (the core Docker Desktop component, HyperKit / xhyve is even a fork of FreeBSD's Bhyve).

That said. If the main workflow is Linux. Just use Linux rather than faff with VMs.


----------



## rootbert (Oct 25, 2021)

zader said:


> the problem is .. the scale..
> What tools, or methods would you use to accomplish the same thing using FreeBSD/Jails/ZFS and such ..
> 
> some questions ...
> ...



Our fleet on IoT devices is much larger than the one you mentioned. As I have said many times in different threads, FreeBSD lacks application level containers and the respective tooling to make such deployments simple (e.g. a self-hosted repository for containers).
to your questions:
*) 5k devices is a small amount, but it always depends on what they do on the backend
*) we do not ship scripts etc.  ... we use compiled packages for the platform - images to flash.
*) we use git. we work in a high security environment where we can only merge to master branch when at least 2 internal persons have reviewed the patch, then the contractor uses internal validation methods to comply with the criterias, then the update is pushed to the backend where the devices download it and update. keys have a short lifetime and are generated by a hardware security module. rollbacks are not possible, if we ship defect firmware the whole fleet stands still and lots of people will die.
*) prometheus, aws cloudwatch

zfs replication is no option, we cannot accept any dataloss so we use a highly available distributed database.


----------

