# Sysadmin advise request



## fred974 (Nov 24, 2016)

Hi,

I am looking for some advises from the experienced system admin here 

We are a small web development business and we offer free hosting for all our clients..

When we started, we decided that it was best to keep each client (domain) inside their own jail so if a web application get compromised, then it is contained and cannot affect our other clients..

So far, we have 21 jail all running their own webserver and version of php but all sharing 1 common database (also in a jail).

This setup sounded like a good idea at the time but as we are starting to grow, we found a few issue with it. For example, managing and updating the jails and their software is prouving challenging. We started looking at poudriere but could get it to work for us as some jail has different version of php. 

Anyway.. the reason for this post is this...Everyday, my inbox is flooded with daily/weekly run output email report for all the jails and I find it difficult to keep up to look at all of them snf often endup discarding them without reading it all.

How do you guys deal with this problem?
Do you think my setup is overkill for my needs?
I will welcome any sugestion and tools that you guys use

Thank you all 
Fred


----------



## getopt (Nov 28, 2016)

fred974 said:


> I am looking for some advises from the experienced system admin here


Ha! You are setting the hurdle high for answering. Obviously the experienced ones are too busy for an answer. So as an completely unexperienced one I jump in, if you don't mind. 


fred974 said:


> Anyway.. the reason for this post is this...Everyday, my inbox is flooded with daily/weekly run output email report for all the jails and I find it difficult to keep up to look at all of them snf often endup discarding them without reading it all.


Get an idea first of what you want to be informed on. Usually this are some specific strings in a logfile's line, such as "error", "warning", "failed", "not"... etc.

You can easily run tailored cronjobs scanning for the words you are interested in using grep(1) mailing you the filtered output.  

I have disabled redundant logfiles in jails which would be related to specific functions of the jail-host. Make up your mind on this. And there is always the possibility to look at a jail's  logfiles from the the jail-host so mailing might be avoidable here and then.

General rule: Use daily or shorter frequencies for emails only for the absolute high grade alerts you have to act on immediately. For periodic reviews weekly or monthly work might be sufficient. Tailor it for the _priorities in your daily workflow_, i.e at what hours do you want which infos.


----------



## bvansomeren (Nov 28, 2016)

Hi,

I'm doing something similar (different target audience, but also using jails in a similar way).
If you're looking to scale this to dozens and hundreds of jails there are a few bits of advice:

Setup the jail using nullfs for the base system. It allows updating the base in one go
Minimise what runs in your jails. They don't need to run email, reduce cron jobs. Limit what periodic does
Use automation like Ansible, Salt, Puppet or Chef to setup and maintain the jails
As for the logging. I'm looking at something like ELK to handle the log files in a central location.

I'm on the fence about nullfs, ZFS clones seem sufficient. I just make templates for the kinds of services I run, for example:

Jails with OpenJDK 8
Jails with PHP70
Jails with PHP56
Jails with MySQL
Jails with NGINX
I clone these and create a configuration in /etc/jail.conf for quickly spinning up and use Ansible to set each jail up with the right settings.
It's possible on a jail by jail basis to deviate from this. For example to add more PHP modules for specific projects.

Hope this helps.


----------



## fred974 (Nov 29, 2016)

hi bvansomeren 
Thank you for sharing your advise, they are very valuable.
I will look at ELK tomorrow. I hear about ansible automation but don't really know what I need to automate.. Could you please give example?
What jail utility do you use?
I curently use sysutils/iocage and sysutils/bbcd


----------



## sko (Nov 30, 2016)

Here are some neat ansible playbooks and roles available for iocage: https://github.com/JoergFiedler/freebsd-ansible-demo  (also avalable via ansible-galaxy)

Nice thing about this solution: you can roll out jails with ansible in parallel to manually created/already existing iocage jails.


----------



## bvansomeren (Nov 30, 2016)

Hi,

I like the example by sko it's very complete and shows you just what you can do with automation and jails.
Not sure If it's frowned on to list your own work, but I just got done committing my own first role to handle jail setup on Github.
To be honest it's pretty simple and if you managed to do this by hand than automating it should be pretty simple.

Good luck!


----------



## Oko (Nov 30, 2016)

fred974 said:


> This setup sounded like a good idea at the time but as we are starting to grow, we found a few issue with it. For example, managing and updating the jails and their software is prouving challenging. We started looking at poudriere but could get it to work for us as some jail has different version of php.



We use`iocage update jail-name` for more involved custom jails like the one I use for Gogs I `iocage clone` the original jail preform an update and then when everything works as expected I promote it to the master.



fred974 said:


> Anyway.. the reason for this post is this...Everyday, my inbox is flooded with daily/weekly run output email report for all the jails and I find it difficult to keep up to look at all of them snf often endup discarding them without reading it all.


Sure vanilla FreeBSD has awful defaults. I was able to deal with e-mail noise by studying  FreeNAS configuration files https://github.com/freenas which indeed have a very sane defaults for e-mail notification.


----------



## fred974 (Nov 30, 2016)

Oko In which folder did you find the mail config files?


Oko said:


> iocage clone the original jail preform an update and then when everything works as expected I promote it to the master


How do you promote it back to the master?


----------



## bvansomeren (Dec 1, 2016)

Oko said:


> Sure vanilla FreeBSD has awful defaults. I was able to deal with e-mail noise by studying FreeNAS configuration files https://github.com/freenas which indeed have a very sane defaults for e-mail notification.



Thanks for the tip, that sounds useful


----------



## fred974 (Dec 28, 2016)

Hi guys,

After a lot of reading I've decided to installed Ansible..
Could you please advise if I should install it on the FreeBSD host or jail? Or a completely different machine?
In FreeBSD how do I create 'Dynamic Inventory' of the jails?

Thank you


----------



## storvi_net (Dec 29, 2016)

You can use a seperate Jail, or the host as ansible server, or just use your workstation - It totally depends on your situation.

Did you already read this blog-post? https://www.keltia.net/howtos/jail-mgmt-with-ansible/

Regards
Markus


----------

