# Having trouble installing haproxy



## max21 (Nov 23, 2017)

Anyone know why I’m getting this error?  I'm trying to install haproxy into a jail on Virtualbox with FreeBSD as host.  The first few time I tried with /usr/ports/net/haproxy.  Afterwards I found all of it dependencies and installed them first.  Then I installed /usr/ports/sysutils/hatop which is especially required by haproxy.  I had no problem installing ports for other web applications so far until now.


```
warning:
 taking address of packed member 'branches' of class or structure
      'eb_node' may result in an unaligned pointer value [-Waddress-of-packed-member]
        root->b[side] = eb_dotag(&new->node.branches, EB_NODE);
                                  ^~~~~~~~~~~~~~~~~~
59 warnings generated.
gmake[2]: Leaving directory '/ram/usr/ports/net/haproxy/work/haproxy-1.7.9/contrib/halog'
===>  Staging for haproxy-1.7.9
===>   haproxy-1.7.9 depends on file: /usr/local/lib/libcrypto.so.42 - found
===>   Generating temporary packing list
install  -s -m 555 /ram/usr/ports/net/haproxy/work/haproxy-1.7.9/haproxy /ram/usr/ports/net/haproxy/work/stage/usr/local/sbin/
install  -s -m 555 /ram/usr/ports/net/haproxy/work/haproxy-1.7.9/contrib/halog/halog /ram/usr/ports/net/haproxy/work/stage/usr/local/sbin/
install  -m 444 /ram/usr/ports/net/haproxy/work/haproxy-1.7.9/doc/haproxy.1 /ram/usr/ports/net/haproxy/work/stage/usr/local/man/man1
/bin/mkdir -p /ram/usr/ports/net/haproxy/work/stage/usr/local/share/doc/haproxy
(cd /ram/usr/ports/net/haproxy/work/haproxy-1.7.9/doc/ && /bin/sh -c '(/usr/bin/find -Ed $1 $3 | /usr/bin/cpio -dumpl $2 >/dev/null 2>&1) &&  /usr/bin/find -Ed $1 $3 \(   -type d -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 755 "$@"'\'' . {} +  -o -type f -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 0644 "$@"'\'' . {} + \)' COPYTREE_SHARE \* /ram/usr/ports/net/haproxy/work/stage/usr/local/share/doc/haproxy)
*** Error code 127

Stop.
make[1]: stopped in /usr/ports/net/haproxy
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/hatop
#
```

There are *3654 *warning.

With the crazy ones looking like this to near end of the install ...
Line: *11801 *- ->  Line: *21752*

That's a lot.


```
:ebtree/ebmbtree.h 'branches'             :from resultresult300 of  :134 ininwarning14 class  : :to anan  or  -122 unalignedunalignedtaking
structure   warning       pointerpointeraddress: [-Wconstant-conversion]'eb_node'  
valuevalueof
         taking                        *msg_type = PEER_MSG_STKT_INCUPDATE_TIMED;may[-Waddress-of-packed-member][-Waddress-of-packed-member]packed
  addressresult

member                                   ~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~                         up_ptr = &old->node.node_p;                        up_ptr = &old->node.node_p; of
in

'branches'   packedan                                  ^~~~~~~~~~~~~~~~                                  ^~~~~~~~~~~~~~~~of src/peers.c

 member:unalignedclass 259  'node_p':pointerebtree/eb32tree.hebtree/ebmbtree.hor 16 :: of:value438300structure   :: class[-Waddress-of-packed-member]1114'eb_node' :: orwarning
  may :         new_left = eb_dotag(&new->node.branches, EB_LEFT);
structure
       warningwarningresult'eb_node'implicit                             ^~~~~~~~~~~~~~~~~~: :  
inmayconversion
 takingtakingan      fromebtree/eb32tree.h   result :addressaddressunaligned 'int'444   in :ofofpointer to23   an :packedpackedvalue 'char'    unaligned membermember[-Waddress-of-packed-member] changes  pointer warning'branches''node_p'
 value:                   head->branches.b[EB_RGHT] = eb_dotag(&new->branches, EB_NODE);value ofof
 from  [-Waddress-of-packed-member] takingclassclass                                                      ^~~~~~~~~~~~~128
```

Pointer errors?


----------



## max21 (Nov 23, 2017)

Unless I actually missed something, for what I see Freshports had no reference to include python as a dependency (strange), and when I included it anyway it still did not work.  I use FreeBSD 11 stable, and its version of ports.

libpcre.a : devel/pcre
gmake : devel/gmake

sysutils/hatop
net/haproxy


The pkg way works!

```
New packages to be INSTALLED:
   hatop: 0.7.7
   haproxy: 1.7.9
   python27: 2.7.14
   readline: 7.0.3
   libffi: 3.2.1_1
   python2: 2_3
```

CORRECTION:  that is if you use `pkg install hatop`

So haproxy is standalone.  I guess something is wrong with the port version or my updated SVN compiler is missing something.  However, it did compile everything else.


----------



## SirDice (Nov 24, 2017)

max21 said:


> Then I installed /usr/ports/sysutils/hatop which is especially required by haproxy.


sysutils/hatop is not required by net/haproxy.


----------



## max21 (Nov 28, 2017)

SirDice said:


> sysutils/hatop is not required by net/haproxy.


SirDice, I removed debugging using the make.conf.  It did compile.  I kind of forgot what I did but the port version worked.  Anyway, may I ask you one quick question here?  I know you recommend to place haproxy on the host but is it ok to place haproxy or nginx in the first jail to do reverse-proxy for a few other jails with public websites.  I'll be using a single server at a datacenter.  It will lead me to eliminating this question in my next thread.  I’m beginning to understand jail networking, but it seems that jailing can be design for difference use.  Even if all this additional security is not needed (reverse-proxy in jail), I want to try it anyway.  BTW, no more size and color to threads for me.  It been a habit for docs on my mate-desktop.

https://forums.freebsd.org/threads/54445/


----------



## Snurg (Nov 28, 2017)

Didn' t try ports. Just directly installed into jail, it was -j option iirc. It works fine together with pf for several jails serving different web domains. Didn't try out hatop, as haproxy has sweet logging.


----------



## max21 (Nov 28, 2017)

Snurg said:


> Didn' t try ports. Just directly installed into jail, it was -j option iirc. It works fine together with pf for several jails serving different web domains. Didn't try out hatop, as haproxy has sweet logging.



What is IIRC?  Is this for iocage, ezjail  or jails in ZFS.  I only know how to build jails manually.  The clues I just found might be related to those and one of them mention something about mounting a jail inside a jail or something like that.  What key words could you provide so I can do a search?  I am excited that this is actually possible.  Thanks Snurg.  I believe I just read some of your replies to a few  threads within the past few days.  I'm gathering many facts as I can before posting my final question to make sure there be no better way to go.  So I cant miss anything, anymore.


----------



## Snurg (Nov 28, 2017)

iirc=if i remember correctly.

I actually did the manual jail configuration without iocage, ezjail etc, because:
1st these still use the old jail approach, which is not only a bit clumsy and difficult to maintain manually, but also cursed with a deprecation warning, and
2nd I had the feeling that learning necessary to deal with those kinds of "jail frontends" is practically the same as the new jail approach (using jail.conf and rc.conf), so I saw little use in learning iocage, ezjail etc.

It's a bit sad that the handbook, which is otherwise very good, does *not* mention the new (quite easy-to-handle) jail.conf method. So it's best to read the jail.conf manpage in addition to the handbook.

On my server, every jail has its own private IP, runs its own web server environment, a jailed haproxy forwards the domains (which are in the http(s) header) to the appropriate private IPs (without decrypting while passing through, a strength of haproxy), and PF in turn forwards the packets to the appropriate jails and also takes care that the haproxy can only be accessed by cloudflare and local hosts, so foreign portscanners don't see my web servers.


----------



## max21 (Nov 28, 2017)

Snurg said:


> . . .  On my server, every jail has its own private IP, runs its own web server environment, a jailed haproxy forwards the domains (which are in the http(s) header) to the appropriate private IPs (without decrypting while passing through, a strength of haproxy), and PF in turn forwards the packets to the appropriate jails and also takes care that the haproxy can only be accessed by cloudflare and local hosts, so foreign portscanners don't see my web servers.



So that’s how it works! But haproxy don’t work along.  It needs CloudFlare to do SSL termination and cloud flare provides some extras.

So that is why many threads speaks of  using nginx to do reverse-proxy with SSL termination.  Now I’m searching for the pros and cons of each, if any.  This link made your most exclusive reply crystal clear for me.  I thought CloudFlare was like rackspace or something, and I had no clue of how any of this worked, whatsoever, but in my gut I knew something should work as described.  It was all in your wording, and then this link.

https://martensson.io/cloudflare-universal-ssl-with-haproxy/

This is getting more interesting by the minute.

Two Beers and a Button for you my guy


----------



## SirDice (Nov 28, 2017)

max21 said:


> But haproxy don’t work along. It needs CloudFlare to do SSL termination and cloud flare provides some extras.


Huh? No it doesn't. HAProxy is perfectly capable of SSL terminating. It has done so since version 1.5.


----------



## max21 (Nov 28, 2017)

SirDice said:


> Huh? No it doesn't. HAProxy is perfectly capable of SSL terminating. It has done so since version 1.5.



SirDice, I forgot about that.  Most of what I’m seeking to do I read about in older threads where HAProxy was kind of new.  However, passing data thru without decrypting, and leaving nothing behind to scan got to be a wonderful thing.  I’ll make up for the addition overhead, if any, elsewhere.  BTW: I think I was wrong about the port version worked.  I'm going to try again just to make sure.  I got the port version and the pkg version install information inside /var/cache and /var/db/ports.  So I'm not sure, but I think the port version dependency python2-2_3 fail.  I'll find out tonight.  I never leave unknown things behind for something else.  So it makes no since to start doing that now, even though I’ll trade the world for a Snurg setup.


----------



## SirDice (Nov 28, 2017)

Let me know what you're running into. I use HAProxy for my own stuff and I've set it up for a client. So I'm quite sure it builds correct.

Here's my own haproxy.conf with some bits changed, you can use it as an example config.


```
global
        maxconn 30000
        daemon

        log /dev/log local2

        user nobody
        group nobody

        stats socket /var/run/haproxy.socket mode 777 level admin

        tune.ssl.default-dh-param 2048

        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3 no-tls-tickets

defaults
        log global
        option httplog
        option dontlognull
        mode http

        option httpclose
        option abortonclose
        option forwardfor header X-Real-IP
        option http-server-close

        timeout connect 5000
        timeout client 50000
        timeout server 50000

        errorfile 400 /usr/local/www/haproxy/errors/400.http
        errorfile 403 /usr/local/www/haproxy/errors/403.http
        errorfile 404 /usr/local/www/haproxy/errors/404.http
        errorfile 500 /usr/local/www/haproxy/errors/500.http
        errorfile 503 /usr/local/www/haproxy/errors/503.http

        stats enable
        stats uri /haproxy?stats
        stats realm Statistics
        stats auth admin:changemenow

frontend http-in
        bind 1.2.3.4:80

        reqidel ^X-Real-IP:.*

        default_backend local

        # Letsencrypt
        acl is_letsencrypt path_beg /.well-known/acme-challenge/
        acl is_mail hdr_dom(host) -i mail.example.com
        acl is_webtrees hdr_dom(host) -i webtrees.example.com

        redirect scheme https if is_mail !{ ssl_fc }
        redirect scheme https if is_webtrees !{ ssl_fc }

        use_backend local if is_letsencrypt
        use_backend mail if is_mail
        use_backend webtrees if is_webtrees

frontend https-in
        bind 1.2.3.4:443 ssl crt /usr/local/etc/haproxy/ssl/

        http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"
        #http-response set-header X-Frame-Options DENY
        http-response set-header X-Content-Type-Options nosniff

        tcp-request content accept if { req_ssl_hello_type 1 }

        reqidel ^X-Real-IP:.*

        default_backend local

        # Letsencrypt
        acl is_letsencrypt path_beg /.well-known/acme-challenge/
        acl is_mail hdr_dom(host) -i mail.example.com
        acl is_webtrees hdr_dom(host) -i webtrees.example.com

        use_backend local if is_letsencrypt
        use_backend mail if is_mail
        use_backend webtrees if is_webtrees

backend local
        option httpchk GET /up.txt
        server localhost 127.0.0.1:80 check

backend webtrees
        option httpchk GET /up.txt
        server webtrees 192.168.21.3:80 check

backend mail
        option httpchk GET /up.txt
        server mail 192.168.21.4:80 check
```

SSL is terminated at HAProxy, traffic to the backends is always plain HTTP. The SSL settings also result in an A+ rating.

The X-Real-IP header can be used on the Apache or nginx backends to get the 'original' client's IP address (or else everything will appear to come from the HAProxy machine). A nginx running on localhost is used for Letsencrypt to automatically update the SSL certificates.


----------



## max21 (Nov 28, 2017)

SirDice said:


> Let me know what you're running into. I use HAProxy for my own stuff and I've set it up for a client. So I'm quite sure it builds correct.
> 
> Here's my own haproxy.conf with some bits changed, you can use it as an example config.



Thank SirDice,

Its going to take me a minute to complete my setup and figure out this file.  I don't want to munk this up.


----------



## max21 (Nov 28, 2017)

I just did some checking, which don't seems so important now, at this moment. ... It was hatop that threw things off for me.


----------



## SirDice (Nov 28, 2017)

HAProxy doesn't have any _run_ dependencies, it does have 2 _build_ dependencies.


----------



## Snurg (Nov 28, 2017)

I think I'll write up some HOWTO these days, because the configuration I described is a thing apparently quite some people would like to have.

It is a bit complex, because the necessary configuration things regarding host and jails spread over more than a dozen files.
For example, it took me quite some time to figure out how to get SSL passthru, so HAproxy does not need to tamper with the actual data and preventing the actual web server doing the stuff (which limits security in the end). Aside of that, I would lie to my users if I would say them that they have a secure connection while actually using some decrypting and thus eavesdropping reverse proxy. 

Regarding PF, there are still quite a few things in my pf.conf that I didn't yet figure out how to do correctly (traffic counting, bandwidth limiting/distributing, etc).
In the Howto I could then also explain the relationships between jail.conf and rc.conf, what parts of the configuration belongs where, of which I didn't find a complete documentation anywhere, and had to find out with much, much reading.


----------



## max21 (Nov 29, 2017)

Snurg said:


> iirc=if i remember correctly. . . .
> 
> It's a bit sad that the handbook, which is otherwise very good, does *not* mention the new (quite easy-to-handle) jail.conf method. So it's best to
> 
> read the jail.conf manpage in addition to the handbook.



I come to realize that this is fair.  Those folks invented or bettered those jail frameworks to make it easy for new and future FreeBSD users . . . One of them wrote the Handbook of Handbooks IMO,  FreeBSD INSTALL GUIDE.  However, it’s FreeBSD responsibility to improve upon great ideas for everyone.  The authors would use those improvement wherever possible to make things friendlier for GUI type users.  Us connoisseur mostly know we need to drill down a bit to find developers type information.



Snurg said:


> I think I'll write up some HOWTO these days, because the configuration I described is a thing apparently quite some people would like to have.
> . . .


You have gotten too use to your own innovation, it's only second-nature to you.  If it works the way you describe, even a bit, it would take the way jails are build or used to a whole new level.  With that said … 

*Bring it on!*  Look at all the files it takes to make FreeBSD tick.  A jail is _like_ a system by itself that you build!  However, most of use fail to use our imagination.  Evidently, you did not!  You did more with only a dozen plus files … that’s something to be admired.

About jail.conf and rc.conf;  although so very simple, I have 8 files per  jail.  I use each set to start and do all for its own jail.  Automation is my goal.  I use my jail.conf and rc.conf only when I feel like it, mainly to make sure all startup information match.  I am not limited when I use my own files.  I can add more of what can be accomplished - - but I am restricted when I use the jail.conf and rc.conf. they both enforce limitations.  So, when it comes to files, anything less then what it takes to run FreeBSD need not be that complicated, just more clever.  Once all the files are organized, you realize the only complexity was learning how to set it up . . . though this is major *.

Three of the eight are .csh scripts, InstallWorld, start-jail, and kill-environment,  Using .csh scripts are deem unsafe,  but now I don’t think so.  Once started it should never have to restart.  Also consider the fact that it depends on how the entire system is being used … using my-jails on a LAN with un-trusted users, sure there’s risk firing up  .csh scripts.  For private intercommunication among my-jails running web-servers already started by csh, 1 hour – 20 years, I don’t think so.  There are FreeBSD applications that use .csh script, or two, so why tell users who knows what they’re doing, for years, not to do it, with no evaluation of the situation,  I believe the jail system use at least one .csh call under the hood, if not my way is stronger.  

If you think its worth checking out, at your request I’ll post it as a HOW TO by Sunday night.  This will give me time to clean it up a bit.  I planned to do that someday anyway.  It’s so cool to see how every single thing kicks in under the hood just like what you indicated above about jail.conf and rc.conf.  What’s going to blow my mind is to see a FreeBSD server running remotely for many years just like in the old blogs I use to read.  5x – 6x They were loving it, geeking for life with fewer visits to the keyboard.

Snurg, now curiosity is killing the cat.  I could be way off base but I think you also use a few .csh calls  but don’t want to be bother with the “_please don’t do that_” replies, so you worked along, like myself.  I also think some special device(s) may be involved, but that is only a shot in the dark.  Whatever the case, I hope you find the time to create the HOW TO.

BTW:  could you tell me how you set your alliances for your haproxy jail and your public web-jails.  I’m not sure which to use even for a standard WEB-JAILS setup.  I only been using the third one in the list below which allow me to build ports and packages.


.......................................................................................................................................................................................................
For those who don't know, *.csh is no play toy*.  It will wipe out your entire system in a blink of an eye if you make a single mistake, or don’t know what you are doing.  If you see it and don’t know it LEAVE IT ALONG.

So that why people warns us.  But for those who do the warning, understand the situation first, and don't interfere with progress when you know it's productive.  It's the user choice and it's his machine he risk during development.

No mistakes allow.

Now you know all your *ABC's*
.......................................................................................................................................................................................................



Now it’s time to study haProxy with SirDice example and learn something.

This is what I’m after:
https://forums.freebsd.org/threads/63261/

Skywalkers

over and out . . . >


*1)* for a remote dedicated or VPS public web server running Snurg jail solutions

```
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.0.0.1/24"
ifconfig_lo1_alias0="inet 10.0.0.100/32"                      # haproxy
ifconfig_lo1_alias1="inet 10.0.0.101/32"                      # domain1
ifconfig_lo1_alias2="inet 10.0.0.102/32"                      # domain2
```

*2)* for port forwarding a way to connect to your home or work server from laptop while at McDonald's

```
cloned_interfaces="lo1"
ipv4_addrs_lo1=" inet 10.1.1.1 netmask 255.255.255.0"
ipv4_addrs_lo1_alias0="inet 10.1.1.100 netmask 255.255.255.255"
ipv4_addrs_lo1_alias1="inet 10.1.1.200 netmask 255.255.255.255"
```

*3)* for connection to web from jails to do such things as installing ports, ftp or use for virtual hosted websites

```
ifconfig_em0_alias0="inet 10.2.2.1/32"
ifconfig_em0_alias1="inet 10.2.2.102/32" 
ifconfig_em0_alias2="inet 10.2.2.103/32"
```

If this is correct, I'm really ready to roll?


----------



## Snurg (Nov 29, 2017)

I have slept over it and thought about it.
As I am configuring my desktop atm for use as desktop and development/testing server, it's probably best to partition the whole thing some way like this:
1. basic PF configuration
2. jailed secure DNS server (there is a good Howto already, to which I only will add some useful information)
3. Haproxy jail and a web server jail dedicated for obtaining Let'sEncrypt certificates
4. a template jail for the used certificates etc (or several, one for apache, one for nginx, ...)
5. maybe some scripts for routine things like cloning new jails from templates and preconfiguring them (ip, hostname, rc.conf etc)
6. finally some documentation about the routine processes in creating/managing jails

As this your thread is about haproxy, I post my haproxy.conf (anonymized and boiled down to two example web servers):


```
# 10.1.1.10 : jailed haproxy
# 10.1.1.11 : jailed webserver for site_without_https.org
# 10.1.1.12 : jailed webserver for site_with_http_and_https.org

global
maxconn 2000
user haproxy
group haproxy

defaults
timeout client 30s
timeout server 30s
timeout connect 10s

frontend ft_http
bind 10.1.1.10:80
mode http
acl http_site_with_http_and_https hdr(host) -i site_with_http_and_https.org
acl http_site_with_http_and_https_www hdr(host) -i www.site_with_http_and_https.org
acl http_site_without_https hdr(host) -i site_without_https.org
acl http_site_without_https_www hdr(host) -i www.site_without_https.org
use_backend backend_site_with_http_and_https_http if http_site_with_http_and_https
use_backend backend_site_with_http_and_https_http if http_site_with_http_and_https_www
use_backend backend_site_without_https_http if http_site_without_https
use_backend backend_site_without_https_http if http_site_without_https_www

frontend ft_https
bind 10.1.1.10:443
mode tcp
acl https_site_with_http_and_https req_ssl_sni -i site_with_http_and_https.org
acl https_site_with_http_and_https_www req_ssl_sni -i www.site_with_http_and_https.org
use_backend backend_site_with_http_and_https_https if https_site_with_http_and_https
use_backend backend_site_with_http_and_https_https if https_site_with_http_and_https_www

backend backend_site_without_https_http
mode http
server server_site_without_https_http 10.1.1.11:80

backend backend_site_with_http_and_https_http
mode http
server server_site_with_http_and_https_http 10.1.1.12:80

backend backend_site_with_http_and_https_https
mode tcp
server server_site_with_http_and_https_https 10.1.1.12:443
```

As you can see, the config is totally basic and primitive, but it works for me. (Suggestions are welcome!)
Hope it helps.


----------



## max21 (Nov 29, 2017)

Snurg said:


> I have slept over it and thought about it.
> As I am configuring my desktop atm for use as desktop and development/testing server, it's probably best to partition the whole thing some way like this:
> ...
> ...
> ...



Thanks Snurg

I like basic & primitive, that why I stuck with FreeBSD.  Now I can do a few of the right things at the same time again.  Cleaning up my jails scripts would be the perfect place to start.  I’ll post my primitive way of building jails in within 36 hours.  There might be something in there that you can use right now.  Take your time to do all of what you need.  All I needed were some facts.  You guys provided me enough of that to keep me busy for a week.


----------



## max21 (Nov 30, 2017)

> As I am configuring my desktop atm for use as desktop and development/testing server, it's probably best to partition the whole thing some way like this:



More ideas:

ada0s1: 20 - 32GB


```
/dev/ada0s1a - /   512 - 2048
/dev/ada0s1b - swap   64 - 1024
/dev/ada0s1d - /tmp   64 - 1024
/dev/ada0s1e - /var     1024 - 4096
/dev/ada0s1f - /usr     17000 - 21504
FREE 1024MB
.......................................................
ada0s2:        for VM's and bk/restore ada0s1. Disaster Protection

/dev/ada0s2a /       20 - 32GB   install all of FBSD here
/dev/ada0s2b - swap   1024   only a swap is needed 
/dev/ada0s2d - /mydir/d   150GB
/dev/ada0s2e - /mydir/e   150GB
/dev/ada0s2f - /mydir/f   150GB
/dev/ada0s2g - /mydir/g   150GB
/dev/ada0s2h - /mydir/h   300GB
FREE 1024MB
.......................................................
ada0s3  ###GB for windows or more FBSD snaps (i,j,k,l,m)

.......................................................
ada0s4  EXTENDED   use Arch to recover from a disaster of disasters
ada0s5  1024mb       Arch swap
ada0s6  1536mb       Arch Linux

......................................................
ada0s7    msdos
ada0s8    msdos
ada0s9    msdos
ada0s10   msdos
ada0s11   msdos
ada0s12   msdos
ada0s13   msdos
ada0s14   msdos
ada0s15   ntfs
ada0s16   ntfs
ada0s17   ntfs
ada0s18   ntfs
ada0s19   msdos
```

I actually build my most important FreeBSD development VM on ntfs-16 and backup both FreeBSD host’s (ada0s1) and (ada0s2a) and all VM’s on the last partition, msdos-19.  I also did all of this on a 500GB hard drive because I knew one day a flash drive will at least be available for that size in the near future.  So my size is difference. It works for me, but these sizes are better.  I wish I had plan for the 1GB flash-drive.

This way you don't need all of those extra partitions and can do everything within FreeBSD ufs or whatever.  Anyway, it may be helpful.  It's no better then the rest but no less than the best when it comes to doing everything possible on a single machine, pocket-size and cell ready.

I trust dd more, because he don't care.

This is why I keep the host’s and most of my vm’s as small as needed, but my main devel vm is 131GB.

*BACKUP*:

```
dd if=/dev/ada0s16 bs=64k | gzip -c | split -b 3999m - /mydir/win/p/fbsd-devel/fbsd-devel.gz.
```
*RESTORE:*

```
cat /mydir/win/p/fbsd-devel/fbsd-devel.gz.* | gzip -dc | dd of=/dev/ada0s16 bs=64k
```


----------



## max21 (Nov 30, 2017)

Wow! only 2 exact matches on the WWW.  You sure did do a lot of reading and thinking.  Even haproxy (manuals) never thought about it.  I going to read it over and over and over again.  I'm glad I choose haproxy.  It was because of a reply to a thread made by SirDice.  see ya

https://forums.freebsd.org/threads/54445/


```
ft_http
```


----------



## max21 (Nov 30, 2017)

You guys are not going to believe this:
To make sure that there be no flaw in my upcoming jail demo, I  updated to revision 326375 at SVN.  I make buildworld, 7-hours. Once I got to this, make -DBATCH_DELETE_OLD_FILES delete-old-libs this is what I got:

By hand because Virtualbox terminal
has no select, so in this order:

```
/usr/lib/libbsnmptools.so.0
/usr/lib/libbsnmptools.so
/usr/lib/libgssapi.so.10
/usr/lib/debug/usr/lib/libgssapi.so.10.debug
plus 42 more files with .debug extensions
….
….
>>> Old libraries removed
```
Absolutely nothing else was in there to be removed.

So that’s why!  I update with SVN weekly but had not buildworld for over a month.  Evidently, the debug code caused the issues.  However, if this did not happen, I’ll be struggling right now with the wrong reverse-proxy, forever.  Well I guess this thread is SOLVE, with another happy camper prepared to kick *.  My ears will be open, and you guys know where I will be ...  and to think, I still got 24 hours to make my machine XMAS clean 

See you latter Skywalkers.

Thank you FreeBSD.


----------



## SirDice (Nov 30, 2017)

Good point, I hadn't even considered that. Probably because it's almost automatic for me. I've done so many buildworld I don't even think about what I'm doing, it's mostly muscle memory.


----------



## Snurg (Nov 30, 2017)

Cleaning up after make buildworld... oh yes thats a thing I realize I practically never did...    Thanks for pointing at that!

And thanks for post #20 too!
Here is the background why:

When I was starting my reverse-proxied server experiments last year, I used squid as reverse proxy because I knew it a bit.
I did the first parts of my web project (sort of simple-to-use CMS) using only http.
When it was time to add https support, I found out that the claims of squid (and many other software which does reverse proxying) to be able to handle https must be interpreted in a different manner than I meant.
I had to learn that they mean it that way "you must make our proxy an eavesdropping man-in-the-middle!".

And that made me to investigate a bit how https works. I already knew for long time that the https protocol had been updated shortly after its introduction because it had a severe flaw - it lacked the unencrypted  clear-text domain name. So it was necessary to decrypt the packets to find out which domain on that particular IP the packet was addressed to. And this flaw forced hosters to unencrypt packets at the server handling the IP - often the proxy/load balancer and not the actual web server. Which would break the so-praised "security"... So a thing called "SNI" was introduced.

So I had to search what software actually supports SNI, and this was why I ended up installing haproxy.
But when I was configuring, I found out that virtually all pages on the web dealing with haproxy https configuration are about the man-in-the-middle concept, and *not* about simple passthrough. And it took me quite a while to find the req_ssl_sni needle in the huge haystack which the haproxy documentation is...

When I read post #20 I thought, hmm is this still so rare to find, I did a search for "req_ssl_sni" again, this time on Bing. Why do I mention Bing, you might ask. The answer is that I am a lazy ass and just used google because it is the default search engine of Firefox. But Palemoon, which I use as my browser since FF Quantum made me change my default browser, does not offer Google in its search bar 

And what jumps into my eyes then? One of the first hits was this page !
Quote:


> I finally found a solution. It wasn't in the documentation though.
> 
> Use -m end instead of -i for wildcard



One thing what I felt like a sore in my haproxy.conf was the need to explicitly name each subdomain.
You put me into the tracks to find a solution for that just by chance  This why my "Thanks" for post #20


----------



## max21 (Dec 1, 2017)

I might as well post this here.  Put this in your jail-tool collection.  The main problem is too many parts that have to be ran by hand.  Some one once suggested a way to use .sh for all scripts but it didn’t work. Nevertheless, it’s great for testing, and runs great it but one too many script to total shutdown.  Thanks Snurg …  we _ALL_ were right on time.

It’s great that more came out of this thread then expect, the good the bad and the ugly.  Now even I know how they got the charlatan tweet.  Tricky Dick, trying to be slick now getting hack by his own sponsor’s sticks.  How else could Putin know about Hillary and the Trunkster love affair. . .



I now comment out openssl, then install sshguard and libressl for all jails.  I really just build one ail, I give it only what all should have, then copy it with new name.  but it's kind of tricky and confusing. It only take minutes to build a new one then customized each, might as well.

*InstallWorld tells the whole story
A_InstallWorld*

```
#!/bin/csh
#                                                                PROXY - INSTALL WORLD
#
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        is there and directory empty
mkdir /mnt/z/proxy
cd /mnt/z/proxy && chflags -R noschg * && rm -Prf *; sleep 2;
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        paste-in custom files
cp -pr /mnt/j/MAKE/proxy/config-files/REQUIRED/script1/.q /mnt/z/proxy/.q
cp -pr /mnt/j/MAKE/proxy/config-files/REQUIRED/script1/.s /mnt/z/proxy/.s
sleep .5
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        set  environment
cd /mnt/z/;
setenv RESORT_0 /mnt/z/proxy;
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        make system directories
mkdir -p $RESORT_0/dev
mkdir -p $RESORT_0/etc
mkdir -p $RESORT_0/usr/ports
mkdir -p $RESORT_0/usr/src
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        installworld
cd /usr/src/
make installworld DESTDIR=$RESORT_0 # SRCCONF=/etc/src.conf
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        mergemaster
sleep 2
mergemaster -a -C -D ${RESORT_0}
sleep 2
cd /usr/src/etc
cp /etc/resolv.conf $RESORT_0/etc
cp /etc/mnt/j/MAKE.conf $RESORT_0/etc
# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        distribution
sleep 2
make distribution DESTDIR=$RESORT_0 OPTIONS_UNSET=OPENSSH OPTIONS_UNSET=OPENSSL
sleep 2
rm -Pf $RESORT_0/etc/ssl/openssl.cnf
mkdir -p $RESORT_0/usr/local/openssl
cp /etc/ssl/openssl.cnf $RESORT_0/etc/ssl
cd $RESORT_0/usr/local/openssl/
sleep 2
ln -s ../../../etc/ssl/openssl.cnf openssl.cnf

# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        delete temproot on host

cd /var/tmp/temproot && chflags -R noschg * && rm -Prf * && cd .. && rm -Pr temproot

# . . . . . . . . . . . . . . . . . . . . . . . . . . . .        customize - add/remove
mkdir /mnt/z/proxy/usr/local/www
rm -Pr /mnt/z/proxy/sys

cp -pr /mnt/j/MAKE/proxy/config-files/with-dns/dhclient-enter-hooks /mnt/z/proxy/etc/dhclient-enter-hooks
cp -pr /mnt/j/MAKE/proxy/config-files/with-dns/dhclient.conf /mnt/z/proxy/etc/dhclient.conf
cp -pr /mnt/j/MAKE/proxy/config-files/with-dns/resolv.conf /mnt/z/proxy/etc/resolv.conf
cp -pr /mnt/j/MAKE/proxy/config-files/ssh/sshd_config /mnt/z/proxy/etc/ssh/sshd_config

cp -pr /mnt/j/MAKE/proxy/config-files/periodic.conf /mnt/z/proxy/etc/periodic.conf
cp -pr /mnt/j/MAKE/proxy/config-files/rc.shutdown /mnt/z/proxy/etc/rc.shutdown
cp -pr /mnt/j/MAKE/proxy/config-files/sysctl.conf /mnt/z/proxy/etc/sysctl.conf
cp -pr /mnt/j/MAKE/proxy/config-files/syslog.conf /mnt/z/proxy/etc/syslog.conf
cp -pr /mnt/j/MAKE/proxy/config-files/src.conf /mnt/z/proxy/etc/src.conf
cp -pr /mnt/j/MAKE/proxy/config-files/services /mnt/z/proxy/etc/services
cp -pr /mnt/j/MAKE/proxy/config-files/rc.conf /mnt/z/proxy/etc/rc.conf
cp -pr /mnt/j/MAKE/proxy/config-files/hosts /mnt/z/proxy/etc/hosts
sleep 1
ln -s /mnt/z/proxy/usr/local /mnt/z/proxy/Proxy_Local
cd /

echo "......."
echo " DONE. "
sleep 1000
exit


# FOR BIG-JAIL TO COMPLETE:  if folder is full = 4.2 min - if empty = 3 min
#      
# make buildworld
# make installworld DESTDIR=$RESORT_0
```

*b_delete.cell*

```
#!/bin/sh
# ................................................     delete PROXY jail
#
echo "deleting PROXY jail. 91 seconds to complete."

cd /mnt/z/proxy && chflags -R noschg * && rm -Prf *

echo "......."
echo " DONE. "
sleep 3
```

*c_untar.cell*

```
#!/bin/sh
# ................................................  reinstall PROXY jail
#
tar jxvf /mnt/j/MAKE/proxy.tgz -C /mnt/z/

echo "42 seconds"
echo ".........."
echo "   DONE   "
sleep 3
```

*d_start*

```
#!/bin/csh
# ................................................      Start PROXY jail
#
mount_nullfs -o /dev/ada0s1f /usr/ports /mnt/z/proxy/usr/ports;
mount_nullfs -o /dev/ada0s1f /usr/src /mnt/z/proxy/usr/src;
sleep 1;
setenv RESORT_0 /mnt/z/proxy
cd $RESORT_0
mount -t devfs devfs /mnt/z/proxy/dev
devfs -m /mnt/z/proxy/dev rule -s 4 applyset
devfs -m /mnt/z/proxy/dev rule apply path tun0 unhide

ln -s dev/null kernel
jail $RESORT_0 proxy.cell 10.0.0.1 /bin/sh
```

*e_kill-environment*

```
#!/bin/csh
# ................................................      Kill Environment
#
umount -A -t nullfs
umount -A -t devfs
sleep 2
unsetenv RESORT_0
unsetenv RESORT_1
unsetenv RESORT_2
unsetenv RESORT_3
unsetenv RESORT_4
unsetenv RESORT_5

echo "-----";
echo "-- jail is now Close!";
#cat /dev/null > /mnt/z/site1/var/log/nginx-access.log;
#cat /dev/null > /mnt/z/site1/var/log/nginx-error.log;
#cat /dev/null > /mnt/z/site1/var/log/nginx-header.log;
#cat /dev/null > /mnt/z/site1/var/log/php-fpm.log;
echo "-----";
echo "Logs are Clean";
echo "-----";
echo "-----";
sleep 3
```

*tar-it*

```
#!/bin/sh
# ................................................          Backup PROXY
#
echo "backing up PROXY. 74 seconds to complete."

rm -Pr /mnt/j/MAKE/proxy.tgz; sleep 2;

cd /mnt/z && tar cvzf /mnt/j/MAKE/proxy.tgz proxy/

echo "......."
echo " DONE. "
sleep 3
```

*I once had it then I lost it. So, I had other things to do.
view-env*

```
#!/bin/csh
# ................................................         View Commands
#
cd $RESORT_0
printenv $RESORT_0 /mnt/z/proxy

sleep 999
exit
```

*z_RemoveFiles*

```
#!/bin/sh
# ................................................ PROXY jail:
# ................................................ Remove un-needed files
rm -Pr /mnt/z/proxy/mnt
rm -Pr /mnt/z/proxy/net
rm -Pr /mnt/z/proxy/proc
rm -Pr /mnt/z/proxy/boot
rm -Pr /mnt/z/proxy/mntdia
rm -Pr /mnt/z/proxy/rescue
rm -Pr /mnt/z/proxy/etc/X11
rm -Pr /mnt/z/proxy/etc/zfs
rm -Pr /mnt/z/proxy/etc/motd
rm -Pr /mnt/z/proxy/etc/hosts
rm -Pr /mnt/z/proxy/etc/bluetooth
rm -Pr /mnt/z/proxy/etc/sysctl.conf
rm -Pr /mnt/z/proxy/etc/periodic.conf
rm -Pr /mnt/z/proxy/etc/pkg/FreeBSD.conf

#rm -Pr /mnt/z/proxy/etc/syslog.conf

rm -Pr /mnt/z/proxy/usr/obj
rm -Pr /mnt/z/proxy/usr/src
rm -Pr /mnt/z/proxy/usr/ports
rm -Pr /mnt/z/proxy/usr/tests

#rm -Pr /mnt/z/proxy/usr/share/openssl
rm -Pr /mnt/z/proxy/usr/share/examples
rm -Pr /mnt/z/proxy/usr/share/firmware
rm -Pr /mnt/z/proxy/usr/share/games
rm -Pr /mnt/z/proxy/usr/share/doc
rm -Pr /mnt/z/proxy/usr/share/man
rm -Pr /mnt/z/proxy/usr/local/man

cp -p /mnt/j/MAKE/proxy/config-files/hosts /mnt/z/proxy/etc/hosts
cp -p /mnt/j/MAKE/proxy/config-files/rc.conf /mnt/z/proxy/etc/rc.conf
cp -p /mnt/j/MAKE/proxy/config-files/sysctl.conf /mnt/z/proxy/etc/sysctl.conf # useless so far
cp -p /mnt/j/MAKE/proxy/config-files/resolv.conf /mnt/z/proxy/etc/resolv.conf
#cp -p /mnt/j/MAKE/proxy/config-files/syslog.conf /mnt/z/proxy/etc/syslog.conf
cp -p /mnt/j/MAKE/proxy/config-files/periodic.conf /mnt/z/proxy/etc/periodic.conf
cp -p /mnt/j/MAKE/proxy/config-files/FreeBSD.conf /mnt/z/proxy/etc/pkg/FreeBSD.conf

echo "......."
echo "......."
echo " DONE. "
sleep 3
```

*.q = you MUST run this first to close terminal and such.
 after closing run .e_kill-environment so /dev don't get stuck*

```
#!/bin/sh
#  ..............         STOP all then quit, don't forget to kill-environment. I have
#  ..............         not figure a way to include it?  This terminal must be closed.
service cron stop
service syslogd stop
service sshguard stop
/usr/bin/pkill sshd
/usr/bin/pkill nginx
/usr/bin/pkill ld-elf
/usr/bin/pkill syslogd
/usr/bin/pkill ld-elf32
/usr/bin/pkill cron
/usr/bin/pkill clean_var
rm -Pr /var/run/*

kill -TERM -1
kill -KILL -1

#/usr/local/etc/rc.d/varnishd onestop;   sleep .5;
#/usr/local/etc/rc.d/nginx onestop;      sleep .5;
#/usr/local/etc/rc.d/php-fpm onestop;    sleep .5;
#/usr/local/etc/rc.d/mysql-server onestop;       sleep .5;
#
#cat /dev/null > /var/log/nginx-access.log
#cat /dev/null > /var/log/nginx-error.log
#cat /dev/null > /var/log/nginx-header.log
#cat /dev/null > /var/log/php-fpm.log
```

*.s = run this after jail termial is open.*

```
#!/bin/sh
#  ..................... System Varables
sh /etc/rc
sleep 1
exit
```
[/b]


----------



## max21 (Dec 1, 2017)

# #####################################################
# #####################################################
# It seems that all of this runs in jail. sysctl makes
# make rc.conf tick but it dose nothing else for me, but
# it must be there even if empty or else jail don't work
# #####################################################
# #####################################################

If you want to see some action, go to
/var/run You will like it a lot inside there.
 I wipe it completely out for fun just to see it all
come back.  This is how I know when I screw something
up, elsewhere.  I compair it to host /var/run for a doulble check,
but his count is largers.  Well that's all the debugging you will ever need.

*Add to jail /etc/rc.shutdown*

```
# Insert other shutdown procedures here
service cron stop
service syslogd stop
service sshguard stop
/usr/bin/pkill sshd
/usr/bin/pkill nginx
/usr/bin/pkill ld-elf
/usr/bin/pkill syslogd
/usr/bin/pkill ld-elf32
/usr/bin/pkill cron
/usr/bin/pkill clean_var
#rm -Pr /var/run/*
```

*dhclient.conf*

```
supercede domain-name-servers 69.164.196.21, 96.90.175.167; # OpenNIC - sDice
```

*dhclient-enter-hooks*

```
# I use it on host and jails SirDice.
add_new_resolv_conf() {
        # We don't want /etc/resolv.conf changed
        # prevents dhclient from touching it
        return 0
}

#add_new_routes() {
        #route add -net 10.0.0.138 -iface $new_ip_address
        #route add default 10.0.0.138
#}
```

*resolv.conf*

```
#    OpenNIC  
nameserver 69.164.196.21
nameserver 96.90.175.167
```

*rc.conf*

```
network_interfaces=""
hostname="proxy.ka.cell"
ifconfig_em0_alias0="inet 10.0.0.1 netmask 255.255.255.255"

inetd_flags="-wW -a 10.0.0.1"

tcp_drop_synfin="YES"
icmp_drop_redirect="YES"
icmp_log_redirect="NO"

rpcbind_enable="NO"
cron_enable="YES"
cron_flags="$cron_flags -J 15"

sshd_enable="YES"

ip6addrctl_enable="NO"
ip6addrctl_policy="ipv4_prefer"
ipv6_activate_all_interfaces="NO"
auto_linklocal="NO"

virecover_enable="NO"
haproxy_enable="YES"
```

*put this in the hosts rc.conf, and install sshguard in jail too.


		Code:
	

# The boom-boom:  I thank SirDice for this one too.  I love watching it work.  Hope I did not mess up the pasting.
#
sshguard_watch_logs="/mnt/z/proxy/var/log/auth.log:/mnt/z/site1/var/log/auth.log:/mnt/z/site2/var/log/auth.log"


hosts


		Code:
	

#::1                    localhost
127.0.0.1               localhost
10.0.0.1        proxy proxy.ha.cell
#
96.47.72.71             pkg.freebsd.org pkg


periodic.conf


		Code:
	

# ........................................  ls -l /etc/periodic/*
# ........................................  self-create, touch and chmod 600
# ........................................ /etc/periodic.conf
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"
#
##......................................................................  max21 search
#
daily_backup_pkgdb_enable="NO"
daily_status_mailq_enable="NO"

hourly_output="root"                # user or /file
weekly_show_success="NO"            # scripts returning 0
weekly_show_info="YES"                # scripts returning 1
weekly_show_badconfig="NO"            # scripts returning 2

.
.
.*


----------



## Snurg (Dec 1, 2017)

Thank you, I see we have somewhat different approaches.
I must admit that I probably never will learn programming using sh. Even simple conditionals for error handling are so difficult and need so much tedious typing, and the results are so hard to understand. I already have succumbed to my laziness and will stick with Perl. As a good Unixoid FreeBSD has Perl in the base, so I don't see any reason not to use it.

And, what I do not understand yet: Are there other reasons aside of deleting the temp and log files etc, for having extra scripts for starting and stopping jails? I mean, why not just `service jail start myjail` etc?

On the other hand, it would be probably much better to make sure that all important services get shut down cleanly (`service nginx stop`, for example) instead of just killing them. So I should take more care of rc.shutdown than in the past. Thank you for pointing me at that!

And what's really sweet, too, I learned from you how to prevent `dhclient` from tampering with resolv.conf. That had often annoyed me in the past.

A suggestion though: Removing configuration lines for disabled things like sendmail and inetd makes the config shorter and easier to read.


----------



## SirDice (Dec 1, 2017)

Snurg said:


> As a good Unixoid FreeBSD has Perl in the base


Perl has been removed from the base since FreeBSD 5.0. It's often installed though, typically as a dependency of some other port (I'm a heavy Perl user myself  ).


----------



## max21 (Dec 2, 2017)

Snurg said:


> Thank you, I see we have somewhat different approaches.
> I must admit that I probably never will learn programming using sh. Even simple conditionals for error handling are so difficult and need so much tedious typing, and the results are so hard to understand. I already have succumbed to my laziness and will stick with Perl. As a good Unixoid FreeBSD has Perl in the base, so I don't see any reason not to use it.
> 
> And, what I do not understand yet: Are there other reasons aside of deleting the temp and log files etc, for having extra scripts for starting and stopping jails? I mean, why not just `service jail start myjail` etc?
> ...



No brag, but there is nothing I don’t know about what I done or what seems not so wise to do.  I just keep stuff separated until I find the time to play with it again.  I presented it to you to help provide some ideas for your real HOW-TO.  Just dedicate to it for a week or two then all of your question will be answered.  Pay attention to my tiny notes.  Afterwards you know exactly what you can do with it - - you are not bound by the system anymore - - you got an opportunity to piece it all together to suit your needs, afterwards.  For me, I got more testing to do such as delete-unwanted-file down to the bone, that will work with all future versions of FreeBSD.  Sure src.conf can do most of that but who want to installworld--jail after each and every failed test.  That is only one example.  Once I’m done all I need to do is automate these scripts, and if another script can’t run it … C++ will.  Right now it’s all about the web for me.

Making everything disappear when I shutdown is one of the many things of what it’s all about.  It ensure me that there is nothing left to find and see.  And above all it's amazing to have it all come back again like the jail was created for the first time.  See my friend, there is a reason for everything here.

jail.conf and rc.conf will NOT complete job, all the time, or ever again after it get locked in by the system by those conf files.

Now you know why people been having jail issues for years … the darn thing got stuck (is my guess)!

For example, use your own [filejail.conf[/file].  You must use ONESTART/ONESTOP because this jail is standalone to the bone.  This only happen most of the time, and it can happen at the wrong time as seen below.  It happen after the second onestop, then onestart.  Now how many people even thought to check?  That is what you got to do until you know for sure.  That’s why I say dedicate for at least 2 week and keep your eyes open.

Start the jail and shut it down a few times.  Afterwards, you will find all of those devices still in there.  So if you had 1000 jails turn-off after use, you might have 9000 links to the kernel floating around all over the place, including null-mounts that were once hidden, now  wide open.  And to make the matter even worse, you got to reboot to recover to test anything jail related again.  That is not secure.  Whatever it is, I don’t want it, so I start thinking out-side the box a long time ago.  I don’t care that it don’t look pretty,  lt runs like a plain jail, but I have near _TOTAL_ control, per se.

For example:
*This is what jail.conf leaves you:*

```
(~) service jail onestart proxy
Starting jails: proxy.
(~) 
(~) service jail onestop proxy
Stopping jails: proxy.
(~) cd /mnt/z/proxy/dev
(/mnt/z/proxy/dev) ls
fd      null    pts     random  stderr  stdin   stdout  urandom zero
(/mnt/z/proxy/dev)
. . . 
. . . and this is all you get even when it is
. . . generating the ssh_host_keys for the first time.

.  . .NOTE the left-over devices
```

*Now this is what you get with the scripts that I posted:*

```
# ./.s
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Setting hostname: proxy.ka.cell.
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Generating RSA host key.
2048 SHA256:3RMWM05PiroGyPlFlw+oOrtDlPKEHi+FBg9MGakSamE root@proxy.ka.cell (RSA)
Generating ECDSA host key.
256 SHA256:6noOz+g3jBbWHBNFaWXS5nicu2Om2wEmIhdLUEAg/CE root@proxy.ka.cell (ECDSA)
Generating ED25519 host key.
256 SHA256:noe/Dmt8gK2GedKaMQtBrMWYNcUWNhfJLRdJcFjIzBE root@proxy.ka.cell (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.

Sat Dec  2 05:59:58 UTC 2017

#
. . .
. . .
I bet more than 99% of all jails users never
saw this before!  It’s running like a real system.
```

But you got to make one extra call kill it all.
The question is; Is it worth it for the unknowing user?

./.q = shutdown terminal
./.e_kill-environment

Also, don’t forget to use this.  I forgot to post it.  Now FreeBSD jail-system don’t need to do anything.  It’s the kernel job to follow orders:


```
/root/.cache
setenv	EDITOR	vi
setenv	PAGER	more
setenv	BLOCKSIZE	K

setenv RESORT_0  < -- here
setenv RESORT_1  < -- Now it's buried so
setenv RESORT_2        deep even beastie can't find it

if ($?prompt) then
```
I don't know what FreeBSD really does, but it's sure fun trying to figure him out 



... but, still, trying to explain all the details about this jail all night long done gotten me confused.  I’ll post it in the HOW TO section once complete and properly worded (that's the hard part), someday.  Thanks for the tip about rc.conf Snurg

Back to solution-1 for me

See ya latter


----------



## max21 (Dec 4, 2017)

Snurg said:


> ....
> ....
> . . . On the other hand, it would be probably much better to make sure that all important services get shut down cleanly (`service nginx stop`, for example) instead of just killing them. So I should take more care of rc.shutdown than in the past. Thank you for pointing me at that!



I just realize I did not answer to this seemly kind of a question, _As usual_, I wondered off into everything else trying to figure out what was actually being ask.  Anyway, the reason I kill off most or all applications is because they are only suppose to be jailed processes, _virtual-like_.  However, I do cleanly shutdown any process that may affect the host.  “iirc”, I did indicate all of this is work in progress ...  and I learn that there is nothing wrong if you kill a process as long as you clean-up behind it then flush-it if you find any leftover and if you do, you'll be glad you found it.  My goal was to know for a fact at how far one can go without affecting the host.  The only thing I had to leave along was sshguard because of the special function for it is in the host rc.conf.  That’s all I needed to know.  Now that I have experienced all of this  , I might find a reason why to use it in production.

```
service cron stop
service syslogd stop
service sshguard stop
/usr/bin/pkill sshd
/usr/bin/pkill nginx
/usr/bin/pkill ld-elf
/usr/bin/pkill syslogd
/usr/bin/pkill ld-elf32
/usr/bin/pkill cron
/usr/bin/pkill clean_var
rm -Pr /var/run/*
```
What surprise me even more was that you can totally wipe out /var/run and all that other stuff and I still never have a problem.  And as far as cleaning, It don’t get better then rm Pr, then flushing out the entire environment to close the rest of the shell.  The only thing I might save would be the logs.  It’s worth it.


Good grief!  It’s weird that I now come to realize what you meant by “_I see we have somewhat different approaches._”  I swear, I really didn’t completely understand.  Here you were ready to write about the relationship between jail.conf and the rc.conf and I been not using either for a very long time without knowing it .. with the exception of sshguard, until now.  And to speak of missing the mark.  Dang!  No wonder why I been tripping.  For those who think I’m slower than molasses ..  I agree, but you got to admit I been doing a little bit better lately.



more about HAProxy:

Can a Reverse Proxy use SNI with SSL pass through?

https://serverfault.com/questions/625362/can-a-reverse-proxy-use-sni-with-ssl-pass-through


----------



## Snurg (Dec 4, 2017)

I still prefer to shutdown programs the normal way. I don't like, for example, if I cannot clearly determine from the logs whether a program has exited abnormally (problem) or was just killed (no problem) if the program shutdown messages are missing in the logs.

Regarding the idea of a howto, I realize it makes much more sense to do the thing another way around.
Just the sheer quantity of files needing to be touched on the host side only when one creates or deletes a jail: /etc/rc.conf, /etc/jail.conf, /etc/pf.conf, /etc/fstab, occasionally also /etc/hosts, /etc/haproxy.conf.
On the jail side there are also a number of files: /etc/rc.conf, /etc/fstab, optionally also /etc/hosts, /etc/jail.conf.

This is about a dozen files minimum. And then it gets even more complicated. If you do not only want to clone base system jails, but preconfigurable jails for, say, web servers or even web application servers like forum servers.
You then need to handle even more files, depending on the kind of jails you want to clone. If you want to clone, say, applications like BBs that use, say, SQL servers, you'll even need to provide a seamless way to configure the cloned jails from inside. For all that I use a sort of "plugin concept" so one can configure his clone jails using selectable modular sets of config scripts.

And I want all that usable from either fully console interactive-dialog-based to fully automated operation (via scripts), for example to administer the whole stuff via CGI apps some time later (essential if you want to host).

*You see, this whole configuration chore is incredibly cumbersome and error-prone. I do not like the idea doing that manually.* But to be able to see if my script work correctly, I only use it in simulation mode. So I get a complete list of actions, like a custom-generated shell script, combined with the list of file modifications, that I can check out and refine until I am satisfied, so I can do the first real tests, first in a jail, and finally on my real systems. And when this works, I can release the stuff to the public.

The output also can serve in seeing what all needs to be touched when administrating such stuff. The few insane people like me, you and some others here who want to learn more of the "inner workings" can study these outputs and get the idea. The more normal people, who just want to get it work, just can use the easy-to-use frontends (command line or even easier with CGI). I think this is a better approach.
And if you ask yourself why I do this. Having this done is just a prerequisite for my main project, so I must do that anyway. And why not share it, when it could profit the FreeBSD community)


----------



## max21 (Dec 5, 2017)

Snurg said:


> I still prefer to shutdown programs the normal way. . . .


How could daemons possibly agree with each other?  I guess that is why beastie is in charge.

Your post is very, _very_ informative, and for production, as planned, I will be changing the way I shutdown applications among many other things that you speak of even though there are many other ways to detect how a application or the jail itself was shutdown.  I am not anti-jail.conf.  I got a jail.conf to run the same jail and I build on that jail.conf for what it can accept so to match the untouchable jail;  outside of rc.conf, pf.conf, jail-fstab and sysctl.conf, on the host, as I go.  I thought I mention that much earlier.  It’s like Double Mint Gum, it has two, two kicks in one.

As far a jails are concern, I’m in no rush.  I have all winter to prefect it now that I know what it needs, at the very least to match your requirements.  Thanks for all of that.  If you do document your HOW TO could you please add some information about what sysctl calls can works inside the jail rc.conf.  I already know that a jail would/could use the host sysctl, but my jail might not, however I do know that a jail rc.conf could be used to call some of those sysctl functions.  I want to know what are some of the common ones needed, and how to test that it works, regardless.  All I get are error warning with sysctl.conf inside a jail with _anything_ in it.  From what I gather, jails are required to have /etc/sysctl.conf there or else rc.conf will not work completely and  syslogd will not show up inside /var/run.  I learn these things only by making everything disappear at shutdown time (total-teardown), then a full-first-run-boot.  This could be just because of the way I build jails, but I doubt that.  jail.conf prove that they both are of the same family.



Snurg said:


> On the jail side there are also a number of files: /etc/rc.conf, /etc/fstab, optionally also /etc/hosts, /etc/jail.conf.


I did not know a jail could have a jail.conf.  Did you mistakenly include it?



Snurg said:


> . . . on the host side only when one creates or deletes a jail: /etc/rc.conf, /etc/jail.conf, /etc/pf.conf, /etc/fstab, occasionally also /etc/hosts, /etc/haproxy.conf.



About HAProxy, it seems that you have it installed on the host.  Are there any pros and cons vs HAProxy being installed in jail?  I been planning to use it while jailed.

Here I found a very important point when it comes to learning HAProxy SNI.


> HAProxy HTTPS setups can be a little tricky. So make sure you have a working one first before adding SNI to the mix.


https://stuff-things.net/2016/11/30/haproxy-sni/


----------



## Snurg (Dec 5, 2017)

max21 said:


> did not know a jail could have a jail.conf.  Did you mistakenly include it?


I want to explore jail in jail  Like the traditional Russian figurines.







max21 said:


> About HAProxy, it seems that you have it installed on the host.


No. All what is needed on the host is the base system and Perl. All other stuff jailed.
Atm I have almost finished the basic functions of the script and expect to do the first dry runs today or tomorrow.
So I need only install FreeBSD and Perl, and then run the script in automatic mode to configure the jails system and PF, and set up haproxy and DNS jails for me. Giving me a more preconfigured base ready to start the whole web stuff instantly, without need to do the same configuration chores again.


----------



## max21 (Dec 6, 2017)

Snurg said:


> I want to explore jail in jail  Like the traditional Russian figurines. ...
> . ...
> No. All what is needed on the host is the base system and Perl. All other stuff jailed.
> Atm I have almost finished the basic functions of the script and expect to do the first dry runs today or tomorrow.
> So I need only install FreeBSD and Perl, and then run the script in automatic mode to configure the jails system and PF, and set up haproxy and DNS jails for me. Giving me a more preconfigured base ready to start the whole web stuff instantly, without need to do the same configuration chores again.



That did it!  I was about to send another gigantic post seeking verification about a complete jail install. … So,all I need to do is include only this part:

After more research, It seem that there are two types of jails.. The technical terms might be jails and jailed-shell.  Also, someone at FreeBSD pipermail explained the complete differences, and pounded it in.  I wish I could re-find that post but I can’t.

Well, its time to get started

Thanks Snurg

Thanks SirDice

I should be-able to walk right up on it.  No time to monk around, I'm going to use my _jail.conf_ and learn some Perl.  

Hey, I just thought, it ain't over until the fat lady sing.  Can't wait to read more about your project.  Good Luck.  Need anything done, let me know.


----------



## max21 (Dec 6, 2017)

> . . . as long as you have routable public IP space you can easily route it through the firewall and keep the publics right on the proxies.


This is what this thread was supposed to be about before the installation problem bit-me,  haproxy_behind_a_firewall.  I’m glad that happened.

*haproxy-devel-1.8.1.txz*
JIT -  2017-Dec-05 19:30
.


----------



## max21 (Dec 6, 2017)

That’s the way it goes:

*haproxy-1.7.9*
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy
All previous versions produce the same error, including the port version.

*haproxy-devel-1.8.1.txz*
/usr/local/sbin/haproxy: Undefined symbol "stat@FBSD_1.5"
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy


Any ideas at what’s going on?


----------



## Snurg (Dec 7, 2017)

No idea. I just did `pkg -j haproxy <haproxypkg>`.
I used the current production (not devel) version I found listed on freshports.
The jail's rc.conf contains these haproxy-related lines:

```
hostname="haproxy"
haproxy_enable="YES"
haproxy_flags="-f /etc/haproxy.conf -d"
```


----------



## max21 (Dec 7, 2017)

pkg -j haproxy <haproxypkg> don’t work for me.  It’s probably a Perl thing.  Even without the <haproxypkg>  I thought it should had work but it didn’t.  I don’t expect anyone to know, especially about the new version. It’s suppose to be their best version ever but it seem to me they worte haproxy for Linux and screwed it up for FreeBSD.  Now we get this ---> … *stat@FBSD_1.5* is NO where to be found, absolutely NO-WHERE, and NOTHING no where near about it, either.  FreeBSD might have to reverse-engineer it just to make it work… evidently, they didn’t do something right or not document it so that we know what to do.

I’m going to go to my friend home tomorrow, he has DSL.  I got Comcast cable and I don’t trust cable like I do DSL.  For years when I clicked the Thank you button, I get a BIG popup WARNING: you’re about to … do you really want to do that ….  Heck no, so I shut it down.

But when I just did that for you guys, I did not care anymore, and when I clicked the Thank you button, both times, for the very first time … there was not pop-up’s.  So, this really make me thing that I was right that there is a man-in-the-middle and he knows this is the only place that I go (a member of) so it’s automatic I guess.  My XP vbox can’t be hacked anymore, but whoever got in already, at least he know he can only go so far.  He use to could turn-off my firewall until I got wise.  I saw how Windows svchost.exe was being use to hack the heck out of me.  It was so funny when I caught it (like peep-ka-boo).  I’m not going to TRIP.

SVN could be messed up on my FreeBSD vbox.  I’m thinking of doing a standard install.  I forgot about this link **sniproxy**.  I'm still glad.. if I did not learn all the above, I would be wasting months to years knowing nothing, once installed.  Just like VPN client, over 3 months, off and on, down the drain, trying.

https://serverfault.com/questions/625362/can-a-reverse-proxy-use-sni-with-ssl-pass-through


```
# pkg install haproxy

Updating FreeBSD repository catalogue...
….
….
New packages to be INSTALLED:

          haproxy: 1.7.9
…
…
[haproxy.ka.cell] Extracting haproxy-1.7.9: 100%
 
# service haproxy start

/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy

#
```


----------



## max21 (Dec 7, 2017)

sniproxy don't seem right.  It might work but haproxy might too if I got to go through all of that.  Looks like I'm going to have to take it to the top starting at the bottom of this page and learn something.

https://www.haproxy.org/download/1.6/doc/management.txt


----------



## max21 (Dec 7, 2017)

SVN forgot something, or buildworld skipped it, or my make.conf has too much clang.. or the system burped.

https://www.haproxy.org/download/1.6/doc/management.txt


> HAProxy is designed to isolate itself into a chroot jail during startup, where it cannot perform any file-system access at all. This is also true for the libraries it depends on (eg: libc, libssl, etc).



No wonder nothing important work out the box since 10.2p7. If I stuck with ports I would had detected something by now.  I wonder would installing something like devel/libcxxrt do the trick.  Maybe it was the cause of the debug code not working.  Now I need to find out all of haproxy eg: libc, libssl, etc - etc -etc.  I show like to know.....  I’ll never *delete-old-lib* ever again!


*buildworld*

```
===> lib/libc (install)

install  -C -o root -g wheel -m 444   libc.a /mnt/z/proxy/usr/lib/
install  -C -o root -g wheel -m 444   libc_p.a /mnt/z/proxy/usr/lib/
install: libc_p.a: No such file or directory

*** Error code 71

Stop.

make[5]: stopped in /usr/src/lib/libc

*** Error code 1

make[4]: stopped in /usr/src/lib

make[3]: stopped in /usr/src

make[2]: stopped in /usr/src

make[1]: stopped in /usr/src

make: stopped in /usr/src

*** Creating the temporary root environment in /var/tmp/temproot
```

*/etc/make.conf*

```
# ...........................................      make.conf
MAKE_JOBS_NUMBER?=8
CPUTYPE?=native
OPTIONS_SET=OPTIMIZED_CFLAGS
WITH_CLANG=yes
WITH_CLANG_FULL=yes
WITH_CLANG_EXTRA=yes
WITH_CLANG_IS_CC=yes

cc=/usr/local/libexec/ccache.world/clang
cc=/usr/local/libexec/ccache.world/clang++
cc=/usr/local/libexec/ccache.world/clang-cpp

BUILD_STATIC=1

DEFAULT_VERSIONS+=ssl=libressl
DEFAULT_VERSIONS+=php=7.1

WRKDIRPREFIX=/ram
```

Time to find libc, rebuild the system, or downgrade.  Better to find out now then never


----------



## SirDice (Dec 7, 2017)

Remove everything except DEFAULT_VERSIONS and WRKDIRPREFIX. Then try again.


----------



## max21 (Dec 9, 2017)

I followed your instrustions to a tee.  Same result.  I even deleted everything using rm –Pr and chflags command - - /ram, /usr/obj, /usr/src and /usr/ports, including old files under /var/cache/pkg and /var/db/pkg and ports.  I downloaded the src and ports for FreeBSD-11.1 and did a complete buildworld.  I delete-old-files and I  delete-old-lib.  At least I know I got a perfect FreeBSD-11.1 install:

*First try fail:*

```
cd /usr/ports/net/haproxy
make –DBATCH install clean
Errors:
libiconv-1.14_11
/usr/local/sbin/pkg-static: not found
make[2]: stopped in /usr/ports/converters/libiconv
make[1]: stopped in /usr/ports/devel/gmake
make: stopped in /usr/ports/net/haproxy
```

So I installed the pkg versions of pkg, libiconv and gmake so to let ports knows that something is there.  Interesting enough, it cleared up that situation.  I experienced this before already.

```
pkg install pkg
pkg install libiconv
pkg install gmake
```

Then I tried to reinstall the port version of haproxy and this is what I got:

```
cd /usr/ports/net/haproxy
make –DBATCH install clean
…
                               ^~~~~~~~~~~~~~~~~~
59 warnings generated.
gmake[1]: Leaving directory '/ram/usr/ports/net/haproxy/work/haproxy-1.7.9/contrib/halog'
===>  Staging for haproxy-1.7.9
===>   haproxy-1.7.9 depends on file: /usr/local/lib/libcrypto.so.42 - found
===>   Generating temporary packing list

http://www.haproxy.org
===>  Cleaning for gmake-4.2.1_1
===>  Cleaning for libiconv-1.14_11
===>  Cleaning for libressl-2.6.3
===>  Cleaning for pkgconf-1.3.10,1
===>  Cleaning for haproxy-1.7.9
(/usr/ports/net/haproxy)
```
_see how it took._

*TEST IT:*

```
(~) service haproxy onestart
/usr/local/etc/rc.d/haproxy: WARNING: /usr/local/etc/haproxy.conf is not readable.
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy
(~)
(~)
(~) service haproxy onestart
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy
(~)
```

*Lets see if haproxy is in there:*

```
(~) haproxy -vv
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : LibreSSL 2.6.3
Running on OpenSSL version : LibreSSL 2.6.3
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available filters :
   [SPOE] spoe
   [TRACE] trace
   [COMP] compression
```


```
(~) haproxy -c
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N <maxpconn> ]
        [ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*]
        -v displays version ; -vv shows known build options.
        -d enters debug mode ; -db only disables background mode.
        -dM[<byte>] poisons memory with <byte> (defaults to 0x50)
        -V enters verbose mode (disables quiet mode)
        -D goes daemon ; -C changes to <dir> before loading files.
        -q quiet mode : don't display messages
        -c check mode : only check config files and exit
        -n sets the maximum total # of connections (2000)
        -m limits the usable amount of memory (in MB)
        -N sets the default, per-proxy maximum # of connections (2000)
        -L set local peer name (default to hostname)
        -p writes pids of all children to this file
        -dk disables kqueue() usage even when available
        -dp disables poll() usage even when available
        -dG disables getaddrinfo() usage
        -dR disables SO_REUSEPORT usage
        -dr ignores server address resolution failures
        -dV disables SSL verify on servers side
        -sf/-st [pid ]* finishes/terminates old pids.

(~)
```

I am sick of thinking negative.  It makes me wonder if anti-Net Neutrality has anything to do with it.  Even IP providers has the right to intervene according to this new law, if already in effect.  Other then that I don’t know what to think.  It’s suppose to just work.  I even tried it on FreeBSD-10.2 and that produce the same warning.  It has no business compiling with so many errors _warning_ for such a tiny file.

It makes no since to keep asking more about it.  I’m just going to have to read until I figure out how the entire thing works and what I’m missing -- and I will. I might find another way to start it.  Thanks for all.


----------



## Snurg (Dec 9, 2017)

As you experienced the same behaviour on 10.2, too, that hints that maybe there is some overlooked configuration step to be done.
I remember that I had to do some manual testing (starting haproxy manually in the jail using the console until I found out how to start it up correctly), but do not remember the details what I did.


----------



## SirDice (Dec 13, 2017)

The error is fairly obvious why it's not starting, there is no /usr/local/etc/haproxy.conf. There is no default configuration file so you'll have to create one from scratch. The service simply refuses to start if it can't read its configuration file or if there's a configuration error (that's the function of the precmd part of the rc.d(8) script).


```
(~) service haproxy onestart
/usr/local/etc/rc.d/haproxy: WARNING: [b]/usr/local/etc/haproxy.conf is not readable[/b].
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy
```


----------



## max21 (Dec 18, 2017)

SirDice said:


> The error is fairly obvious why it's not starting, there is no /usr/local/etc/haproxy.conf. There is no default configuration file so you'll have to create one from scratch. . . . .



I spend the entire week cleaning house.  It been weird being away for the old computer this long but it did me good.  I understood you guys replies and I could not wait get back.  Today, I really got into the haproxy configuration.txt so deep until your examples haproxy.conf  became understandable, masterpieces, my haproxy bibles.  Now I’m only inches away.  I think I done got greedy already.  Now I know what haproxy can do for nginx and what nginx can do for apache. Soon I’m going to find out if haproxy can do what nginx reverse-proxy can do for apache child processes (rendering them useless)…  I suspect haproxy can do that too.  Youtube?  … never knew what I was missing until now.  There is little to nothing about haproxy as reverse-proxy, but these been very helpful.  There is nothing like hearing it live.  I imagine all the tech you guy talk at the 9-5.

Nginx vs Apache part in the middle:





About child processes at beginning:





Let's Encrypt through HAProxy





*Prove that it works first.*

```
global

        daemon
        maxconn 256

    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    listen http-in
        bind *:80
        server site1 127.0.0.1:8000 maxconn 32
```

*It WORKS!*

```
# /usr/local/etc/rc.d/haproxy onestart

Starting haproxy.

Available polling systems :

     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK

Total: 3 (3 usable), will use kqueue.

Available filters :

          [SPOE] spoe
          [TRACE] trace
          [COMP] compression

Using kqueue() as the polling mechanism.
```

_Finally_… but I bet it would have took a full semester to get this far anyway.

edit--

You got to know this too.

I forgot to mention that I had pieced together quite a few haproxy.conf files, then after Snurg evaluation and SirDice calculation I knew that my file(s) were not readable for-sure by haproxy.  Without a proper setup to run those full examples or any other examples that I could find that should had worked, the haproxy configuration.txt had a tiny test file, so I tried that today and it worked.  The best thing that came out of all of this _also_ is that I have a full FreeBSD-11.0 stable with all of FreeBSD-10.2p7 /usr/lib and /usr/lib32 included or else Curl would never install, and I got less errors when compiling haproxy … It’s all listed above.

The new CURL proved that those old lib were missing from 11.0 stable and CURL and whatever else definitely needed them.  I wrote them down and founded them all in my old database1 FreeBSD-10.2-p7 jail.  If it was not for that, I’ll would have gave up or something.  So for those of you who never delete-old-lib since 9x or 10x, that is why stuff still works for you.  Not all new ports have caught-up.  I see that at the very least FreeBSD-10.2-p7 lib proved safe to use on 11.0 or else I be reverse-proxying with nginx right now that I think about it.  And I’m glad 10.2-p7 was where I stopped until todays 11.0.  Don't replace lib, only paste those that are missing.  Knocking on wood.  Now we know HAProxy.

Thanks guys!


----------



## Snurg (Dec 18, 2017)

Welcome back, max21
Yes, I know that... when there is a new project it's at first a big pile of work to be done. To keep in the flow, I also often reduce housekeeping chores to a minimum for a while.
Didn't do dishwashing for two weeks and today it's dishwashing day because I ran out of clean plates, knives and forks 

Regarding your mixed system (10.2+11), I'd suggest to just tear it down and start from scratch.
That version mismatches can only lead to more frustrating difficulties, easily avoidable by clean reinstall. This is one of the reasons why I want my jail manager to be able not only to manage jail.conf, but also to actually do generic setup of the most-important things.
I want to be able to do a clean base system install from scratch/CD, then enter `jailboy setup`, and there having the possibility to install/preconfigure a variety of useful jailed applications (in addition to reverse proxy and unbound dnssec cache):
-letsencrypt: container for certificate generator
-webservers: apache, nginx-trap server for disclosing session recording companies' customers
-jailed browsers
-jailed thunderbird
-jailed WINE
-and maybe even more (suggestions welcome)
... and It will be even possible to dynamically launch separate jailed instances of jailed firefox, for example.
This way I hope to make the thing attractive not only for advanced users, but also for every desktop user.

This is a horrible chore I do not want to do manually again. The main steps of that in the handbook are about this (part of the script generated by the setup part of my 'jailboy'):


```
mkdir -p /home/jails/templates/default.ro
cd /usr/src
make installworld DESTDIR=/home/jails/templates/default.ro
cd /home/jails/templates/default.ro
mkdir usr/ports
portsnap -p /home/jails/templates/default.ro/usr/ports fetch extract
cpdup /usr/src /home/jails/templates/default.ro/usr/src
mkdir /home/jails/templates/default.rw /home/jails/templates/default.rw/home /home/jails/templates/default.rw/usr-X11R6 /home/jails/templates/default.rw/distfiles
mv etc /home/jails/templates/default.rw
mv usr/local /home/jails/templates/default.rw/usr-local
mv tmp /home/jails/templates/default.rw
mv var /home/jails/templates/default.rw
mv root /home/jails/templates/default.rw
mergemaster -t /home/jails/templates/default.rw/var/tmp/temproot -D /home/jails/templates/default.rw -i
cd /home/jails/templates/default.rw
rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev
cd /home/jails/templates/default.ro
mkdir rw
ln -s rw/etc etc
ln -s rw/home home
ln -s rw/root root
ln -s ../rw/usr-local usr/local
ln -s ../rw/usr-X11R6 usr/X11R6
ln -s ../../rw/distfiles usr/ports/distfiles
ln -s rw/tmp tmp
ln -s rw/var var
```

And this is only the part to create the basic R/O and R/W skeletons...
You could run this from sh to get that installed in a snap.
(You might wonder why the above does not contain error checking. It gets executed command by command by my script, so that errors are caught.)


----------



## max21 (Dec 20, 2017)

Snurg said:


> . . .
> . . .
> Regarding your mixed system (10.2+11), I'd suggest to just tear it down and start from scratch.
> That version mismatches can only lead to more frustrating difficulties, easily avoidable by clean reinstall.



I remember when GNOME-3 said take or leave it.  GNOME-3 gave up the ability to be a true desktop because they lost control of their code.  Everybody know a Gnome2 type desktop runs like a bat-out-of-he** which will do me just fine until quantum comes about.  It’s no big deal, I just found a way to keep on upgrading without losing my favorite desktop.  I lost the best twice since upgrading from FreeBSD-8.2.  There is little hope once it’s gone.  Support ends 3-months and a day at next release.  However, do understand, that was all about how I maintained the last kick-a** desktop of the century, and not about a machine in production or jails.  I’m about to remove/edit some of the stuff I wrote here.  If you kind-of misunderstood the propose, I can only imagine what might happen to a newbie who try some of the things I wrote.  I don’t want to feel responsible for that.  One mistake, it over, back to Linux or Windows.

Anyway, after reviewing Jailboy, I see now that running many manual jails can be a major problem.  If I have 300 manual-jails, I would have 300 directories full of devices.  I never really thought about that before.  So that is where the stuff hits the fan.  One is better than many, when many can use, just one.  Your project is way above my head _especially the application parts, traps, etc_ and it seems like a timely mission.  You might have to start an early website forum, message board, dedicated thread or something for it where you can document your work and pickup on ideas.  Are you actually just getting started or are you near completion?  It’s kind of like the Nginx story, build with FreeBSD, but in this case it’s specifically  for FreeBSD.  It sounds like something that would put the MAC in a state of shock.

I don’t know if this will help you or not, but as you know may know, I always talk about stripping stuff down to the bone.  Well, last night I nearly created the smallest jail in the world; _I just like to say stuff_ 

The first one is design for pkg’s only.  Thru rapid trial and error I removed many file from /etc and /usr, including nearly everything else in there other than /usr/bin and /usr/sbin so that they only contain what the system need to classify it as a fully functionable jail (ssh, log),etc;.  I was surprise that both, HAProxy  and sshguard only required three files inside of /usr/lib to run.  The thing was they both use the same files or else neither would run:

1) libgnuregex.a
2) libgnuregex.so
3) libgnuregex.so.5

So far I got this kind of jail (_running tiny pkg’s_) down to *98.6*MB untared.  Tar size: *28.3*MB.  That almost the size of a Windows 10 module.  For what I see now I don’t think it is possible to get any jail smaller than *85*MB.  It tars up in exactly *5.5 seconds*.  This is the kind of stuff I come back to from time to time.  My theory is be it right, or wrong, I don’t have another life time to studying networking surrounded by in-house hackers.  So I think I’ll just do some web hosting to earn some jail-bird seeds or to have some fun.  If some genius breaks into the public facing jail, he would have no way to check-out and no tools to work with when needed; because for that jails everything goes to the dumpster such as /bin/chflags, /bin/dd, /usr/sbin/mountd, including everything else that the host and jail applications don’t need.  Then I’ll raise the security level so high that even the host can’t jump the fence.  The only thing he is allowed to do is to detect, destroy and replace.   Above all, I just like learning how far can I go.  Unlike Alcatraz, my jails accept no prisoners.

Whatever the case, FreeBSD got my back!


----------



## Snurg (Dec 20, 2017)

The core jailboy stuff is about 3/4 done (installing and configuring the jail system, either interactive or command-line based).
Atm I am working on the configuration of the reverse proxy and PF things.

While doing the whole stuff I have incorporated several ideas I forgot in the basic concept at the beginning.
For example, grouping jails (think of the unix group model) makes many things easier. Modifying, reconfiguring, starting, stopping, multiple jails at once.

The jailboy thing creates a log of created/changed config files, so I am still doing "simulation" tests only, trying out the newly implemented stuff and refining until I am satisfied with the generated output (config files and batch of shell commands).
When the groups thing is implemented, I can finally do some first actual tests.
I need to create, modify and delete a lot of test jails to make sure I find the most obvious bugs before I can think of releasing a first alpha preview.
The script has >2400 lines atm. An ugly big chunk. But this will be modularly split up later into much smaller core script and (optional) "addons" (think docker etc, but easier to use).

Regarding the "size of the jail", this depends on perspective. An example:
I want Firefox in dynamically creatable jails. However, installed over the base system Firefox adds about a gigabyte of files. Thus to have actually fun with that, it is necessary to move the read-only part of that stuff to a new (read-only) base jail, reducing the "virtual system" that has to be created for each instance to near zero. This way new instances of can be fired up without actually creating a virtual system by copying the _whole OS _stuff.
Since summer even Windows offers a sandboxed version of its Edge browser. (albeit with serious disadvantages, reserves 8gb of physical ram and needs to copy around 1 gb of system to the sandbox. So this takes at least 10 sec to start, and above all it's only available on Enterprise.)
I think such should be easily possible for every desktop FreeBSD user with an 1-click-install, and without the abovementioned disadvantages of Edge.


----------

