# Speed up the building of ports using prebuilt packages?



## Deleted member 66267 (Feb 3, 2021)

Most of the time spent is not of building the actual port itself, but the dependencies of it.

Since I only modify the build options of just this port, could I just grab prebuilt packages for the dependencies to speed up the build?


----------



## chrbr (Feb 3, 2021)

portmaster(8) has now an option to use packages for build dependencies. There are a few other package related options, too. Please check the man page and the options starting with --packages.


----------



## SirDice (Feb 3, 2021)

`pkg install -A `make -C /usr/ports/some/port missing``

See pkg-install(8) for the `-A` option and ports(7) for `make missing`.


----------



## Deleted member 66267 (Feb 3, 2021)

chrbr said:


> portmaster(8) has now an option to use packages for build dependencies. There are a few other package related options, too. Please check the man page and the options starting with --packages.


Thank you. But I like SirDice's answer more because I hear the tool you mentioned was broken and deprecated for a long time in favor of poudriere(8).


----------



## chrbr (Feb 3, 2021)

Dear also-ran,
if you have a machine which can run poudriere(8) this is the best option. Some ports take a long time to build. Therefore the demands on the build machine is steadily increasing. I own old stuff ;-).
But even when you have a weak or medium powerful machine it is worth to learn and run poudriere(8) because it works well (beside the build time issues) and it is the offical and most secure way to build packages.


----------



## SirDice (Feb 3, 2021)

For package building you might also want to take a look at ports-mgmt/synth. It uses the same principles (building packages in a "clean" environment) but works a little different. Its the package build system that's used by DragonFlyBSD. Poudriere is the de facto standard on FreeBSD because it's used on the FreeBSD build clusters. Both tools will allow you to create your own package repositories. The benefit is that you can use the flexibility of the ports, by enabling or disabling options and setting your own defaults (default MySQL version, PHP, Perl, Python, Ruby, etc), while keeping the ease of management from packages (install, upgrade, etc).


----------



## T-Daemon (Feb 3, 2021)

SirDice said:


> `pkg install -A `make -C /usr/ports/some/port missing``


I tried that in the past, it works good until it hits a py-* dependency for example.

There is an in ports(7) undocumented but in /usr/ports/Mk/bsd.port.mk listed target for make(1)

```
install-missing-packages
              - Install missing dependencies from package and mark
                 them as automatically installed
```
No need to run pkg(8).


----------



## CyberCr33p (Jan 4, 2022)

Is it possible to pre-build MySQL 8.0 dependencies (MySQL 5.7 already installed) and mark them as automatically installed and then remove MySQL 5.7 and build MySQL 8.0 to avoid the most downtime?


----------



## CyberCr33p (Jan 4, 2022)

Finally I create a package for mysql80-server and mysql80-client to install it to all servers.


----------



## astyle (Jan 4, 2022)

CyberCr33p said:


> Is it possible to pre-build MySQL 8.0 dependencies (MySQL 5.7 already installed) and mark them as automatically installed and then remove MySQL 5.7 and build MySQL 8.0 to avoid the most downtime?


It depends on what you mean by 'avoiding downtime'. The process would be much simpler if you just build MySQL 8.0 from ports, and let the dependency resolution take care of the rest. On your end of things, does the build process make the rest of the computer go unacceptably slow? It takes about 5 minutes on a Ryzen 5 1400 3.4 GHz to compile MySQL and dependencies. If you want to build MySQL 8.0 (but not install it), all you need to do in /usr/ports/databases/mysql80 is `# make`, with no other arguments.


----------



## SirDice (Jan 4, 2022)

If you go with the package route by building your own packages then you don't typically build those packages on the production server anyway. So the building itself won't cause any downtime. The database will only need to be taken offline when you actually install the resulting package on the machine.


----------



## astyle (Jan 4, 2022)

Even then, the package database is file-backed. Off the top of my head, I think it's BDB, not something used in the same manner as MySQL. The system only needs a few seconds to actually install the packages. Even big ports take less than a minute to install.


----------



## SirDice (Jan 4, 2022)

astyle said:


> Even then, the package database is file-backed.


Has nothing to do with it. When you build databases/mysql80-server, the dependency databases/mysql80-client has to be built and installed first. If you do this on the server that has the actual MySQL 5.7 running installing databases/mysql80-client would remove databases/mysql56-client (conflicting port) and as a consequence it will remove databases/mysql57-server too. You're then in "limbo" mode where the service is still running but the actual executables have been removed from the system. Not a situation you want to be in. So you shutdown the database _before_ starting the build and would then be offline until you can get MySQL 8.0 server to build and install. 



astyle said:


> Off the top of my head, I think it's BDB


pkg(8) uses an SQLite database.



astyle said:


> The system only needs a few seconds to actually install the packages.


Yes, exactly. Create a package from the port on a different system (using poudriere, synth or by hand), transfer the packages and install those on a test system first (to verify the package actually works). You can also test the upgrade from 5.7 to 8.0 here, that way you know in advance what kind of issues you might run into. Once you're satisfied it works you install those packages on the production server. That only takes a couple of minutes max.


----------



## covacat (Jan 4, 2022)

in theory you can pkg delete 5.x client and server install 8.x client and server and restart
server continues to run even if the binary is gone
if some error message files are not mmaped and the server needs them in the interim period that may be a problem


----------



## kpedersen (Jan 4, 2022)

SirDice said:


> You're then in "limbo" mode where the service is still running but the actual executables have been removed from the system. Not a situation you want to be in.


This idea always feels really nasty but pretty much happens every day to a large proportion of desktop users.

Lets say someone is running a Gnome or KDE desktop. They run a pkg upgrade and it pretty much just strips out the old executables and resources and just shoves in new potentially different ones. The old executables are still running and loaded in memory and accessing resources that aren't intended for them. There could even be weird logic that replaces files with original versions if it doesn't "understand" them so half of the new files could potentially be replaced with half old ones again.

Package managers are superior to the ad-hoc Windows way but still seem to benefit with a good ol' restart after


----------



## SirDice (Jan 4, 2022)

kpedersen said:


> This idea always feels really nasty but pretty much happens every day to a large proportion of desktop users.


I do this quite frequently actually but not with 'important' services like databases. With those I just feel safer shutting them down first, then do the update. Any other service is usually easily fixed if you do run into issues but with databases I'm always a bit more careful.


----------



## covacat (Jan 4, 2022)

also having the server port installing the client is a  bug for mysql at least

```
[19:06:41] [host!root]~#pkg delete -f mysql57-client
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
    mysql57-client: 5.7.36

Number of packages to be removed: 1

The operation will free 49 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling mysql57-client-5.7.36...
[1/1] Deleting files for mysql57-client-5.7.36: 100%
[19:06:44] [host!root]~#/usr/local/etc/rc.d/mysql-server stop
Stopping mysql.
Waiting for PIDS: 50129, 50129.
[19:06:58] [host!root]~#mysqladmin -u root -p status^C
[19:07:06] [host!root]~#/usr/local/etc/rc.d/mysql-server start
Starting mysql.
[19:07:11] [host!root]~#pkg install -f mysql57-client
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    mysql57-client: 5.7.36

Number of packages to be installed: 1

The process will require 49 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing mysql57-client-5.7.36...
[1/1] Extracting mysql57-client-5.7.36: 100%
=====
Message from mysql57-client-5.7.36:

--
This is the mysql CLIENT without the server.
for complete server and client, please install databases/mysql57-server
[19:07:17] [host!root]~#mysqladmin -u root -p status
Enter password:
Uptime: 15  Threads: 1  Questions: 2  Slow queries: 0  Opens: 108  Flush tables: 1  Open tables: 101  Queries per second avg: 0.133
[19:07:25] [host!root]~#
```
server can definitely run without the client


----------



## SirDice (Jan 4, 2022)

The client has been a run dependency for the server for as long as I can remember. It's possible it's not actually a run requirement but the port/package certainly has a build and run dependency on it.


```
root@molly:/usr/ports/databases/mysql80-server # make run-depends-list
/usr/ports/ftp/curl
/usr/ports/devel/libevent
/usr/ports/devel/icu
/usr/ports/archivers/liblz4
/usr/ports/archivers/zstd
/usr/ports/devel/protobuf
/usr/ports/devel/libunwind
/usr/ports/devel/libedit
/usr/ports/databases/mysql80-client
/usr/ports/textproc/groff
/usr/ports/lang/perl5.32
root@molly:/usr/ports/databases/mysql80-server # make build-depends-list
/usr/ports/ports-mgmt/pkg
/usr/ports/archivers/liblz4
/usr/ports/devel/libunwind
/usr/ports/devel/bison
/usr/ports/devel/cmake
/usr/ports/devel/pkgconf
/usr/ports/ftp/curl
/usr/ports/devel/libevent
/usr/ports/devel/icu
/usr/ports/archivers/zstd
/usr/ports/devel/protobuf
/usr/ports/devel/libedit
/usr/ports/databases/mysql80-client
```


----------



## covacat (Jan 4, 2022)

USES=mysql:80 implies client
also port forbids version mismatch between client and server which is a lesser problem


----------



## astyle (Jan 4, 2022)

SirDice said:


> The client has been a run dependency for the server for as long as I can remember. It's possible it's not actually a run requirement but the port/package certainly has a build and run dependency on it.
> 
> 
> ```
> ...


A server needs a client installed at least locally, but not the other way around. Gotta be careful about that when specifying the deps - direction of the dependency really matters.


astyle said:


> If you want to build MySQL 8.0 (but not install it)


See the rest of my post. This is the nice thing about forums - you can easily go back and see what was said on the topic earlier - it just requires a bit of attention.


----------



## grahamperrin@ (Jan 4, 2022)

CyberCr33p are you aware of option `-b` for prefetch to speed things up with poudriere?

<https://github.com/freebsd/poudriere/wiki/poudriere-bulk.8-devel#options>

– it's a not previously given answer to the original question, 



> Speed up the building of ports using prebuilt packages?


----------



## CyberCr33p (Jan 4, 2022)

I don't use poudriere.

For minor MySQL upgrades I use portmaster while MySQL running and then I restart to reload the new binary. But for major upgrades we do a full mysqldump, a rsync of /var/db/mysql while MySQL is running, then shudown MySQL, make a new rsync which doesn't take much time, then a "pkg del" for old version, then I do the upgrade to the new major version (if I use portmaster then it needs ~20 minutes to complete because it needs to install the "build" dependencies such as python), I replace the my.cnf with the one that is compatible with new version, and then I start MySQL. MySQL 8.0 doesn't need mysql_upgrade any more as it takes care of it during the MySQL start.

I create mysql80-server-8.0.27.pkg and mysql80-client-8.0.27.pkg as the server I originally build it and all other servers have the same software & versions installed (and all MySQL running dependecies already installed) so now instead of portupgrade I use "pkg add mysql80-server-8.0.27.pkg" (mysql80-client-8.0.27.pkg automatically installed as both files are in the same directory).

So instead of 20-25 minutes downtime I now have 1-2 minutes downtime per server.


----------



## astyle (Jan 4, 2022)

Something wrong with `# pkg upgrade`?


----------



## CyberCr33p (Jan 4, 2022)

I don't use default settings in these ports so pkg upgrade wouldn't work in my case. Also I add this patch too during the build https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259816#c14


----------



## SirDice (Jan 4, 2022)

CyberCr33p said:


> I don't use poudriere.





CyberCr33p said:


> and all other servers have the same software & versions installed


Sounds like you really need to set up your own repository. Honestly, look into it. It's going to make updates and upgrades like this so much easier. Because you can have everything you need already built, ready to be installed from a central, local, repository. And now you've changed to MySQL 8.0 you really want to set `DEFAULT_VERSIONS+= mysql=8.0` and have everything else depend on the right version of the client too.


----------



## CyberCr33p (Jan 4, 2022)

Are you sure I need DEFAULT_VERSIONS+= mysql=8.0 ? I don't use any other software that has MySQL as dependency.


----------



## astyle (Jan 4, 2022)

Pre-build your package with make, then do `#make package`, then find your package, and then do `# pkg upgrade ./my_custom_built_package`.


----------

