# portmaster best practices?



## eldaemon (Oct 9, 2020)

I compile all of my own ports into packages and keep the distfiles so I can rebuild it all offline if need be.

I recently switched over to the 2020Q4 ports tree. Normally, my process is `svn update` (in /usr/ports), `portmaster -a -B -D -G -y --no-confirm -g`. This seems to work pretty well when doing updates before the quarterly jump, but after a quarterly jump it gets more dicey.

For instance, it failed this time. I think libffi was updated and then glib2 failed because Python 3 was built against the older libffi. I rebuilt Python 3 which fixed the issue and I'm able to run `portmaster` again to keep upgrading. Also, I don't think rebuilding a package bumps the package version number, so `pkg upgrade` across my systems wouldn't upgrade it which is another issue.

Is there an extra switch I should be using? Should I be using a different build tool to make this more bulletproof?

Thank you!


----------



## Francisco Reyes (Oct 9, 2020)

Have you considered Poudriere or synth? That may be a potential solution.


----------



## a6h (Oct 9, 2020)

ports-mgmt/portmaster Best practice:

Step #1:
`pkg delete ports-mgmt/portmaster`

Step #2:
Ports:
`cd /usr/ports/ports-mgmt/poudriere && make install clean`
`cd /usr/ports/devel/ccache && make install clean`
OR
Binary:
`pkg install ports-mgmt/poudriere`
`pkg install devel/ccache`

Step #3:

https://www.freebsd.org/doc/handbook/ports-poudriere.html
https://wiki.freebsd.org/VladimirKrstulja/Guides/Poudriere
ports-mgmt/poudriere
devel/ccache
Why?








						Do not use ports-mgmt/portmaster and other tools who build in the main system
					

Please consult Howto build ports with ports-mgmt/poudriere for a set of historically evolved issues concerning the ports tree. Please consult the well-known Recursive Make Considered Harmful for reasons why such tools do not handle the build process theoretically sound -- they do not use make...




					forums.freebsd.org


----------



## olli@ (Oct 9, 2020)

I do not like the black&white attitude “poudriere is the only correct way to build ports, and everything else is wrong”. It’s not that simple.
For example, I’m used to build from ports with some helper scripts, and this works very well for me and it’s comparatively fast. There are no real disadvantages doing it that way.

I don’t use portmaster (anymore), though, so I can’t answer the OP’s particular question.


----------



## Minbari (Oct 9, 2020)

I'm using -ad  -w (to save old shared libraries before deinstall) and sometimes when I don't want to update big ports -i (llvm's, ruby, gcc, etc.). Also, I consider poudriere/synth are useless when I'm running FreeBSD on a single machine which is using quarterly ports.


----------



## chrcol (Oct 9, 2020)

First comment, I dont like the spamming on here from some members trying to push Poudriere, its ok maybe in its own thread, but when someone asks how to do something with portmaster, the answer is not use Poudriere  instead.

Now I dont usually use portmaster, not a fan of its decision making, but I have checked the man page to try and see what went wrong with the flags chosen.  Your flags wouldnt manipulate the build order so I dont know sadly.


----------



## richardtoohey2 (Oct 9, 2020)

Sure, but who gets to choose which one?  portmaster has been fine _for me_ for many years but it doesn't work for everyone. If I was less lazy I'd look at what I actually use it for and figure out how to just extract that part (I think that would be like what olli has described above - a helper script or two, which is probably what portmaster started out as). But it still wouldn't be poudriere.

I'm not helping the OP either - I follow the latest ports rather than quarterly - so   from me now.


----------



## Oclair (Oct 9, 2020)

Francisco Reyes said:


> Have you considered Poudriere or synth? That may be a potential solution.



to a different question.... #offtopic


----------



## Oclair (Oct 9, 2020)

I primarily use portmaster, and use pkg to do some trimming.  I don't get into trouble any more generally.  There are many ins and outs, 

for example a basic updating run 
first check UPDATING, 
then run #rm -R /usr/ports/distfiles/*  
then #portsnap fetch 
and finally #portmaster -ay 

Many times things break or as I like to think go bump in the night and stop portmaster, at such times generally going into the dependent port it's self 
#cd /usr/port/libsomething/brokeportmaster 
# make deinstall reinstall clean

and then things seem to work out

Other times UPDATING (should) mention what to do, which might mean modifying /etc/make.conf to actually set what your own personal preferences are
(here's one of mine) mileage varies
 ##/etc/make.conf override

DEFAULT_VERSIONS+= php=7.3
DEFAULT_VERSIONS+= mysql=10.2m
DEFAULT_VERSIONS+= python3=3.7
DEFAULT_VERSIONS+= ssl=openssl
#DEFAULT_VERSIONS+= ssl=openssl111

MAKE_JOBS_UNSAFE=yes
FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=8

#LICENSES_ACCEPTED+=NONE

OPTIONS_UNSET+= X11 GUI CUPS NLS HAL GSSAPI_BASE KRB_BASE KERBEROS
OPTIONS_SET+= GSSAPI_NONE KRB_NONE
WITHOUT_X=YES
WITH_X=NO
ENABLE_GUI=NO


----------



## drhowarddrfine (Oct 10, 2020)

anonymous9 said:


> Invariably, mixing ports and packages screws over too many, too, which also needs to be addressed in some way.


Packages are created from ports so it is addressed. The reasons for not mixing them is well known.



anonymous9 said:


> ports-mgmt/synth
> ports-mgmt/poudriere
> ports-mgmt/portupgrade
> ports-mgmt/portmaster


I always thought synth was the ed of managers and poudriere was for those installing on multiple servers and workstations. I thought portupgrade was going away and recently read that portmaster was too. Not that I've paid that much attention. I primarily use make and pkg for large programs that take forever to build.


----------



## olli@ (Oct 10, 2020)

anonymous9 said:


> Ports management in FreeBSD needs to coalesce around one method.  Having 20 ways to do the same thing does not always make something better because there are so many choices, especially when dealing with ports. Nobody would ever say, "You know, we have a dozen different methods to upgrade FreeBSD to a new version or release, and that's what makes FreeBSD so great. So many choices."


Having choices is always a good thing, IMHO, and this is one of the main advantages of UNIX-derived systems. You can take the way that is most suitable for _you_, and that doesn’t necessarily has to be the way suitable for most others.

Maybe there are not a dozen methods to upgrade FreeBSD, but I can think of four or five from the top of my head, maybe more. And this is good. There is no “one size fits all”.


----------



## Oclair (Oct 10, 2020)

I really do not understand what the obsession some folks have to remove a tool which manages the ports system.  Perhaps it's good they blame the messenger than the Ports system it's self.  Don't mess with Portmaster, I don't care how you manage your ports pal.


----------



## a6h (Oct 11, 2020)

olli@  You're right. My reply could be count as a hijack-post. I apologise for that.

Onwards.

There's only one minor issue. _Somebody else_ has suggested, because I was mentioning poudriere, then I'm a SPAMMER! I don't think promoting a program, which has a section in the FreeBSD handbook, and has been repeatedly suggested by FreeBSD committers, developers and system administrators counts as SPAM. Because I'm not knowledgeable enough to compare the technical details of different solutions, I have to rely on expert in the field, in this case developers and administrators. That's why I was suggesting to use poudriere instead of portmaster. Also I'm not the developer of poudriere, so I don't have a dog in the fight. All that said, I admit that my reply could be a hijack-form of posting, thus I'm not going to argue further about this topic.


----------



## olli@ (Oct 11, 2020)

vigole said:


> There's only one minor issue. _Somebody else_ has suggested, because I was mentioning poudriere, then I'm a SPAMMER!


Calling you (or anybody else here) a “spammer” is completely inappropriate.

The discussions here should focus on technical contributions, not personal attacks. I guess I’m not innocent either, calling you a hijacker was not the right thing to do, and I have to apologize for that.  When someone asks a question regarding portmaster, it’s not completely unreasonable to mention poudriere as a potential alternative, even though it’s not really an answer to his question.

However, statements along the lines of “poudriere ist the _only_ way to go, and everything else is _just wrong_” are not helpful. It provokes opposition and disagreement, and it causes conflict and division. We can observe that very well in these threads.


----------



## olli@ (Oct 12, 2020)

anonymous9 said:


> Someone new to FreeBSD comes along and asks, "How do I install packages?"  And some say binary packages, while others say only use ports and use poudriere, while others say portmanager, then some say synth. Then the OP starts reading on poudriere and says "I only want to install a package."  Gives up and leaves.


What’s wrong with having one _recommended_ way that works for most users (in particular for people new to FreeBSD), but offering more choices for users who have special requirements?

In the case of installing packages, the recommended way is to use pkg, of course (see the FreeBSD Handbook). If you want to compile from the ports collection yourself, the recommended way seems to be poudriere (although I’m not sure if this is an “official” recommendation). But if portmaster works better than poudriere for a certain user, well, why would you forbid him to use it? Would you want to ban all tools except for poudriere? If you want a monoculture of tools, then you shouldn’t be using an open source operating system with public APIs.

By the way, there are even multiple different ways to install binary packages beside pkg. For example, you can use bsdconfig(8) which provides a full-screen UI for selecting packages. There are also several GUI tools for X11. Having choices is good.


----------



## drhowarddrfine (Oct 12, 2020)

olli@ said:


> Having choices is good.


Having too many choices is bad. There is even a UX term for it that I can't recall, having just woken up.

There should be one general purpose manager one can point a new user to so they can get started. After they get comfortable, let them discover other installers on their own. Perhaps this is a documentation issue more than anything else. When I would train people, I would say, "Here's how you do this. There are other ways you might like better but this is good for now. You'll find there are other ways later that you'll like better but, for now, do it this way."

I do think the Handbook is good about that--limiting choice to two methods of doing things.


----------



## Mjölnir (Oct 12, 2020)

olli@ said:


> What’s wrong with having one _recommended_ way that works for most users (in particular for people new to FreeBSD), but offering more choices for users who have special requirements?


Nothing, except that the defaults present a pitfall for newbies: the version mismatch of ports(7) tree (_latest_ branch) vs. packages (built from the _quarterly_ branch).  The (bad) consequence is that dependencies will be pulled in from another branch when you start to build a single software package from ports on a system installed from binary packages.


olli@ said:


> In the case of installing packages, the recommended way is to use pkg, of course (see the FreeBSD Handbook). If you want to compile from the ports collection yourself, the recommended way seems to be poudriere (although I’m not sure if this is an “official” recommendation).


ports-mgmt/poudriere is semi-official in that the packages for the official repositories are built via Poudrière.  The strength of Poudrière is to handle _profiles_, whereas ports-mgmt/synth might be much easier for the case when users have a small number of FreeBSD hosts.


olli@ said:


> But if portmaster works better than poudriere for a certain user, well, why would you forbid him to use it? Would you want to ban all tools except for poudriere? If you want a monoculture of tools, then you shouldn’t be using an open source operating system with public APIs.


It's not about forbidding, but to help newbies to prevent to shoot themselves in their foot.  Other than that, portmaster manages to let experienced users to shoot into their feet, too... (see post #1 and other threads). Denying the obvious bugs of portmaster à la _"portmaster has worked fine for me for decades"_ is not helpful either.  Note that it is a shell script of >1 kLoC, beeing in a refactoring process & rewritten in LUA since more than a year now; that's a heavy task because some of it's behaviour is broken-by-design.


olli@ said:


> By the way, there are even multiple different ways to install binary packages beside pkg. For example, you can use bsdconfig(8) which provides a full-screen UI for selecting packages. There are also several GUI tools for X11. Having choices is good.


bsdconfig(8) and other (G)UI tools use pkg(8) under the hood.  Please note that some ports are self-contained in the ports(7) tree (scripts) and do not exist as binary package; i.e. these can only be installed from ports(7).  On the genuine topic, my recommendations are

Use the same version of ports(7) tree & package repository, i.e. either use the _latest_ package repository in /etc/pkg/FreeBSD.conf, or manage your _quarterly_ ports(7) tree with svnlite(1), net/svnup (don't mix these two!) or git(1) instead of portsnap(8).  Portsnap only manages the _latest_ ports tree & will vanish soon, anyway.
While this is not strictly neccessary, this will enable you to safely use portmaster(8)'s `-P` and `-PP` switches.  Frontends & GUI tools for Subversion/svn & git(1) exist, e.g. `psearch -c devel -s git -s GUI`
Work around portmaster(8) bugs where possible, e.g. use `make -C /usr/ports/category/someport fetch-recursive` instead of the corresponding portmaster(8) command.  In some cases, you can derive the neccessary tasks from RTFM, in some it means RTSL.
Use ports-mgmt/portconf in combination with portmaster.
Using devel/ccache and/or devel/distcc will speed up long builds regardless of the method you choose.


----------



## olli@ (Oct 12, 2020)

mjollnir, I basically agree with most of what you wrote (with minor disagreement in some details, but that’s really not that important).
Just some notes and additions …



mjollnir said:


> Nothing, except that the defaults present a pitfall for newbies: the version mismatch of ports(7) tree (_latest_ branch) vs. packages (built from the _quarterly_ branch).  The (bad) consequence is that dependencies will be pulled in from another branch when you start to build a single software package from ports on a system installed from binary packages.


Yes, I agree, those defaults can cause trouble. This is why newbies are told to _not_ mix ports and packages (and this is pretty much independent from the tools you use, whether portmaster or something else).

Those defaults have historic reasons. The concept of quarterly branches is relatively new – Not too long ago there was only “current” ports (a.k.a. “head”). Shortly after the quarterly branches were introduced, the decision was made to use them by default for the package repository of new releases, because they were assumed to be more stable than “head” and in a more consistent state, and therefore newbies would have less problems with them. On the other hand, portsnap continued to only deliver “head” – partially because of technical reasons (without major changes, portsnap itself and the infrastructure behind it were not able to handle multiple branches), partially because people who decided to use portsnap and build from the ports collection themselves were assumed to be advanced users who prefer to have the most current software.

There are other tools beside portsnap, of course. For example, net/svnup is quite popular for fetching source and ports. Unlike portsnap, it supports branches, and there is no default. You have to specify the branch that you want. So you can use svnup to fetch a quarterly branch and build from it without problems. Of course, you can do the same with svn, svnlite or git if you want. The advantage of svnup is that it just fetches the files, without all of the overhead of local repository data, so the end result is exactly the same as if you used portsnap, except that you _can_ use branches.

For that reason I’m not afraid of portsnap going away, because svnup is a good replacement. And even if svnup didn’t exist (or stopped working for some reason), svnlite can be used as well. Since there’s a git-to-svn gateway, it’ll continue to work even when the FreeBSD repositories switch over to git. (However, I assume that there will be specialized tools for git at some point in the future, maybe an svnup-workalike called “gitup” or something like that.)



> It's not about forbidding, but to help newbies to prevent to shoot themselves in their foot.


Well, as the saying goes:
“UNIX does not prevent you from doing stupid things, because that would also prevent you from doing clever things.”

But seriously … Well, yes, portmaster has some bugs and shortcomings. I think nobody is denying that. It is probably fair to say that it should not be the first choice for newbies. However, the OP of this thread doesn’t appear to be a newbie.



> Note that it is a shell script of >1 kLoC


Uh … Are you sure? I assumed it was much larger. I have to admit I’ve never looked at it myself, only read about it at various occasions.

My own script is currently 2.2 kLoC, plus a few small helper scripts that add up to another 0.5 kLoC, roughly. (One third of that is comments, though.) Of course, it started as a much smaller script. If I had known at the beginning how it would evolve, I would have written it in Python instead, probably. But I’m not going to rewrite it now. It’s still manageable as a shell script since I made some efforts to have it structured in a useful way.



> bsdconfig(8) and other (G)UI tools use pkg(8) under the hood.


Yes, of course. Likewise, all of the various ports building tools use the ports collection’s Makefile framework under the hood.



> Please note that some ports are self-contained in the ports(7) tree (scripts) and do not exist as binary package; i.e. these can only be installed from ports(7).


There are several reasons why no binary packages are provided for certain ports. For example, some software has license restrictions that prohibit binary redistribution, or you have to confirm a license agreement when you build the port. Also, some ports cripple their options when building a package for redistribution – for example, the binary package of audio/sox has _no_ LAME support (for mp3 encoding), even though the LAME option is enabled by default when you build the port yourself. Furthermore, some ports must always be compiled from source because the package is specific to the local machine for some reason (e.g. it puts hardware-specific things in the binary, or a different key is generated each time you build the port, or similar). Currently there are about 250 ports affected by things like that.



> On the genuine topic, my recommendations are […]


Agreed on all points.


----------

