# Opinion: FreeBSD ports is broken by design



## Mjölnir (Jul 18, 2013)

_[ Split off from http://forums.freebsd.org/showthread.php?t=40818 -- Mod. ]_

I had the same issue when I tried to install x11/kde4-runtime.  Installing misc/help2man directly succeeded.  I.e. the success of the very same task is pure luck.

Unfortunately, I take your answer:


			
				Chris_H said:
			
		

> Greetings,
> Just choose i here. This is a harmless error. It means you have not created the directory for backup packages, or the previous version(s) were not where this version expects them. Won't interfere with the new version. But won't permit the ability to revert to previous version, using the portmaster backup package. That's all.
> --chris


as a confirmation of my opinion: "FreeBSD ports is broken by design".


if portmaster needs a folder for backups, it should create it
If the package is installed the first time, there is no old version

If there's no recursive make, as someone said, then why do we need tools like (so-called) portmaster to correct what make should be able to do on its own?  Why do people constantly have trouble compiling ports?  Did you ever try to update portmaster with itself?  There are numerous bugs like this.


----------



## DutchDaemon (Jul 18, 2013)

Neither ports, nor portmaster, ever give me problems, occasional (and usually quickly fixed) hiccups with a port's Makefile aside. The broken by design comment is ludicrous.


----------



## Mjölnir (Jul 18, 2013)

Well, if you call the random success of a task a hiccup, everything is fine 

But I'm getting suspicious in such a case.  Can you clearly confirm that the ports Makefiles are not recursive?

I agree that it's always possible to run into human-made bugs in the port's configuration, and I do appreciate the effort of all the volunteers who help to make FreeBSD a valuable system.


----------



## DutchDaemon (Jul 18, 2013)

A Makefile is recursive in so far as it will check whether dependencies are installed and up-to-date, and if not, it will switch to the task of performing and updating them. In that sense, portmaster is not needed to install a single port. Once it comes to upgrading several ports in conjunction, portmaster takes the hassle out of figuring out the order, to prevent double work or clobbering/race conditions. With all of its advanced options and flags for things ranging from automatic backups of previous versions, keeping old libraries around, force-reinstalling up- or downstream dependencies, it certainly takes all of the headaches out of figuring out the proper make targets in the proper order. But: you should upgrade ports at least once a week, and not let things deteriorate, and certainly not update selectively, unless there is an overriding reason not to keep up. But that's where you will start encountering problems, which will grow deeper and worse over time. The ports tree is supposed to be used in conjunction, and to be kept up-to-date and fresh. All installed ports need to be of the most recent version. Then you will have no problems, especially if you give new versions a day or two to incorporate last-minute bug fixes before actually installing them. I don't, but I'm more than capable of backing out of changes that go awry, thanks to the advanced nature of portmaster. But to be honest, with the 1000+ ports I have installed on various systems, with a daily update routine, that happens maybe two or three times a year.


----------



## Mjölnir (Jul 18, 2013)

It's just that some other not-so-advanced member said the ports Makefiles are not recursive, and honourable Mr. Peter Miller clearly points out that to call make recursively is not the right way to use it. Thus I'd say my provocative comment is not ludicrous.

In fact, as you say yourself, we need 
	
	



```
portmaster
```
 as a wrapper to fix the deliberately broken dependency graphs, and it is surprisingly good in doing that. I guess the reason to do so was limited RAM of machines in the past, and maybe easy of managing 1000's of ports, too.

But since using ports instead of pre-packaged packages is such a common thing, I'd be happy to see a solution that is theoretically proven sound.  The current solution is not.


----------



## DutchDaemon (Jul 18, 2013)

The ports system is not broken by design. It is complex, yes.


----------



## Mjölnir (Jul 18, 2013)

Amen,  and the world is flat, because I want it to be.

Sorry, but I think your opinion is based on the fact that the portmaster/make combination does work somehow right now, and it would be a big thing to change it.

Nevertheless, you are saying that it uses recursive make, that we get into trouble if we do not keep up with current versions, and that portmaster does what make should be able to do (finding the order of dependencies, i.e. to construct the tree(s)).

I'd like to understand why the current solution was chosen, and I make the assertion: if the ports collection would use non-recursive make, we'd need no portmaster; all portmaster does could be done from included Makefiles Ã  la bsd.port.mk.


----------



## DutchDaemon (Jul 18, 2013)

It is time to take your discussion to the freebsd-ports mailing list (join). Nobody here will address or fix your concerns.


----------



## SirDice (Jul 18, 2013)

As @DutchDaemon noted, you do not _need_ to use portmaster. You can do everything by hand. Seriously not recommended though. Even after using the ports system for the past 15 years I still get a headache thinking about doing it by hand. 

But I think there's some misconception about the term "recursive". I haven't dug into it too much but I don't think the ports system actually runs [cmd=]make[/cmd] by itself.


----------



## Mjölnir (Jul 18, 2013)

SirDice said:
			
		

> As @DutchDaemon noted, you do not _need_ to use portmaster. You can do everything by hand.


That's not true: you'll run into deep deep trouble, because sooner or later you'll encounter broken dependencies (because the Makefiles are recursive).On a dual-core machine, try `make -j 3` in a ports subdirectory. Weird things will happen... not for all ports, but for many. Try, say, x11/kde4. Only for small ports, everything is fine.



			
				SirDice said:
			
		

> Seriously not recommended though. Even after using the ports system for the past 15 years I still get a headache thinking about doing it by hand.


Right! This is exactly what I'm saying: you need portmaster to fix the deliberately broken dependency trees. Among other tasks (like keeping backups), that's basically what it does. Likewise does portinstall, but I'm not talking about that because it relies on libraries and thus makes life even harder. It should be vanished, first from the book, later from the ports.



			
				SirDice said:
			
		

> But I think there's some misconception about the term "recursive". I haven't dug into it too much but I don't think the ports system actually runs [cmd=]make[/cmd] by itself.


That's a matter of definition? The ports system is a collection of data to drive the toolchain to build the software. See above, when you want to install KDE4 without a wrapper you do
`# cd /usr/ports/x11/kde4`
`make package`
and then you're doing a recursive make, and currently it does not what you want. It can not, because, I'm sorry I have to repeat this: the ports system is broken by design because it uses recursive make. If you do the same task with portmaster[/fike], you have a good chance to succeed if you're able to fix some 'hickups'. Some of these may be due to human-made mistakes in the configuration files; we'll always have such, ok. But I suspect many caused by the incomplete design.
I'll assign myself to the mailing list and see what the gurus tell me about their decisions to do it the way it is.


----------



## kpa (Jul 18, 2013)

You have repeated this "recursive make is the problem" mantra for quite a while now but still you haven't actually provided any concrete details why recursive make breaks the dependency tracking. Start delivering finally or take back your claims.

If you're going to the mailing lists with this "it's broken, please fix it" attitude and you can not provide a minimal enough test case to showcase the essence of the problem you're going to get ignored, and quite rightly so.


----------



## wblock@ (Jul 18, 2013)

The "recursive make" article seemed to be about the standard way make(1) is used in projects.  A top-level Makefile, and Makefiles in subdirectories. Build the project, and it recursively builds the subdirectories. Yes, the ports system is like this. There is a Makefile in /usr/ports/, another in /usr/ports/sysutils, another in each of the port directories in there.

But the article was about large project management, and did not, in my opinion, directly relate to FreeBSD ports.  Or rather, not to a typical ports user. Those who `make -C /usr/ports` may find valid complaints in that article.

And to address a question:



> Why do people constantly have trouble compiling ports?



That's an assertion which is not supported by evidence.



> Did you ever try to update portmaster with itself?



Yes, and successfully. Probably at least half a dozen times in the last couple of years. If you're fishing for an example of brokenness, this is not it.

Look, we appreciate that you may have had a hard time with ports. As I've said before, sometimes this is due to a bug in expectations, where it works correctly but does not produce what the user expects. If you want help, start a new thread and we'll try to help you. Trolling with this "deliberately broken" line is not productive for any of us.


----------



## ShelLuser (Jul 18, 2013)

I agree with @kpa that so far we haven't really seen any solid example.

Though personally I'd be much more interested in your ideas on fixing that which you consider to be broken. It's one thing to call something broken, anyone can do that. But to come up with a well set up alternative, that's the harder part.


----------



## Mjölnir (Jul 18, 2013)

kpa said:
			
		

> You have repeated this "recursive make is the problem" mantra for quite a while now but still you haven't actually provided any concrete details why recursive make breaks the dependency tracking. Start delivering finally or take back your claims.


You can read about this in the excellent article Recursive Make Considered Harmful



			
				kpa said:
			
		

> If you're going to the mailing lists with this "it's broken, please fix it" attitude and you can not provide a minimal enough test case to showcase the essence of the problem you're going to get ignored, and quite rightly so.


I do not have this attitude.  I want to know the reasons why the ports building process is like it is, and I do not have a better solution at hand because I do not have enough background knowledge.
Testcase: start a parallel build of a big ports collection directly, like
`# cd /usr/ports/x11/kde4`
`# make -j 3`
Confirm all the default options.  If the dependencies were ok, you should succeed.


----------



## Mjölnir (Jul 18, 2013)

ShelLuser said:
			
		

> Though personally I'd be much more interested in your ideas on fixing that which you consider to be broken. Its one thing to call something broken, anyone can do that. But to come up with a well set up alternative, that's the harder part.


IMHO you don't have to have an alternative _right now_ when you suspect s/th to be wrong.
If s/o came up with a well thought out solution to such a complex task within a day, I'd be even more sceptical than with the current portmaster/make combination 

I'd like to make clear that my current status is to learn to use the available tools, e.g. I'm building my next system in a jail, and that I don't want my _questions_ to be misinterpreted as _assaults_.  That's not my intention.  If someone could tell me why it's not possible to use non-recursive make to build the ports, and thus there is no sound solution but portmaster is the best approximation, I'd be happy.


----------



## kpa (Jul 18, 2013)

Your test case isn't valid because you're forcefully overriding the number of jobs (-j 3) for all dependent ports. The proper way is to let the ports(7) system decide if a port is safe to build with multiple simultanious jobs and set MAKE_JOBS_NUMBER in make.conf(5) if you need to override the default. Take a look at /usr/ports/Mk/bsd.port.mk for details.

Edit: The default was changed very recently to assume that ports are safe to build with multiple jobs and there's a per port knob MAKE_JOBS_UNSAFE that can be set in the port Makefile to say that this port is not safe to build with multiple jobs. There is no need to use the -j flag of make(1) directly, the ports system will automatically use the optimal number of jobs for each port.


----------



## ShelLuser (Jul 18, 2013)

mjollnir said:
			
		

> You can read about this in the excellent article Recursive Make Considered Harmful


That is an interesting read in my opinion.

However, I think there is one flaw in your reasoning; but strictly speaking within the context of the Ports collection. Your suggested solution (Page 8) basically consists of "Use one Makefile" and throughout the paper it also focusses on the situation as a single project. Maybe a very complex and large project, but still a single project.

Yet that doesn't apply, not in the same extends anyway, to the Ports collection I think. Because, as the name somewhat implicates, its not merely a single project but more a whole collection of projects. Individual projects at that, which may indeed share a certain relationship (dependency). However, they leave it up to the upper structure to actually satisfy those dependencies. In a way it is fragmented by design.

Another issue I wonder about, but I have to admit that my knowledge of make isn't too extensive, is if this solution wouldn't result in a management disaster for something like the ports collection. For example I could imagine that if a certain committer make a small mistake it could potentially break the entire ports collection (when using one single Makefile it most certainly would I'd say), instead of simply breaking one single port in the collection.

Of course people aim not to make mistakes in the first place, but accidents do happen from time to time.

So within that perspective I'd rather have a system which may not be as optimal as it theoretically could be than a system which could be technically superior yet leaves much to desire where management is concerned.


----------



## Mjölnir (Jul 18, 2013)

kpa said:
			
		

> Your test case isn't valid because you're forcefully overriding the number of jobs (-j 3) for all dependent ports. The proper way is to let the ports(7) system decide if a port is safe to build with multiple simultanious jobs and set MAKE_JOBS_NUMBER in make.conf(5) if you need to override the default. Take a look at /usr/ports/Mk/bsd.port.mk for details.



Thanks for the hint! I did not know that, and applied it immediately in my build jail.

Anyhow, do it with your correction and it will surely fail directly, but with the portmaster wrapper, it will succeed (except a few hiccups due to human mistakes, that's normal).


----------



## wblock@ (Jul 18, 2013)

mjollnir said:
			
		

> Testcase: start a parallel build of a big ports collection directly, like
> `# cd /usr/ports/x11/kde4`
> `# make -j 3`



No, that will not work.  Not all ports are safe to build in parallel.  Ports that are safe to build in parallel are handled by the ports system.  Do not use -j with ports.

This is a good example of that "expectations bug" I was talking about.


----------



## Mjölnir (Jul 18, 2013)

ShelLuser said:
			
		

> That is an interesting read in my opinion.
> 
> However, I think there is one flaw in your reasoning; but strictly speaking within the context of the Ports collection. Your suggested solution (Page 8) basically consists of "Use one Makefile" and throughout the paper it also focusses on the situation as a single project. Maybe a very complex and large project, but still a single project.
> 
> Yet that doesn't apply, not in the same extends anyway, to the Ports collection I think. Because, as the name somewhat implicates, its not merely a single project but more a whole collection of projects.


Thus the thought arises that it could be possible to construct the Makefile for each collection I want to build on demand (based on the current status of the system), and meanwhile lock the building of others.  I _guess_ that's what portmaster does _somehow_, but it's doing it outside of make's awareness.  Maybe that's exactly the right thing to do; maybe not.  I already stated that portmaster works surprisingly well; nevertheless I suspect it's working on a non-sound foundation and that it applies many quirks to fix what is inheritably broken.  Disclaimer: this is an assumption, not an assault. 



			
				ShelLuser said:
			
		

> Individual projects at that, which may indeed share a certain relationship (dependency). However, they leave it up to the upper structure to actually satisfy those dependencies. In a way it is fragmented by design.


This fragmentation is dangerous, because it can break dependencies.  The traditional tool for this task is make, or the advanced CMake, and they rely on proper input.  I.e., non-fragmented.



			
				ShelLuser said:
			
		

> Another issue I wonder about, but I have to admit that my knowledge of make isn't too extensive, is if this solution wouldn't result in a management disaster for something like the ports collection. For example I could imagine that if a certain committer make a small mistake it could potentially break the entire ports collection (when using one single Makefile it most certainly would I'd say), instead of simply breaking one single port in the collection.


If the Makefile gets constructed on demand, it would only break builds of dependent ports, like now.



			
				ShelLuser said:
			
		

> So within that perspective I'd rather have a system which may not be as optimal as it theoretically could be than a system which could be technically superior yet leaves much to desire where management is concerned.


I do not say there's an easy solution to a very complex problem   Taking all the options, there are billions of billions of possible different FreeBSD systems...


----------



## Mjölnir (Jul 18, 2013)

wblock@ said:
			
		

> No, that will not work.  Not all ports are safe to build in parallel.  Ports that are safe to build in parallel are handled by the ports system.  Do not use -j with ports.
> 
> This is a good example of that "expectations bug" I was talking about.


And, BTW, it's an example that proves we're using make in a wrong way...  If the Makefiles were ok, parallel build would succeed because make can handle parallel builds for decades now.


----------



## wblock@ (Jul 18, 2013)

mjollnir said:
			
		

> And, BTW, it's an example that proves we're using make in a wrong way...  If the Makefiles were ok, parallel build would succeed because make can handle parallel builds for decades now.



Parallel port builds are now the default, and have actually been in use for quite a while.  "Does not do it the way I think it should" does not equal "broken".  But you know this.


----------



## Mjölnir (Jul 19, 2013)

I did not see any single argument from you why it's ok to build a house on an unstable ground, or why the ground I consider unsound, is sound.

If (the Makefiles were ok) then parallel make w/o a wrapper would succeed
That's simple logic => the Makefiles are not ok

If (the Makefiles were ok) then it would not build ports that are already installed and do not need to be updated over and over again
But it does => The Makefiles are not ok

And we need portmaster to fix that.  Still, as long you do not give it the right options, it builds too much.  All I do is asking why and I am sceptical that this is the right solution.


----------



## DutchDaemon (Jul 20, 2013)

One more time.

It is time to take your discussion to the freebsd-ports mailing list (join).

Closed.


----------

