# Dependency bloat in the ports collection



## dpecher (Nov 26, 2020)

Hello everybody, 

I'm sorry that my first post in here has to be a critical one, but I'm seriously nearing the end of my tether. We've come to a point where just compiling the bare-bones xorg port takes 12 hours and installs two different llvm versions and that infernal rust junk that compiles in geological terms and seems to be neccessary only for FreeBSD. I've never had to install that malarkey on any other system, ever. 

Normally I'm not even a regular FreeBSD user anymore - you've lost me when CLang was chosen to supplant a perfectly servicable compiler, but since I'm doing a lot of platform testing as a member of the CDE project team, I still have the latest non-eol'ed FreeBSDs running on my ESX rack. Theoretically I could install from binaries of course, but go back to FreeBSD 10 and some of the packages were compiled by Moses - not very helpful when testing a new software version of your own stuff. 

So here the question: What is it with that dependency bloat? Why do packages compile dependencies that they definitely do not need, and more importantly, is there any action planned to remedy this. At the moment we cannot in good conscience ask our users to compile the latest CDE version if building xorg and git ruins two perfectly usable days of your life. 

Sorry for the rant, 
Danilo Pecher


----------



## shkhln (Nov 26, 2020)

dpecher said:


> you've lost me when CLang was chosen to supplant a perfectly servicable compiler,


You lost whatever argument you had right there. The only thing you can expect from this point is more flame.



dpecher said:


> but go back to FreeBSD 10


Please, don't go back to FreeBSD 10, it's EOL.



dpecher said:


> So here the question: What is it with that dependency bloat? Why do packages compile dependencies that they definitely do not need, and more importantly, is there any action planned to remedy this.


Nothing will be done about that. Defaults are intentionally _not_ minimal to cover as many use cases as possible.



dpecher said:


> At the moment we cannot in good conscience ask our users to compile the latest CDE version if building xorg and git ruins two perfectly usable days of your life.


Who cares? (Xorg doesn't have dependency on Rust, by the way.)


----------



## SirDice (Nov 26, 2020)

dpecher said:


> We've come to a point where just compiling the bare-bones xorg port takes 12 hours


What are you using, a 486? Even my old Core i5 does this in less than an hour.



> and installs two different llvm versions


Nope.

```
root@molly:/usr/ports/x11/xorg # make all-depends-list | grep llvm
/usr/ports/devel/llvm10
```



> and that infernal rust junk that compiles in geological terms and seems to be neccessary only for FreeBSD.


Rust as a dependency of Xorg? Whatever gave you that idea? I think you got things quite backwards here.


dpecher said:


> I've never had to install that malarkey on any other system, ever.


Try building Firefox from source without it. On any other system.



dpecher said:


> Theoretically I could install from binaries of course, but go back to FreeBSD 10 and some of the packages were compiled by Moses


FreeBSD 10 has been EoL for a couple of years, what where you expecting?



dpecher said:


> not very helpful when testing a new software version of your own stuff.


Why are you testing on an EoL version in the first place?


----------



## getopt (Nov 26, 2020)

Folks, don't play this down or smile it away. While he could have written a little more conciliatory he has a point!

I have seen building three different versions of LLVM in Poudriere at a time. And the last Rust stupidity was using it as a standard dependency for LIBRSVG2.

But! Mostly the problems come from "upstream". What could be done fom ports maintainers is to make wise use of port options and prefer those form a minimalistic approach. 

Also I'd like to use a tool for reviewing the dependency-tree before setting port options. A lot can and should be done to make the dependency jungle more transparent for those building their packages from source.


----------



## shkhln (Nov 26, 2020)

getopt said:


> And the last Rust stupidity was using it as a standard dependency for LIBRSVG2.


It's not stupid, upstream has converted the whole library to Rust. You can no longer build it with a C compiler. (If you think this is a notable event, wait until you see the Rust version of libcurl in the ports.)


----------



## SirDice (Nov 26, 2020)

getopt said:


> And the last Rust stupidity was using it as a standard dependency for LIBRSVG2.


There are two different ports for it; graphics/librsvg2 and graphics/librsvg2-rust. These two conflict. Gnome for example seems to depend on the Rust version. I suspect Xorg may have changed accordingly because of this conflict. Or else you would never be able to install Gnome and Xorg packages together. Imagine the flak we would get from that.


----------



## getopt (Nov 26, 2020)

For time being you can use

```
DEFAULT_VERSIONS+=librsvg2=legacy
```
to prevent the Rust port. But this won't hold for eternity.


----------



## SirDice (Nov 26, 2020)

Updated my ports tree as I thought I may have been looking at old info but Xorg doesn't appear to depend on librsvg at all. So I'm still wondering why Rust would be a dependency of Xorg.


----------



## getopt (Nov 26, 2020)

I cannot see the OP relating Rust to Xorg. From my reading he addresses this in addition to the multiple LLVM-versions problem in the ports.

Having a base-LLVM should just be enough, IMO.


----------



## Argentum (Nov 26, 2020)

dpecher said:


> I'm sorry that my first post in here has to be a critical one, but I'm seriously nearing the end of my tether. We've come to a point where just compiling the bare-bones xorg port takes 12 hours and installs two different llvm versions and that infernal rust junk that compiles in geological terms and seems to be neccessary only for FreeBSD. I've never had to install that malarkey on any other system, ever.


Hard to tell, what exactly are you doing, but just out of curiosity just tried to recompile my X-org. It took less than 2 minutes.


----------



## getopt (Nov 26, 2020)

Argentum said:


> It took less than 2 minutes.


Spare your mouthful swaggering as this is in no way helpful at all.
I'm just tired reading this works-here-mentality in forums.


----------



## Argentum (Nov 26, 2020)

SirDice said:


> Updated my ports tree as I thought I may have been looking at old info but Xorg doesn't appear to depend on librsvg at all. So I'm still wondering why Rust would be a dependency of Xorg.


True.

`pkg_tree -v xorg-server-1.20.9,1|grep librsvg` gives nothing.

`pkg_tree xorg-server-1.20.9,1`


```
xorg-server-1.20.9,1
|\__ xkeyboard-config-2.31
|\__ xkbcomp-1.4.3
|\__ pixman-0.40.0_1
|\__ libxshmfence-1.3
|\__ libxkbfile-1.1.0
|\__ libXdmcp-1.1.3
|\__ libXau-1.0.9
|\__ libXfont2-2.0.4
|\__ openssl-1.1.1h_1,1
|\__ mesa-libs-20.2.0_1
|\__ mesa-dri-20.2.0_1
|\__ libepoxy-1.5.4
|\__ libdrm-2.4.103,1
|\__ libunwind-20200331_1
|\__ libudev-devd-0.4.2_1
 \__ libpciaccess-0.16
```


----------



## ShelLuser (Nov 27, 2020)

While the OP seems to be a bit of a drama queen he still has a fair point when it comes to LLVM:


```
peter@vps:/home/peter $ pkg info -x llvm
llvm10-10.0.1_3
llvm11-11.0.0
llvm90-9.0.1_3
```
This is the situation on my VPS server and I don't have anything out of the ordinary running here, merely Apache backed up by Mono, Java (through Tomcat) and PHP using PostgreSQL and MySQL as backend database systems. Mail is handled by Postfix. To add insult to injury I even use DEFAULT_VERSIONS in /etc/make.conf where I have llvm set to 90, as suggested by /usr/ports/Mk/bsd.default-versions.mk.

Yet here we are. Of course now that I have these versions installed I won't be confronted with another build any time soon, but it's still pretty ridiculous to have 3 versions of the same thing installed, especially considering that the version within base could have handled things as well.

There are moments when it truly seems as if FreeBSD is more busy building compilers than the actual software we want to use. It's annoying for sure, but not something to start a whole drama over I think. But I do agree that some quality guidelines might be a good thing here.

At the very least have ports honor /etc/make.conf.


----------



## Argentum (Nov 27, 2020)

ShelLuser said:


> While the OP seems to be a bit of a drama queen he still has a fair point when it comes to LLVM:
> 
> 
> ```
> ...


Agree! The only question is, *who should deal with this problem*?

BTW, in my laptop - `pkg info -x llvm`


```
llvm10-10.0.1_3
llvm11-11.0.0
llvm80-8.0.1_4
llvm90-9.0.1_3
```

And looking `portversion -vr llvm10-10.0.1_3`

has the only llvm which has dependency:

```
[Reading data from pkg(8) ... - 1460 packages found - done]
llvm10-10.0.1_3             =  up-to-date with port
mesa-dri-20.2.0_1           =  up-to-date with port
```


----------



## tingo (Nov 27, 2020)

getopt said:


> But! Mostly the problems come from "upstream". What could be done fom ports maintainers is to make wise use of port options and prefer those form a minimalistic approach.


True. But they didn't do that. As a post on the freebsd-ports mailing list recently told, it seems like the policy / accepted practice for ports / port maintainers is to follow upstream as close as possible.

TL;DR - yes there is a "dependency bloat" in software today. Everybody is doing it, and it fits nicely in with the general bloat (size, functionality, ++) of today's software. Nobody is working to keep things small, efficient, resource friendly and what have you. Sadly.


----------



## sidetone (Nov 28, 2020)

While the person got major details wrong. It is crazy to need to reinstall LLVM for using a graphics driver with Xorg.

Some programs want a different version of LLVM. Which there should only be one needed. Firefox and Thunderbird are Gecko ports which require Rust. Rust is fine, but it's crazy, when the newest version is wanted. I use pkg to install Rust, and the latest available versions of LLVM. When building a program that wants the latest LLVM or Rust, and the wanted version isn't available from packages, I use portmaster with the -i option to forgo (re/-)building LLVM or Rust from ports.


----------



## facedebouc (Nov 28, 2020)

On my desktop computer I don't see those dependencies you are describing here.
I only have one llvm :

```
$ pkg info -x llvm
llvm80-8.0.1_4
```
and no rust.
It's an only packages installed system. I don't compile any ports.
My packages are built with ports-mgmt/poudriere for all my FreeBSD computers.
My repository is hosted on a VPS and it's great.


----------



## drhowarddrfine (Nov 29, 2020)

It seems, then, that the OP is building a desktop from ports rather than using packages. 
If the maintainers did not stick closely to upstream, then we'd hear complaints about how their software doesn't work on their system for all kinds of reasons.


----------



## kpedersen (Nov 29, 2020)

For most of my installs I generally just use packages and yet often end up with a considerable number of llvm versions (only slightly less than python installs). Most of them came from the xorg meta-package and dependencies due to I believe some of the AMD code generation stuff.

That said I have a fairly fresh install and only have 2 llvm packages installed suggesting that things have improved recently.


----------



## dpecher (Nov 30, 2020)

SirDice said:


> Updated my ports tree as I thought I may have been looking at old info but Xorg doesn't appear to depend on librsvg at all. So I'm still wondering why Rust would be a dependency of Xorg.



It comes in at compiling gobject-introspection, which in turn seems to have been adde through ticking the vmware-driver in the x11-drivers menu. I already paid attention not to add any features and leave everythingat default, but when running FreeBSD in a virtual machine, you sort of have to add the vmware graphics and mouse drivers, and that's when half the ports tree seems to be added, including llvm9 and rust. That's an instant 4 hour time penalty on an 8th gen i5.


----------



## dpecher (Nov 30, 2020)

drhowarddrfine said:


> It seems, then, that the OP is building a desktop from ports rather than using packages.
> If the maintainers did not stick closely to upstream, then we'd hear complaints about how their software doesn't work on their system for all kinds of reasons.


Yes, I build the x11/xorg package from source, yes. But that seems to be quite effective in building half the port tree by the look of it. 

I run FBSD11 and FBSD12 in VMs on an ESX rack and installing the binary packages can leave you with fairly outdated packages (for instance binary xorg install pulls llvm8, while building from source pulls llvm 10 and llvm 9). I use these machines as build servers for our CDE sources. That they work with llvm8, well we tested that half a year ago. What we need to know is whether they build against the latest versions, so regular rebuilds are neccessary. For instance, the latest Gcc10 wrecked our builds, which you wouldn't find out going by using binary pkgs. (This last topic is not specific to FreeBSD though. It mainly concerned NetBSD, and a load of Linuces).


----------



## SirDice (Dec 1, 2020)

dpecher said:


> installing the binary packages can leave you with fairly outdated packages


They're just as up to date as everything else. Note that there's a difference between quarterly packages (updated once every three months) and latest (which follows the ports tree as close as possible). What you think is "old" is probably from the quarterly branches, switch your packages to latest.


----------



## Jose (Dec 1, 2020)

shkhln said:


> Nothing will be done about that. Defaults are intentionally _not_ minimal to cover as many use cases as possible.


This is a shame. Do we really need _two_ versions of LLVM to compile editors/vim?


			'editors/vim needs devel/llvm90 and devel/llvm10' - MARC
		


Do I really want to enable Wayland everywhere because someday someone might use it?


			Vote: making wayland=on default


----------



## shkhln (Dec 1, 2020)

Jose said:


> This is a shame. Do we really need _two_ versions of LLVM to compile editors/vim?
> 
> 
> 'editors/vim needs devel/llvm90 and devel/llvm10' - MARC
> ...


You _don't have to_ enable anything if you build packages from source. "Minimal" _defaults_ would mean less people able to use the official binary packages, which would lead to more people rebuilding LLVM, which in turn means _more_ complaints.


----------



## acheron (Dec 1, 2020)

Yet another troll?


----------



## sidetone (Dec 1, 2020)

A lot of things require Rust. A lot of programs will require Rust anyway. I still don't get why you keep saying you need Rust for Xorg or its drivers. Have things changed that much radically most recently for that to be needed for it? Whenever I rebuilt my Xorg recently, things break.

As for LLVM, different sets of packages insist on two different versions of it. Actually, all that's really needed is not the full compiler, just the build utilities equivalent to binutils. GNU Binutils will actually do (or has recently). They put utilities and compilers into a monolithic package for each version.

While it's often discouraged to tinker with /usr/ports/Mk/ and /usr/ports/Mk/Uses/, versions for all programs can be set to one needed ports LLVM, then give feedback to maintainers or the mailing list on if that works. Then, be ready to overwrite the ports/ directory with a fresh `svnup` or `svn`. The one that won't likely build is the Xorg drivers that actually do require the latest version of LLVM (actually its utilities). So, it should only need the base LLVM plus the latest one from ports (but not necessarily the latest build). Furthermore, it's likely only the latest utilities for LLVM that are needed from the latest version, which aren't separated from the LLVM compiler.

Maybe the solution is to ask for them to separate LLVM's utilities from the LLVM compiler build. So that dependencies don't ask for hours of additional compile time, when something actually needs the latest compiler utilities, and not the whole LLVM set. Also, so that the compiler in the base system is enough for everything requiring LLVM/Clang.

GNU Binutils at a recent time had the needed components to build everything on various processors. LLVM's Clang utilities are limited, many parts of it only work for a limited amount of processors, and parts of it are not finished. Parts of LLVM Clang build for everything across all processors. When a small improvement is made, they set it to download a newer update of LLVM/Clang by default.

LLVM/Clang itself isn't the problem. The problem is the utils which are unfinished. I believe parts from GNU binutils are used to fill in the gaps. Not all programs need all of the build utilities or the latest ones, but some that ask for the latest LLVM version do or have in a most recent time.


The quickest fix is to see which versions of LLVM and Rust are wanted, cancel that build, then install these two (plus a Rust dependency) from packages. Then use the -i option with `portmaster` for your desired program to avoid rebuilding LLVM and Rust, when a package for the latest build for these is unavailable. Using packages from the lang category for the most part doesn't interfere with building from ports.


----------



## jb_fvwm2 (Dec 2, 2020)

Diverting just a little, recently some new ports which require rust want to rebuild it even though it is already installed, meaning "wait and see if there is a package"


----------



## sidetone (Dec 2, 2020)

sidetone said:


> use the -i option with `portmaster` for your desired program to avoid rebuilding LLVM and Rust, when a package for the latest build for these is unavailable.


----------



## dpecher (Dec 2, 2020)

Right, two days and a bit of down-winding later. First, I appologize for the tone of my first post. Never post when you're po'ed about something. 

I understand that most people install from binary packages, which of course is the sensible option when you configure a machine that you actually want to use, but that's not the purpose of the machine I was ranting about. I should probably have made that clearer. 

I'm doing the build testing for CDE, the legacy Unix desktop that many old hacks like myself started their career on and still can't let go of. This being legacy software, you are of course vulnerable to suffer breakages from newer packages. Case in point: GCC 10, which made -f-no-common a default flag. That in turn wrecked all our builds on every distro that updated to gcc10. (Mainly the Redhat and Debian derivates), so I need to check the latest upstream builds. 

And this is where my disappointment with the current state of OSS comes in. The FreeBSD ports are only the most extreme example, other package management systems have the same problems to a degree - feature creep and dependency bloat. 

Take VIM for example. That thing has GUI options ranging from Xaw to GTK3 - guess which one is the default? Right, GTK3, so, unless you wade through one hundred eleventy config menus or write an mk.conf on NetBSD or make.conf on FreeBSD, which then isn't always honoured by the look of things, you have a hard time automating stuff, and if you go by default, you build half the OSS world that you then don't need. I guess that's why people moved away from building from source in the first place. 

So my wish here would be to go with sensible minimal defaults. If people want all the bells and whistles, let them *add *the features, not requiring me to disable them. Every added feature usually introduce whole smorgasboard of new dependencies. Why not go for the minimum? In the case of VIM that would be no GUI at all. Please let me decide that I want one, and if so, which one. 

As for librsvg - I really need to bite my lip on that one. Sure, the authors of it have every right in the world to ditch a language that has worked perfectly well for decades (and still does) in favour of going for the latest fad - all power to them, but it exposes a problem that has been around in the opensource world for quite a time now: people don't care about other people's work as much as they should. 

The success of an OSS project (especially a library) is measured by how many people choose to use it for their project. In that regard, I'd say librsvg doesn't do too badly. But that success comes with consequences. If you make a major decision for your project, it affects all those who chose to rely on it. So by going rust, you force other people to abandon their reliance on your software or deal with stuff they never saw coming and were never asked about. That, and a 2 hour time penalty when building the now new dependencies for your own project. Not to mention that you suddenly depend on code written in a language you are not overly familiar with. And if I learned something in the 80s - don't mix languages. Everybody who ever tried to import a Turbo-Pascal library into a Turbo-C project will know what I mean. 

What would have been the damage in just sort of sealing librsvg as a sort of 'finished project', so people who are satisfied with it can stick with it, and rustifying it under a new name? librsvg-ng or something.

Sorry, I was rambling again, but these are my thoughts on it.


----------



## getopt (Dec 2, 2020)

dpecher said:


> Sorry, I was rambling again, but these are my thoughts on it.


No! No! No rambling at all.  Your thoughts are highly welcome!


----------



## kpedersen (Dec 2, 2020)

Agreed. I think your "rambles" are pretty much on the mark too. I am also fairly annoyed by the sheer number of dependencies that software these days bring in. Part of this is the nature of "modern" software being a sprawling mess and the other part of this is the FreeBSD ports system being a little too inclusive of features that only one or two people care about.

The issue is that I am not sure there is a solution that will please everyone. I personally would like a stripped ports collection where the absolute minimum of features (and thus dependencies) is the default. However this would likely upset a few people. For example would Vim support the X11 clipboard? Would Mutt support imap?

I was on the original team that open-sourced CDE (one of my patches also has a visual bug on the dtpanel that I plan to fix too!). However I kind of pulled away once clang came in vs gcc and I was far too over my head in terms of time needed to patch it to compile. I have much respect for those who achieved this!

In fact I am recently developing a UI toolkit with zero dependencies other than libX11 to try to address this very problem (albeit in a minor way for now).

You might consider using packages but then use ports only for "leaf" software. So any software that is not depended on by others is generally safe to be mixed with packages. I tend to do this. A recursive make on any port is normally... not a good use of the family computer time XD.


----------



## Jose (Dec 2, 2020)

dpecher said:


> As for librsvg - I really need to bite my lip on that one. Sure, the authors of it have every right in the world to ditch a language that has worked perfectly well for decades (and still does) in favour of going for the latest fad - all power to them, but it exposes a problem that has been around in the opensource world for quite a time now: people don't care about other people's work as much as they should.


The commit that added the non-rust version to default versions mentions that there are platforms on which Rust is not available. I'm hoping this leads to a fork. Hopefully there are enough people that care about resource-constrained environments where Rust doesn't make sense. I'm not against Rust on principle. It's clearly not ready for prime time, though.


----------



## acheron (Dec 3, 2020)

jb_fvwm2 said:


> Diverting just a little, recently some new ports which require rust want to rebuild it even though it is already installed, meaning "wait and see if there is a package"


Which ports?


----------



## acheron (Dec 3, 2020)

Jose said:


> The commit that added the non-rust version to default versions mentions that there are platforms on which Rust is not available. I'm hoping this leads to a fork. Hopefully there are enough people that care about resource-constrained environments where Rust doesn't make sense. I'm not against Rust on principle. It's clearly not ready for prime time, though.


Why does rust doesn't make sense on a resource constrained environment? llvm11 or gcc10 are better in this regard? Do you know how many gigabytes of disk space / ram llvm11 requires to build?


----------



## jb_fvwm2 (Dec 3, 2020)

acheron said:


> Which ports?


accessibility/sctd, for example [ a few others similar ]


----------



## Jose (Dec 3, 2020)

acheron said:


> Why does rust doesn't make sense on a resource constrained environment? llvm11 or gcc10 are better in this regard? Do you know how many gigabytes of disk space / ram llvm11 requires to build?


It's not just that you have to have Rust in addition to at least one of those two, it's also that compiling with Rust takes far longer and is much more resource intensive than compiling C.


----------



## acheron (Dec 4, 2020)

Jose said:


> it's also that compiling with Rust takes far longer and is much more resource intensive than compiling C.


Proof?


----------



## SirDice (Dec 4, 2020)

In case people have been following along and are wondering what happened. Apparently our tolerance was mistaken for an inability to act. As a result getopt received a couple of warning points and a temporary ban. The disruptive posts have been removed.


----------



## Jose (Dec 4, 2020)

acheron said:


> Proof?


Purely anecdotal from me watching top(1) as my system builds things. It's a pretty beefy system, and things that build in Rust manage to get its attention for a nontrivial amount of time.

I also suspect Rust's piggishness is what's causing Firefox to drop out of the package repository:




__





						'Re: 12.2-STABLE pkg upgrade remove firefox then need firefox-esr' - MARC
					





					marc.info
				




But I can't prove it.


----------



## SirDice (Dec 4, 2020)

I don't think that has anything to do with Rust in particular. I've been having build problems due to resource starvation for a while too, not all of them related to Rust. I had a period when building actually caused the build server to crash and reboot. Quite annoying. Seems to have settled down a bit now, at least for my builds, some things still take forever but at least its not crashing halfway through a poudriere run any more.


----------



## facedebouc (Dec 4, 2020)

I am building quarterly ports with ports-mgmt/poudriere on my desktop machine while using it for other purpose. I only have 8Gb of memory and use 3 cores of my AMD FX6300. In /usr/local/etc/poudriere.conf I have the setting `USE_TMPFS=no` and don't go out of swap space (8G). My filesystem is ZFS but  I am limiting ARC with `vfs.zfs.arc_max="3G"` in /boot/loader.conf.


----------



## facedebouc (Dec 4, 2020)

I have to say that building www/firefox-esr launches rust instances on all my cores which is greatly slowing down my computer despite starting poudriere with `nice +20`. But after a while all is returning to normal situation.


----------



## sidetone (Dec 5, 2020)

Maybe only an upgrade to Clang or build utils is needed. LLVM and Clang are in the same port or package. LLVM shouldn't have to be recompiled to get the recent version of Clang or the build utilities (Bin/Elf-utils) in the case where only a port demands it, rather than the user wants to use the whole set of LLVM/Clang.

Is it correct that Rust uses LLVM that's in base or the default one in ports? I know it's said that Rust uses LLVM. I'm trying to see if this is how it's implemented from using the one in base or the default one from make.conf.


----------

