# Ports Build Time



## markbsd (Nov 7, 2013)

Is there a database or even a thread of average/expected build times for individual ports (including build and run dependencies)?

This would be a really interesting data set. It may influence users' decisions to install packages or build ports. Or even to determine possible problems such as abnormally long build times.

I'm convinced something is wrong with either portmaster, or this install of FreeBSD; I've just reached 6.5 hours (and counting) of elapsed time building xfce4 using portmaster! Something has to be wrong! This cannot be normal.


----------



## Juanitou (Nov 7, 2013)

Why 6.5 hours would be wrong instead of 10 minutes or 10 days? It all depends on your computer, the code being compiled, the dependenciesâ€¦ and over all, your patience and the expectations related to systems providing binary installs/updates!

Nothing is wrong with portmaster.


----------



## markbsd (Nov 7, 2013)

I think it is wrong because last time I ran `make install clean` from x11-wm/xfce4 it did not take even half as long. And it is on the same computer. My patience is running thin, but this doesn't change the fact that it's been over 6.5 hours to compile Xfce4


----------



## trh411 (Nov 7, 2013)

ports-mgmt/portmaster will always install any dependent ports before installing the specified port.

If you are installing x11-wm/xfce without first having installed x11/xorg, then portmaster will have to install x11/xorg first, and that could take a couple to a few hours depending on the hardware. There is nothing wrong with ports-mgmt/portmaster.

If one's only interest is speed, then one should install packages because that is almost always faster then building ports.


----------



## markbsd (Nov 7, 2013)

x11/xorg was already installed, I did that overnight whilst sleeping so don't know how long that took, though it was running for at least a couple hours before I went to sleep. You must be right, using portmaster is in effect the same as `make install` anyway, right?

Speed isn't my only concern, but I am shocked at how long this is taking. I should have just went with openbox and customized my own desktop.


----------



## trh411 (Nov 7, 2013)

If x11/xorg was already installed, then I agree that 6 hours and counting to install x11-wm/xfce is excessive. If ports-mgmt/portmaster is still spewing output to the shell session, I would let it keep running.

Does `top` show anything instructive about other unusual activity on the system?


----------



## markbsd (Nov 7, 2013)

It's still furiously spewing out compiling nonsense. `top` shows nothing unusual; cc/make/gmake or similar, rc.conf options (ntpd, wpa_supplicant, powerd, etc), and a half dozen other basic activities (Sendmail, sh, cron, syslogd, etc).

I'm wondering if I've selected anything weighty in the config after running `portmaster x11-wm/xfce4`, but I really only selected maybe 4 or 5 things (which I can't even remember now).

Plus, this is a completely fresh install of the base system. I literally did a brand new install, ran:


```
# portsnap fetch
# portsnap extract
# portsnap update
# portsnap fetch update
```

switched to PKGNG (then stupidly updated PKGNG when I already had the latest version!), installed portmaster, then ported xorg and now xfce4. The system has done nothing else at all.

Maybe I should abort it, run another clean install and start again?


----------



## markbsd (Nov 7, 2013)

Plus, from glancing at the screen from time-to-time, it appears to be installing a heap of junk I would have thought wasn't needed; gnutls, glib-networking, pkcs11, libsoup...

I mean, what the hell are these things?


----------



## trh411 (Nov 7, 2013)

As ports-mgmt/portmaster makes its way through the installation of the specified and all dependent ports, it will show its progress, e.g. 13/68, which means it is working on the 13th of 68 ports. That will give you an idea of how far along it is. Can you tell how far along the install it is?

You have to be quick to see this info, as it scrolls by pretty fast.


----------



## jb_fvwm2 (Nov 7, 2013)

markbsd said:
			
		

> Is there a database or even a thread of average/expected build times for individual ports (including build and run dependencies)?


Maybe a subforum, with just threads each pertaining to a port, where users could post fixes/information. For example, pre-[cmd=] portmaster [/cmd] I ran ports with a script which left information in the port directory after a build:
 /usr/ports/devel/boost-libs 
5:54 - 6:36
18:00 - 21:10 
12:29 - 13:00
I surmise it [threads] would be better to read through than a wiki.  Don't know how easy it would be to have them all alphabetical, though, or some other schema. And may lessen the number of PR's sent.


----------



## markbsd (Nov 7, 2013)

Thanks, @trh411. I'll try to catch it. Is there not a command I can run from another terminal to get an indication?


----------



## markbsd (Nov 7, 2013)

Wow! 73/210 -- this can't be right? WTF? Now 76/210! I feel like crying  What the hell must I have selected to have over 200 ports? This is madness!


----------



## trh411 (Nov 7, 2013)

markbsd said:
			
		

> Plus, from glancing at the screen from time-to-time, it appears to be installing a heap of junk I would have thought wasn't needed; gnutls, glib-networking, pkcs11, libsoup...
> 
> I mean, what the hell are these things?!



They are either build or run-time dependencies required by x11-wm/xfce. Sometimes the most innocent looking port can pull in tons of dependencies when you build it. But that is one thing I like about using ports-mgmt/portmaster. It shows you everything it will install before it starts, giving you a chance to "opt-out" if you don't like it.


----------



## markbsd (Nov 7, 2013)

jb_fvwm2 said:
			
		

> Maybe a subforum, with just threads each pertaining to a port, where users could post fixes/information. For example, pre-[cmd=] portmaster [/cmd] I ran ports with a script which left information in the port directory after a build:
> /usr/ports/devel/boost-libs
> 5:54 - 6:36
> 18:00 - 21:10
> ...



This is a really good idea!


----------



## trh411 (Nov 7, 2013)

markbsd said:
			
		

> Wow! 73/210 -- this can't be right? WTF? Now 76/210! I feel like crying  What the hell must I have selected to have over 200 ports? This is madness!



When I ran through the pre-install check for x11-wm/xfce4 on my system, it showed 68 dependent ports would be installed. Since I have several hundred ports already installed, I'm guessing I had many of the dependent ports already installed. Since you are starting from a smaller base, it's got a lot to install.


----------



## markbsd (Nov 7, 2013)

trh411 said:
			
		

> They are either build or run-time dependencies required by x11-wm/xfce. Sometimes the most innocent looking port can pull in tons of dependencies when you build it. But that is one thing I like about using ports-mgmt/portmaster. It shows you everything it will install before it starts, giving you a chance to "opt-out" if you don't like it.



I think I've foolishly selected an innocuous looking "dev" option, which has now screwed me. I mean, Xfce4 only has 13 run-time dependencies, no way in hell there is ~200 build dependencies!


----------



## markbsd (Nov 7, 2013)

trh411 said:
			
		

> When I ran through the pre-install check for x11-wm/xfce4 on my system, it showed 68 dependent ports would be installed. Since I have several hundred ports already installed, I'm guessing I had many of the dependent ports already installed. Since you are starting from a smaller base, it's got a lot to install.



So it is entirely possible this is just normal and I haven't selected something I shouldn't have?

I've half a mind to abort this, reinstall FreeBSD, install xorg-minimal and start again 

What a waste of an entire afternoon and night!


----------



## markbsd (Nov 7, 2013)

Thing is, I can't tell how long x11/xorg took as I went to sleep. It would be good to know if this is taking longer than xorg, because wouldn't you think xorg should take longer than xfce4?


----------



## trh411 (Nov 7, 2013)

I can't help but wonder whether one of the dependent ports of x11-wm/xfce might require devel/gcc to build instead of the default devel/clang. On my system it takes almost 3 hours to build devel/gcc.


----------



## markbsd (Nov 7, 2013)

I don't know. I'm sure I saw gcc -- or maybe it was cc1 -- in `top` so you may be right. Either way, it's now been 7 hours and it's only up to ~80 of the 200+. Only 80 dependencies installed in 7 hours is ridiculous! Now it's running python2.7 -- what the hell?

I either abort it right now and reinstall FreeBSD before going to sleep. Or, let it run and just go to sleep. By morning it will have to be finished and I can then delete x of the absurd 200 odd dependencies installed and get on with my life.

I can do a fresh install and start the xorg-minimal `make install clean` in around 30 minutes then go to bed, wake up and get openbox going. I can't decide!


----------



## trh411 (Nov 7, 2013)

Well, since you already have X installed, you might want to leave it as is and consider a more lightweight window manager instead of a full blown desktop like x11-wm/xfce.

You may be interested in the following thread from the *Howtos* Section of the forum which walks you through building a minimal desktop environment Minimal FreeBSD Desktop.


----------



## wblock@ (Nov 7, 2013)

We still don't know how fast your computer is, or what the situation was.  Did you start with no ports installed?  Were there outdated packages installed?  We know you had deleted build dependencies, so those have to be built.

So: stop.  Take a deep breath.  Don't obsess over the number of ports, it's not really indicative of anything.  Many of the standard dependencies can take a while to build but then are mostly static.  For example, gettext takes a while to build, but does not change often.  So once it's installed, it does not need to be installed again.

Switch to another console and try `ps axww | grep portmaster` to see what it is doing.  Or look in the compile output.  Scroll Lock can be used to pause the output.  When paused, the arrow keys and Page Up/Page Down can be used to scroll around.


----------



## kisscool-fr (Nov 7, 2013)

Do you have any option set in your portmaster's configuration files?


----------



## Beastie (Nov 7, 2013)

markbsd said:
			
		

> What the hell must I have selected to have over 200 ports??? This is madness!


Have you ever seen the number of dependencies Xfce has?



			
				markbsd said:
			
		

> I've half a mind to abort this, reinstall FreeBSD, install xorg-minimalist and start again


DON'T!!! This is totally unnecessary. What's wonderful about FreeBSD is that the base system and third-party ports are entirely separate entities.
You can get back to a clean install with 2 simple commands:
`# rm -R /usr/local`
`# rm -R /var/db/pkg`



			
				markbsd said:
			
		

> What a waste of an entire afternoon and night!


"Experience is one thing you can't get for nothing." (Oscar Wilde)
"Experience is what you get when you didn't get what you wanted." (Randy Pausch)

Stay positive


----------



## wblock@ (Nov 7, 2013)

Beastie said:
			
		

> You can get back to a clean install with 2 simple commands:
> `# rm -R /usr/local`
> `# rm -R /var/db/pkg`



The short version is `pkg_delete -a`.


----------



## Deleted member 9563 (Nov 7, 2013)

Beastie said:
			
		

> Have you ever seen the number of dependencies Xfce has?



Good point. In my (little) experience, the very large GTK system can be time consuming to compile.


----------



## SirDice (Nov 7, 2013)

If you can do without it's GNOME compatibility you can turn off GVFS in x11-fm/thunar. You might want to turn off mousepad too. That seems to depend on a lot of GNOME stuff too nowadays.


----------



## markbsd (Nov 8, 2013)

trh411 said:
			
		

> Well, since you already have X installed, you might want to leave it as is and consider a more lightweight window manager instead of a full blown desktop like x11-wm/xfce.
> 
> You may be interested in the following thread from the *Howtos* Section of the forum which walks you through building a minimal desktop environment Minimal FreeBSD Desktop.



Thanks for that link! I'm going to do that.



			
				wblock@ said:
			
		

> We still don't know how fast your computer is, or  what the situation was.  Did you start with no ports installed?  Were  there outdated packages installed?  We know you had deleted build  dependencies, so those have to be built.
> 
> So: stop.  Take a deep breath.  Don't obsess over the number of ports,  it's not really indicative of anything.  Many of the standard  dependencies can take a while to build but then are mostly static.  For  example, gettext takes a while to build, but does not  change often.  So once it's installed, it does not need to be installed  again.
> 
> Switch to another console and try `ps axww | grep portmaster`  to see what it is doing.  Or look in the compile output.  Scroll Lock  can be used to pause the output.  When paused, the arrow keys and Page  Up/Page Down can be used to scroll around.



It's an old notebook, about 10 years old. I can't even tell you the hardware off the top of my head; 2 GB RAM, 50 GB HDD, P_x_

I started with nothing. Fresh install >> PKGNG >> Portmaster >> Xorg >> Xfce4

Thanks, @wblock@. `ps axww` reveals that portmaster is currently working on:

libxfce4menu
glade3
atlas
py-gtk2
py-numpy
xfce4
thunar
squeeze
xfce4-panel
libexo



			
				kisscool-fr said:
			
		

> Do you have any option set in your portmaster's configuration files?



I do:


```
#Always delete stale distfiles without prompting (-d)
ALWAYS_SCRUB_DISTFILES=dopt
# Be verbose (-v)
PM_VERBOSE=vopt
# Install packages for build-only dependencies (--packages-build)
PM_PACKAGES_BUILD=pmp_build
# Delete build-only dependencies when finished (--delete-build-only)
PM_DEL_BUILD_ONLY=pm_dbo
#Suppress the build confirmation message (--no-confirm)
PM_NO_CONFIRM=pm_no_confirm
```

Maybe I've enabled some option that is making this process excruciatingly long?



			
				Beastie said:
			
		

> Have you ever seen the number of dependencies Xfce has?
> 
> DON'T!!! This is totally unnecessary. What's wonderful about FreeBSD is  that the base system and third-party ports are entirely separate  entities.
> You can get back to a clean install with 2 simple commands:
> ...



Apparently, it only has 13 run dependencies. Not sure how many build.

Thanks for the advice (and words of encouragement). So, if I CTRL+C this process, I can revert to a clean base system with those two commands?



			
				wblock@ said:
			
		

> The short version is `pkg_delete -a`.



I think this needs to be done...

The Xfce4 port build is STILL GOING! That makes run-time 20 hours and counting! LOL

Something has to be wrong, this is absurd!


----------



## trh411 (Nov 8, 2013)

markbsd said:
			
		

> It's an old notebook, about 10 years old.



I have a 5 year old Intel Core2Duo system. A state-of-the-art Intel i7 quad core system is nearly 5 times faster (according to popular CPU benchmarks) and uses memory that is twice as fast as mine. It takes me about 2 hours to install the entire x11/xorg meta port.

Extrapolating backwards, your system is probably 1/5th as fast as mine, which means about 10 hours to install the entire x11/xorg meta port.

Certainly not scientific, but I have a better understanding now why your builds run so long.

Maybe you should install packages?


----------



## markbsd (Nov 8, 2013)

I think I will. I'm going to abort this failure, delete everything, and use `pkg install`.

What is the pkg command equivalency of `pkg_delete -a`.

Alternatively, I might give FreeBSD a chance on one of my newer systems. Last time I tried it refused to cooperate.


----------



## Juanitou (Nov 8, 2013)

markbsd said:
			
		

> The Xfce4 port build is STILL GOING??! That makes run-time 20 hours and counting! LOL
> 
> Something has to be wrong, this is absurd!



The first time I built a complete Xfce4 desktop with Gimp, Inkscape, Firefox, Java, etc. it took at least three full days on a salvaged old computer. Thereâ€™s nothing wrong or absurd, itâ€™s just slowâ€¦ :e


----------



## markbsd (Nov 8, 2013)

I can't believe this is normal! It's only up to 118/210! At this rate, just Xfce4 -- never mind a browser and Java -- will take over two days! Wow


----------



## trh411 (Nov 8, 2013)

markbsd said:
			
		

> I think I will. I'm going to abort this failure, delete everything, and use `pkg install`.
> 
> What is the pkg command equivalency of `pkg_delete -a`.
> 
> Alternatively, I might give FreeBSD a chance on one of my newer systems. Last time I tried it refused to cooperate.



Just drop the underscore ... `pkg delete -a`. This command will delete all installed packages from the system and clear the database.


----------



## wblock@ (Nov 8, 2013)

markbsd said:
			
		

> ```
> #Always delete stale distfiles without prompting (-d)
> ALWAYS_SCRUB_DISTFILES=dopt
> # Be verbose (-v)
> ...



Yes, deleting build dependencies is a mistake.  But that won't have an effect until the next time.



> Apparently, it only has 13 run dependencies. Not sure how many build.



Sure.  However, those ports also have build and run dependencies.  A typical system with X and xfce will have something like 400-500 ports installed.

If you were to stop and then restart the build, all of the ports it had already installed would not need to be installed again.  But your settings to delete build dependencies will defeat that, forcing them to be rebuilt.  That includes big things that are slow to compile, like gcc-6.4 at present.  Deleting all ports magnifies that, discarding all the work that has been done.

If you stop throwing away all the work each time, it will eventually complete and then take much less time to update or add.


----------



## kpa (Nov 8, 2013)

When I was still using the plain ports(7) and ports-mgmt/portmaster to build my ports I found it very helpful to figure out the most common run/build time dependencies and install them separately before launching bigger builds. One very helpful tool is `make all-depends-list` for a big port like x11/xorg. Also `make missing` is very helpful. Intersecting outputs from those when run on different ports should give you an idea what those very common ports are. From memory I can list at least a couple that are used all the time: lang/perl5.16, lang/python27, devel/autoconf, devel/automake, devel/libtool, devel/gmake, devel/gettext and many others.


----------



## xibo (Nov 8, 2013)

trh411 said:
			
		

> I have a 5 year old Intel Core2Duo system. A state-of-the-art Intel i7 quad core system is nearly 5 times faster (according to popular CPU benchmarks) and uses memory that is twice as fast as mine. It takes me about 2 hours to install the entire x11/xorg meta port.
> 
> Extrapolating backwards, your system is probably 1/5th as fast as mine, which means about 10 hours to install the entire x11/xorg meta port.



Extrapolation is no good here, as ports installation times are affected by numerous sources. My dual Xeon E5-2650v2s (512GB memory) can build KDE4 with all options enabled and all dependencies in less than two hours if it's installing from/to it's zfs raid, but it takes more than a day to install them on the USB media it normally uses only for the bootloader and kernel.

On a modern HDD or SSD, the GEOM buffer drains that finally write data on the storage media after the kernel had taken responsibility of them will likely occur while make already is working at the next port, which eventually even builds on a filesystem backed by another GEOM, i.e. make is not at all affected by the ${PREFIX} media being slow.
On slow storage media (such as old HDDs or my USB flash media), however, the draining can take place while the data is still actively being written, blocking the install or cp proccess and slowing not only the installation process but the whole system, as every access on the draining GEOM that cannot be proccessed by the file cache - which is filled with port's data to be written - will also be blocked until the draining completes.

Once a port is large enough (libreoffice, some elements of KDE, ...) not to fully install into the buffers without being forced to drain them while doing so, the installation process is slowed by an order of magnitude. Additionally to the limit imposed by the amount of memory in the system that can be used for buffers, FreeBSD will keep "dirty" buffers (buffers that need to be written to the storage) for no more than 30 seconds before commiting them to the storage media even if there's enough buffers free, so a system with slower instructions and/or instruction scheduling, worse branch prediction, less logical processors, worse memory caches, slower memory, less memory bandwith, ..., or in general, a dated system, is more likely to run into this 30 seconds timeout than a current one.


----------



## kisscool-fr (Nov 8, 2013)

markbsd said:
			
		

> I do:
> 
> 
> ```
> ...





			
				wblock@ said:
			
		

> Yes, deleting build dependencies is a mistake.  But that won't have an effect until the next time.





			
				wblock@ said:
			
		

> If you were to stop and then restart the build, all of the ports it had already installed would not need to be installed again.  But your settings to delete build dependencies will defeat that, forcing them to be rebuilt.  That includes big things that are slow to compile, like gcc-6.4 at present.  Deleting all ports magnifies that, discarding all the work that has been done.
> 
> If you stop throwing away all the work each time, it will eventually complete and then take much less time to update or add.



From my experience, I stopped using options for portmaster. Specifically the one that said delete build dependecies. How I understand it is that for every port you want to install, portmaster will do the job for the build dependecies, then the port itself and at the end of the port installation it will delete build depencies. And that for all the ports. Imagine if for example all your ports need the same build dependecies, it will rebuild them for each port and this time consuming. You see ?

You can keep other options if you want, but the "delete build dependencies", I recommend to comment it.


----------



## kpa (Nov 8, 2013)

There's a problem with portmaster when used with PKGNG and you'd like to use packages for build only dependencies to save time, it's not possible to do so at the moment. This means portmaster will build all build time dependencies as ports if they are not installed already. This can be very time consuming.


----------



## jb_fvwm2 (Nov 8, 2013)

Missing parameter (not yet exists)... "build from port but use older packages for build time dependencies... if they exist." Perhaps someone can start a [Sticky] `portmaster` thread with 'wanted upgrades' 'optimal switch configurations' 'FAQ use cases' to lessen the explanations scattered throughout multiple threads, quite often duplicating content, as least it seems.

There does not seem to be a wiki page for it; it seems to me that the information would be more useful here as a sticky.


----------



## markbsd (Nov 9, 2013)

Thanks, everyone.

I've just decided to avoid ports on this system and stick to `pkg install`s. If I build up the courage to install FreeBSD on one of my newer computers I might return to ports, but the reward doesn't justify the cost (time) invested in this instance. In fact, I've been reading up a lot on ports vis a vis `pkg install`s and it seems ports don't really offer much more of a benefit at all. So, I may just remain with `pkg install`s notwithstanding what architecture I'm running.


----------



## Juanitou (Nov 9, 2013)

A real-life example (far more serious then that any professional story) of using ports to obtain a great reward: My old box has spent ~20 hours updating the base system to FreeBSD 10.0-BETA3 and compiling the experimental ports providing KMS for its ATI Radeon video card, which is quite slow or totally unusable with the current ports. It was my wifeâ€™s birthday and I wanted to offer her a better experience with this computer she mainly uses to watch internet streams. Thanks to the dedication in compiling these ports, Firefox navigation and video performance have significantly improved and we could also have some fun with SDL games that were unplayable before the update.

A happy wife for her birthday, what better reward could I ask for?


----------



## markbsd (Nov 9, 2013)

So, ports provided a significantly faster operation of _x,y,z_ programs than what `pkg install` provided?


----------



## Beastie (Nov 9, 2013)

wblock@ said:
			
		

> The short version is `pkg_delete -a`.


Yes, but it often leaves traces everywhere in /usr/local and is slower since it has to navigate through the entire dependency>root package hierarchy instead of just deleting files indiscriminately.



--8<--





			
				markbsd said:
			
		

> I've been reading up a lot on ports vis a vis pkg installs and it seems ports don't really offer much more of a benefit at all.


The only advantage that ports have over pre-built packages is that you can use non-default build options, including 1) the removal of some parts/dependencies you don't need and that make the application bigger (and sometimes slower) and 2) the inclusion of experimental features you may need.

If you don't really care about either, then definitely stick with packages. At least for the time being.

In the future you may want to build ports on your newer machines, create packages from these and install them on the older machines.


----------



## Juanitou (Nov 9, 2013)

markbsd said:
			
		

> So, ports provided a significantly faster operation of _x,y,z_ programs than what `pkg install` provided?



Thatâ€™s it, since no packages are provided for these recently ported versions of libGL, dri, libdrmâ€¦ For SDL games, it was not faster operation what they provided, but plain usability.

Anyway, my message was rather tongue-in-cheek, donâ€™t take too seriously such a particular example. On the other hand, only recently the packages infrastructure has been reinstated after the famous security incident. People relying only on packages have had to find alternatives for some months.


----------



## markbsd (Nov 9, 2013)

I take your point. I haven't noticed any performance decline now that I'm only using packages, but my experience is very limited. I'm not familiar with the security incident; FreeBSD, to me, is synonymous for secure!


----------

