# Parallel builds+downloads, 'make build-recursive'



## vanessa (Oct 5, 2013)

There are multiple threads in the forum discussing how to speed up building ports. All target the -j option or the MAKE_JOBS_SAFE variable. But half of the time goes away downloading dist files. 

So *why* can't these two processes run in parallel - the next distfile being downloaded while building the current one? Or am I just missing an existent nifty switch for enabling this behaviour?


----------



## Juanitou (Oct 5, 2013)

ports-mgmt/portmaster do that. From its man page:


> While recursing through dependencies, if you are not using any of the --packages* options, a 'make checksum' process will be launched in the background to either verify that the correct distfiles are available or start downloading the new ones.


----------



## vanessa (Oct 5, 2013)

Good tip - I didn't know that! I am still building ports manually because I (think to) have more control of the steps involved. 

However I severely miss one other function besides parallel downloads: _'make build-recursive'_. As ports often fail to compile halfway down the chain, I end up with some of the dependancies already installed while others are still not compiled. It would be fine to build all of the dependencies first but install them altogether in a second step and in one go.

Does anybody know of a workflow how to achieve this?


----------



## ShelLuser (Oct 5, 2013)

vanessa said:
			
		

> However I severely miss one other function besides parallel downloads: _'make build-recursive'_. As ports often fail to compile halfway down the chain, I end up with some of the dependancies already installed while others are still not compiled.


The problem here is basically the structure of the ports; in the end they are all individual components which may have a dependency but that dependency only goes as far as to check if a specific version is installed or not.

More than often a dependency is either set as "_if product x is present_" or "_if version y or higher of product x is present_". The system doesn't really care much more beyond that.

And that's what you're experiencing here. The best way to solve some of these problems is to write a script which can handle some of the hassle and keep track of your ports and their updates. Well, it has been mentioned before but that's where ports-mgmt/portmaster comes in. It's basically a shell script which checks for all this.

I think it's a very good idea to start off manually so that you really get a good feeling for the way things work. Been there myself earlier this year. However, while you do maintain more control over your ports you'll also notice that you may end up with too much control since the ports collection hasn't automated much itself.

What you basically have are several small features which, when combined, can lead up to a very powerful setup. The only thing you'd need to do is piece the whole thing together.

Which could make a very good experience, but you'll notice that most people don't bother anymore these days and simply resort to portmaster (myself included).

Hope this can give some ideas.


----------



## vanessa (Oct 5, 2013)

Well, let me explain what my point is. Since Thursday (!) I am trying to build KDE on FreeBSD 10 HEAD. As it doesn't work out of the box (meaning with the predefined options for the ports involved), I am constantly either fixing something or dropping things I don't really need. The problem is, that every time a compilation fails, I end up with many packages installed. This prevents going on by simply reconfiguring and building again, because:
1) building them only doesn't work, as dependencies get automatically installed (despite telling the meta port 'make build' only)
2) I have to first deinstall the packages, before building them again with different options (or not building them at all because of dropped options). Otherwise I could only force overwriting them by setting FORCE_PKG_REGISTER. Either way it's a mess and a huge time loosing factor.

Actually this is a bug, because 'make build' should *build* only and not install anything, even for dependencies or meta ports. Or there should be a 'make build-recursive' like I am proposing here.


----------



## ShelLuser (Oct 5, 2013)

It starts to make some sense now. First of all though, you might already be aware of this, but please keep in mind that FreeBSD-10 is basically marked unstable. It's the developers playground so to speak, so while it may work for you now it can just as easily break tomorrow due to applied changes.



			
				vanessa said:
			
		

> The problem is, that every time a compilation fails, I end up with many packages installed. This prevents going on by simply reconfiguring and building again, because:
> 1) building them only doesn't work, as dependencies get automatically installed (despite telling the meta port 'make build' only)


First off keep in mind that there are 2 types of dependencies for a port. You have the run dependency which is required for the port to function, but you also have the so called build dependencies. These are required to actually build or compile the port. Sometimes they overlap, but very often they don't.

So if you issue `# make build` then all the build dependencies will get automatically installed, simply because the port needs them at this stage. You can check these for yourself by using the `make build-depends-list` while `make run-depends-list` will show you the ports required for usage (see also ports(7)).



			
				vanessa said:
			
		

> 2) I have to first deinstall the packages, before building them again with different options (or not building them at all because of dropped options). Otherwise I could only force overwriting them by setting FORCE_PKG_REGISTER. Either way it's a mess and a huge time loosing factor.


I wonder if you would be able to use `# make deinstall-all` here. Not sure because the manual page is a bit unclear, but it seems to me as if it would uninstall the main port and all its dependencies.

So this really isn't a bug but simply another type of dependency.

But do keep in mind that using FreeBSD-10 is bound to give you problems and issues like these. The best way to overcome those is to stick with the production version which is FreeBSD-9.2 at the time of writing.

(*Edit*):

When going over the thread I suddenly realized that I forgot something I was planning to mention. Better late then never I suppose. Maybe you already know about this, in that case just ignore ]Unix(-like)[/noparse] commandline. So if you're working with a pristine setup (where root uses the csh shell) then you could use this when you're inside the ports directory you're working on:


```
foreach a (`make build-depends-list`)
  make -C $a deinstall clean;
  end;
```
After typing the first line you'll get into special command prompt where you can enter the rest of the commands.

This is also doable in more common shells such as sh or bash. Then you'd use something like this: `# for a in `make build-depends-list`; do make -C $a deinstall clean; done`.

Hope this can help too.


----------



## wblock@ (Oct 5, 2013)

The build step involves making sure necessary build dependencies are already installed.  So it will build and install those dependencies.  In other words, "build" applies to this port, not the dependencies.


----------



## vanessa (Oct 8, 2013)

ShelLuser said:
			
		

> First of all though, you might already be aware of this, but please keep in mind that FreeBSD-10 is basically marked unstable. It's the developers playground so to speak, so while it may work for you now it can just as easily break tomorrow due to applied changes.



Well, the 'playground' should/must be already quite stable - in two months it's showtime.


----------



## vanessa (Oct 8, 2013)

ShelLuser said:
			
		

> (*Edit*):
> 
> When going over the thread I suddenly realized that I forgot something I was planning to mention. Better late then never I suppose. Maybe you already know about this, in that case just ignore
> 
> ...



Thank you for the input, very kind of you  

The problem with deinstall is that you can't deinstall a package when third ports depend upon it, unless:

 you force this -> in this case you have to (remember to) re-install the port/s back again. If the next build fails or such ports fall out as dependencies, you end up with a missing bit. 
 or let it in and force the overwrite -> in this case it might be that there will be no overwrite (again, because of other dependencies), so you end up with a package nobody needs
In both cases you must manually fix the problem and delete old or unneeded stuff. But this is not that bad - the REAL problem is, that you have to track ALL those packages over several hours or even days in your head! So go compile KDE with some failing builds and your brain will go buum.


----------



## SirDice (Oct 8, 2013)

vanessa said:
			
		

> Well, the 'playground' should/must be already quite stable - in two months it's showtime.



Don't hold your breath in the mean time. Schedules have a tendency to slip.


----------



## ShelLuser (Oct 9, 2013)

vanessa said:
			
		

> The problem with deinstall is that you can't deinstall a package when third ports depend upon it, unless:
> 
> you force this -> in this case you have to (remember to) re-install the port/s back again. If the next build fails or such ports fall out as dependencies, you end up with a missing bit.
> or let it in and force the overwrite -> in this case it might be that there will be no overwrite (again, because of other dependencies), so you end up with a package nobody needs


Well spotted. Yes, what I wrote up above is basically exactly what point 1 does. If you try to uninstall a package using the package manager (either using `# pkg_delete` or `# pkg delete`) you'll need to apply some virtual force because they won't allow you to remove anything which has dependencies.

But if you run a `# make uninstall` those dependencies will only turn into a warning ("removing anyway").



			
				vanessa said:
			
		

> But this is not that bad - the REAL problem is, that you have to track ALL those packages over several hours or even days in your head! So go compile KDE with some failing builds and your brain will go buum.


Indeed.

Those one liners are only a liable option when you're going to pursue the build right then and there, otherwise you risk inconsistency. It's also why it's generally a good idea to eventually "give in" and start utilizing some programs which can do some of all this work for you. Well, either that or device a strategy of your own.

For example]portmaster[/file] has an option where it can keep track of the ports it just build in order to prevent you from having to re-build the same ports over and over again while you're actually focussing on an issue with only one of the whole lot.


----------



## vanessa (Oct 9, 2013)

In the mean time I found a killer solution: ZFS snapshots  Each failed build goes then straight back to hell ï¿½e


----------



## hedgehog (Oct 10, 2013)

vanessa said:
			
		

> In the mean time I found a killer solution: ZFS snapshots  Each failed build goes then straight back to hell ï¿½e



There is even better solution. You can build everything in clean jailed environment using ports-mgmt/poudriere and upgrading your host system using generated packages repository. The main advantage is that you can do anything while building, you can even stop the process and continue any time - your system will stay unaffected.

Once everything built, you just hit `# pkg upgrade` and here you go: your installed ports will get upgraded in a few minutes using your repository.

HOWTO: http://forums.freebsd.org/showthread.php?t=38859


----------



## vanessa (Oct 10, 2013)

How could I have missed poudriere!? Yes, this is the real alternative to ZFS, thanks!

OK, to sum up the 2 ways of clean building from ports:
1) ports-mgmt/poudriere 
2) ZFS snapshots

+1 for merging/moving poudriere into the default building process.


----------



## hedgehog (Oct 10, 2013)

vanessa said:
			
		

> OK, to sum up the 2 ways of clean building from ports:
> 1) ports-mgmt/poudriere
> 2) ZFS snapshots


Actually, you still can use shapshots with the first method. I always create shapshots for /usr/local and /var/db/pkg before doing `# pkg upgrade`. Having backup wont hurt, especially if it's created within seconds


----------

