# [ports][GIT][portsnap] Warning about INDEX



## zirias@ (Apr 6, 2021)

In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.

Right now, I learned the following in an IRC channel full of devs:

```
< XXXXXX> INDEX is currently dangerously misleading, that needs to be fixed first
< XXXXXX> I looked into fixing it but given the dependencies between INDEX and portsnap it made sense to wait for portsnap to die first
< Zirias> I never use it anyways. But some workflows need it…
< Zirias> XXXXXX: what exactly is misleading? does it affect a locally created one?
< XXXXXX> currently the dependency info in it can be outright wrong due to failure to interpret flavors
< Zirias> uh!
< XXXXXX> it assumes it can take a port origin without flavor (i.e. cat/port rather than cat/port@flavor) and translate that to a package name
< XXXXXX> which is wrong
< Zirias> so it's essentially broken since flavors are supported? :o
< XXXXXX> yup
```
(anonymized the handle here and removed timestamps)

I just think anyone should be aware and think twice about using something that relies on INDEX.


----------



## trev (Apr 6, 2021)

I can't say that I've ever had a problem with INDEX when using portsnap nor now when using gitup. My flavoured ports continue to be updated using portupgrade.

Is there any actual manifestation of the issue?


----------



## PMc (Apr 7, 2021)

The INDEX was always broken, since ports have options. It should list the dependencies for each port - but as soon as you change an option for some port, the dependencies may change, and then the INDEX (specifically the one that is downloaded) is no longer accurate.

I know only portupgrade that would use the INDEX: And an inaccurate INDEX leads to some prereqs built by portupgrade, others (that are missing in the INDEX) built directly by make. Those built by portupgrade will respect the variables configured in /usr/local/etc/pkgtools.conf, those built by make will not know them.

Now with flavors the chaos might be a little more imminent - and that one might even be fixable, while the essential problem that the INDEX is never really accurate as soon as options bring in other dependencies - that is not fixable.


----------



## bagas (Apr 7, 2021)

How then to be?
If I need to update the software on the server.
Earlier I just looked at pkg version | grep '<' .
If everything is ok then portupgrade -arR .


----------



## zirias@ (Apr 7, 2021)

PMc said:


> The INDEX was always broken, since ports have options. It should list the dependencies for each port - but as soon as you change an option for some port, the dependencies may change, and then the INDEX (specifically the one that is downloaded) is no longer accurate.


A locally created INDEX doesn't have _this_ problem unless you change options after creating it, but



PMc said:


> Now with flavors the chaos might be a little more imminent


From what I was told (see the IRC log snippet), it's not a little more imminent but outright broken. If flavors are involved, creation of the index doesn't set them when looking for dependencies, so you will get wrong dependencies (probably based on the default flavor, even if a different one is used).


----------



## Argentum (Apr 7, 2021)

Zirias said:


> In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.


Noticed 


```
[Reading data from pkg(8) ... - 1484 packages found - done]
Fetching the ports index ... /usr/bin/env  fetch -am -o /usr/ports/INDEX-13.bz2 https://www.FreeBSD.org/ports/INDEX-13.bz2
/usr/ports/INDEX-13.bz2                        77% of 2271 kB 1815 kBps    01s
fetch: /usr/ports/INDEX-13.bz2 appears to be truncated: 1802911/2326023 bytes
*** Error code 1

Stop.
make: stopped in /usr/ports
failed to fetch INDEX!
```

*Building index locally.*


----------



## mickey (Apr 7, 2021)

Is there any compelling reason not to go completely without the INDEX file then? What tools do actually use it? So far I can tell that `pkg version -I` uses it, but is also able to use the ports tree or the repository catalog instead. Also ports-mgmt/portmaster has options to make use of the index file at some point, but this doesn't seem mandatory either.

In the past I have used `make fetchindex` without thinking too much about where that index is actually used. But in light of the facts that the index seems flawed in several ways it's probably time to question it's usefulness.


Argentum said:


> Fetching the ports index ... /usr/bin/env fetch -am -o /usr/ports/INDEX-13.bz2 https://www.FreeBSD.org/ports/INDEX-13.bz2 /usr/ports/INDEX-13.bz2 77% of 2271 kB 1815 kBps 01s fetch: /usr/ports/INDEX-13.bz2 appears to be truncated: 1802911/2326023 bytes *** Error code 1


This is also one problem I've seen occuring one too many times when fetching the index. I think I saw some discussion about it and that it's probably related to some sort of HTTP accelerator in use on the website, but can't say for sure. Fact is, this has been happening for years.


----------



## zirias@ (Apr 7, 2021)

mickey said:


> Is there any compelling reason not to go completely without the INDEX file then?


IMHO, no. I never used it, I build my local pkg repository with ports-mgmt/poudriere, and `pkg` gets dependency information from the repository. I don't think INDEX is all too useful for other workflows either. It definitely isn't unless the flavors problem is fixed, and then, you have to create it locally if you changed any port options, default versions etc and you have to re-create it on any change or ports update. I'll keep going without it


----------



## mickey (Apr 7, 2021)

Zirias said:


> It definitely isn't unless the flavors problem is fixed, and then, you have to create it locally if you changed any port options, default versions etc and you have to re-create it on any change or ports update. I'll keep going without it


Eh no, no, nonono. So not doing this on a daily basis for each machine (as they have different port options), which also wouldn't work cause the ports tree is a read-only NFSv4 mount. INDEX no more


----------



## PMc (Apr 7, 2021)

Zirias said:


> A locally created INDEX doesn't have _this_ problem unless you change options after creating it,


That's correct - but the offering to change the options usually appears when make starts to build some port. You could run `make config-recursive` beforehand to get all the options together first.

Later I found out that even when all dependencies are properly found and configured, the ports themselves may look for already installed software and then configure themselves differently on what is already present, consequentially again changing their dependencies.
So I stopped to use portupgrade for creating the depencency-lists, created my own dependency list, and built the ports in a limited `chroot` where only the identified prereqs are installed.

Then I figured that under these circumstances there is no need to use portupgrade at all.

But there are still ports doing really crappy things. For instance, this construct was in editors/emacs not so long ago:


```
# has graphics/ImageMagick been compiled with OPENMP?
.if ${PORT_OPTIONS:MMAGICK} && ${:!${GREP} -sc " \-fopenmp " ${LOCALBASE}/libdata/pkgconfig/ImageMagick.pc || true!} == "1"
USES+=        compiler:openmp
.endif
```
This is the makefile that wants to build emacs. Now what does this do: it searches for the graphics/ImageMagick port (which is supposed to be already installed), reads the configuration options of that libary from the /usr/local/libdata/pkgconfig/ImageMagick.pc file - and then decides on which compiler to require as a prereq for building emacs!

You cannot capture that in an INDEX file - and it is bogus in any way, because: only after make reads this makefile, it will decide to build+install ImageMagick as a prereq.

What we would need, is. to have all the ports' options kept in some kind of database where other ports can do a lookup. (It does already exist under /var/db/ports, it is just not suitable for arbitrary querying.)



mickey said:


> Is there any compelling reason not to go completely without the INDEX file then?


No. Lately I forgot to configure a proper path where to write it, so it isn't written anymore. But since I already ceased to use portupgrade, there is nothing present that would require the INDEX.


----------



## bagas (Apr 7, 2021)

PMc said:


> But since I already ceased to use portupgrade, there is nothing present that would require the INDEX.


How do you update the software on the server?
portupgrade is convenient, what's the best way to update the software?


----------



## zirias@ (Apr 7, 2021)

bagas said:


> what's the best way to update the software?


This depends a lot on your actual scenario AND your personal preferences.

I like my current setup with only one machine building software (using ports-mgmt/poudriere) and all other machines installing with `pkg` from that repository, exposed by www/nginx.


----------



## SirDice (Apr 7, 2021)

mickey said:


> Is there any compelling reason not to go completely without the INDEX file then? What tools do actually use it?


If I remember correctly port-mgmt/portupgrade but that port has quite a number of other issues too, I stopped using it a really long time ago. When ports-mgmt/portmaster was available I switched over. Then ports-mgmt/poudriere came around and used that. Never looked back since. 



Zirias said:


> I like my current setup with only one machine building software (using ports-mgmt/poudriere) and all other machines installing with `pkg` from that repository, exposed by www/nginx.


Same here. It's so convenient, having a single server set up to build a bunch of different repositories for different versions. Then simply use pkg(8) on all my other machines and updates are a breeze.


----------



## mickey (Apr 7, 2021)

PMc said:


> No. Lately I forgot to configure a proper path where to write it, so it isn't written anymore. But since I already ceased to use portupgrade, there is nothing present that would require the INDEX.


portupgrade ... wait what? ... already? Better late than never I guess 

My index is gone, portmaster masters updating without it (no sorcery involved) and pkg version automatically switches to using the ports tree if no index was found, but I usually dont bother using it, as it just takes additional time without adding anything useful:


----------



## PMc (Apr 7, 2021)

bagas said:


> How do you update the software on the server?
> portupgrade is convenient, what's the best way to update the software?


There is no "best" way - it depends on your site layout and requirements/preferences. portupgrade is convenient if you just have a few machines, build (some) ports locally and need to keep track (or at least _some_ track) on the dependencies.
If you need anything more big or more structured/reliable, then I would recommend: build all ports on one machine, bring them to the target machines (by whatever means, NFS, http, zfs-send, as appropriate), and create some custom config in /usr/local/etc/pkg/repos/ to install from your own repo.


----------



## bagas (Apr 8, 2021)

Removed old ports.
Got new ports with git.
I create an index file.



> cd /usr/ports/ && make index
> ...
> --- describe.x11-toolkits ---
> --- describe.x11-wm ---
> ...





> Update soft.
> portupgrade -arR
> [Reading data from pkg(8) ... - 398 packages found - done]
> [Updating the portsdb <format:bdb_btree> in /usr/ports ... - 0 port entries found  ..... done]
> ...


----------



## bagas (Apr 8, 2021)

downloaded the base index.
make fetchindex
pkg version | grep "couchdb2"
I do not have such an installed couchdb2 port.


----------



## richardtoohey2 (Apr 8, 2021)

You might have hit this: https://marc.info/?l=freebsd-ports&m=161785223918772&w=2


----------



## bagas (Apr 8, 2021)

if I delete /usr/ports/databases/p5-AnyEvent-CouchDB now, then the next time the ports are updated, it will appear again.


----------



## jmos (Apr 8, 2021)

mickey said:


> So far I can tell that `pkg version -I` uses it



You mustn't use the "-I" option: The INDEX file is used (if present) by default! Anyone uses portsnap has/had the INDEX, and anyone switched to gitup lost it. So expect many people noticing that a `pkg version -vl '<'` (looking for non up to date packages) slowed extremely down.


----------



## mickey (Apr 8, 2021)

jmos said:


> You mustn't use the "-I" option: The INDEX file is used (if present) by default!


Personally I never use `pkg version` at all cause I don't want to know what needs updating, I want to get the update done. Therefore I just run portmaster and if there is nothing to update it probably took about the same time as running pkg version.


----------



## chrcol (Apr 8, 2021)

why the need to anonymize names and timestamps, even reveal the channel/server.


----------



## bagas (Apr 8, 2021)

jmos said:


> You mustn't use the "-I" option: The INDEX file is used (if present) by default! Anyone uses portsnap has/had the INDEX, and anyone switched to gitup lost it. So expect many people noticing that a `pkg version -vl '<'` (looking for non up to date packages) slowed extremely down.


What to do then?
Before updating the software on the server, I want to know what will be updated.


----------



## jmos (Apr 8, 2021)

bagas said:


> What to do then?


Your system works as before, it just takes more time.


----------



## bagas (Apr 8, 2021)

Had to drop the local index(make index) and download the base index (make fetchindex).
The oddities of switching to git!


----------



## bagas (Apr 8, 2021)

jmos said:


> Your system works as before, it just takes more time.


did not understand, more time for what?


----------



## jmos (Apr 8, 2021)

bagas said:


> did not understand, more time for what?


For anything the INDEX file was asked before (like the mentioned `pkg version -vl '<'`). As you noticed you can create it after every GIT update by yourself (`make index` - takes its time, too), or download it (`make fetchindex` - but assume that to be faulty when using git/gitup, as it wasn't build upon the files you've fetched, so that isn't really an option).


----------



## bagas (Apr 8, 2021)

jmos said:


> For anything the INDEX file was asked before (like the mentioned `pkg version -vl '<'`). As you noticed you can create it after every GIT update by yourself (`make index` - takes its time, too), or download it (`make fetchindex` - but assume that to be faulty when using git/gitup, as it wasn't build upon the files you've fetched, so that isn't really an option).


I mean too!
I cannot create a local index, I have to download the base one.


> cd /usr/ports/ && make index
> ...
> --- describe.x11-toolkits ---
> --- describe.x11-wm ---
> ...


----------



## zirias@ (Apr 8, 2021)

jmos said:


> For anything the INDEX file was asked before (like the mentioned `pkg version -vl '<'`). As you noticed you can create it after every GIT update by yourself (`make index` - takes its time, too), or download it (`make fetchindex` - but assume that to be faulty when using git/gitup, as it wasn't build upon the files you've fetched, so that isn't really an option).


Please read my initial post. INDEX will be wrong in any case. It will possibly be "more wrong" when downloaded instead of locally created. But in a nutshell, you just shouldn't rely on it at all.


----------



## jmos (Apr 8, 2021)

Zirias said:


> Please read my initial post. INDEX will be wrong in any case. It will possibly be "more wrong" when downloaded instead of locally created. But in a nutshell, you just shouldn't rely on it at all.


Using the selfmade INDEX just for comparing version numbers of the ports tree and the installed packages is save, flavors do not affect them  But I agree, it is much better to go on without it 

The missing of databases/couchdb2 is a bug (should be removed 2021-06-23 - but it's gone already); Either it should be added again, or databases/p5-AnyEvent-CouchDB has to be fixed not to use it.


----------



## zirias@ (Apr 8, 2021)

jmos said:


> Using the selfmade INDEX just for comparing version numbers of the ports tree and the installed packages is save, flavors do not affect them


That's, unfortunately, wrong. As `makeindex` doesn't consider the flavor when deriving the package name, it will go wrong as soon as you have a package from a non-default flavor.


----------



## olli@ (Apr 8, 2021)

FWIW, I have the following lines in /etc/periodic.conf on all of my FreeBSD machines:

```
weekly_status_pkg_enable="YES"
pkg_version_index="-P"
```
Actually `-P` is the default if there is no INDEX file, but it’s better to specify it explicitly, just in case an INDEX file happens to be downloaded or generated accidentally.
 
Also, using `-P` has the nice side effect that orphaned packages will be reported correctly (i.e. port origins that don’t exist anymore), which is not the case when an INDEX file is used.


----------



## chrcol (Apr 8, 2021)

I have a workaround I am using on portupgrade now, in the pkgtools.conf you can specify environment variables for specific ports, I added flavor=tiny for devel/git and it does work.  How long I will be doing this for I dont know though, seems inevitable I will have to switch to tools that dont use the INDEX file.


----------



## reddy (Apr 8, 2021)

Zirias said:


> In the context of ports moving to GIT, *as well as (expected) deprecation of portsnap*, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.



Can someone tell me more about this? Why will portsnap be deprecated (and when is it expected to happen)?


----------



## scottro (Apr 9, 2021)

It will be quite awhile yet. However, I think many people feel, (including myself), that it's easier to start getting used to using git now. 
In another thread, I've discussed INDEX for my use. My only use of it is to sometimes find ports, using a package called psearch. For that purpose, using make fetchindex works, at least at present. I don't do much port searching, but it does happen. When I tried doing make index, however, psearch (or running make search in the ports directory), didn't work. 

So, I think that in part, it depends upon what use you make of index. For *my very* limited use, I can use git, and then run make fetchindex. If you use it for other tasks, it may cause problems. 
(I would also wait for a better answer than mine, from people who know more, such as Zirias).


----------



## ShelLuser (Apr 9, 2021)

Well, the whole INDEX procedure is definitely unreliable. When I try to build the index it bugs out after detecting a circular dependency:


```
--- describe.x11-servers ---
--- describe.x11-themes ---
--- describe.x11-toolkits ---
--- describe.x11-wm ---
make_index: Circular dependency loop found: php73-pear-Net_SMTP-1.9.0 depends upon itself.

 Done.
```

But meanwhile:

```
peter@vps:/home/peter $ ls -l /usr/ports/INDEX-12
-rw-r--r--  1 root  wheel  0 Apr  9 03:39 /usr/ports/INDEX-12
```
Very convincing   

I think I'm going to forget about building INDEX files entirely because building them takes a lot more time than the additional time for `pkg version` to run.


----------



## grahamperrin@ (Apr 9, 2021)

reddy said:


> … Why will portsnap be deprecated (and when is it expected to happen)? …



Presence and planned/prospective removal of portsnap ▶ https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504229 (can't say _when_ removal will occur, however removal _was_ mentioned in the context of deprecation).

Use of portsnap ▶ https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504232 (links to manual pages).


----------



## grahamperrin@ (Apr 9, 2021)

scottro said:


> … My only use of it is to sometimes find ports, using a package called psearch. …



ports-mgmt/psearch – interesting, thanks.
scottro you've seen this already, for the benefit of other readers:



olli@ said:


> I don’t know exactly what features psearch provides, but maybe Porgle is useful for you. It’s a web-based search engine for the FreeBSD ports collection. It has low browser requirements, it also works fine with text-based browsers (lynx, w3m, links).



Entirely different, but surprisingly useful (and people are sometimes unaware of its existence): ports-mgmt/pkg-provides

If you'd like more from FreshPorts, consider raising an issue (or pull request); Dan Langille is very pleasantly responsive.


----------



## olli@ (Apr 9, 2021)

reddy said:


> Why will portsnap be deprecated (and when is it expected to happen)?


So far I haven’t seen an _official_ statement regarding the deprecation of portsnap, so this is speculation.
However, portsnap is still present in stable/13 and in 13.0, so my assumption is that it will be supported for the lifetime of that stable branch, which is *five years from now (i.e. 2026)*.
FWIW, portsnap hasn’t been removed from -current (a.k.a. main) either, although this could happen at any time, in theory, since -current doesn’t give any guarantees whatsoever.


----------



## zirias@ (Apr 9, 2021)

olli@ said:


> So far I haven’t seen an _official_ statement regarding the deprecation of portsnap, so this is speculation.


The _plan_ to do so is officially announced (portmgr@ hat):




__





						[HEADS UP] Planned deprecation of portsnap
					





					lists.freebsd.org
				




So, yes, _when_ exactly this will happen is speculation, but it's unlikely to see a 180° turn here.


----------



## olli@ (Apr 9, 2021)

Zirias said:


> The _plan_ to do so is officially announced (portmgr@ hat):
> 
> 
> 
> ...


Yes, of course, I don’t have doubt about the deprecation itself. But since there is no deadline yet, it doesn’t appear that immediate action is required now, or in the near future.

Apart from that, I wouldn’t call a mail to the -ports list an “official announcement”. Typically, portsnap users are not on that list.
Such an important announcement _must_ be posted to the -announce mailing list, _and_ to a news entry on the web page, _and_ to an entry in /usr/ports/UPDATING. Preferably after _all_ mentions of portsnap have been removed from the documentation (especially the Handbook), being replaced with instructions suitable for non-developers how to use git or gitup to do what portsnap did. Also don’t forget the ports(7) manual page.
 
If that has been done, I would call it “official”.


----------

