# portsnap fetch error



## nedry (Apr 13, 2021)

Hi I tried doing a `portsnap fetch` and I get the following error:

```
root@bsdcompile:~ # portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Mar 31 14:10:16 BST 2021 to Tue Apr 13 15:20:42 BST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open bc296f38560f99bc3850120ee604754fe95618059f581c280573c208e7c28a89.gz: No such file or directory
metadata is corrupt.
```
How do I clear the metadata so I can update?
Thanks, Nedry


----------



## SirDice (Apr 13, 2021)

nedry said:


> How do I clear the metadata so I can update?


It's stored in /var/db/portsnap. You can remove the entire contents of that directory (don't delete the directory itself) and it'll fetch everything fresh. Might take a while though, that directory is also the cache where those *.gz files are stored.


----------



## nedry (Apr 13, 2021)

Hi I deleted all data in /var/db/portsnap but i ge the same error:

```
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open bc296f38560f99bc3850120ee604754fe95618059f581c280573c208e7c28a89.gz: No such file or directory
metadata is corrupt.
```


----------



## Deleted member 30996 (Apr 13, 2021)

```
root@bakemono:/ # portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Mar 31 10:23:08 CDT 2021 to Tue Apr 13 09:20:42 CDT 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... gunzip: (stdin): unexpected end of file
metadata is corrupt.
root@bakemono:/ #
```

When it's fixed I'll just do `portsnap fetch extract` and overwrite the corrupt metadata.


----------



## SirDice (Apr 13, 2021)

Yeah, seems there's something going wrong somewhere. It's dead slow and I keep getting corrupted files too.

```
root@fbsd-test:~ # portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching public key from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Tue Apr 13 02:29:08 CEST 2021:
338d0da0ff87d35854564889da42ad9556937e0d3b8683          59 MB   40 kBps 25m23s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Tue Apr 13 02:29:08 CEST 2021 to Tue Apr 13 16:52:34 CEST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open 963f0abcacba24a39f612c9ac012e2e91b1b7937f51e5c221b0512d0df751328.gz: No such file or directory
metadata is corrupt.
```


----------



## olli@ (Apr 13, 2021)

Currently work is ongoing to move the portsnap servers from svn to git. It seems that the road is a little bumpy. It will be fixed shortly. Just a little patience.


----------



## Deleted member 30996 (Apr 14, 2021)

It seems to be fixed now and everything is back up to speed.

I ran `portsnap fetch update` and got a full update without running `portsnap fetch extract` to overwrite the corrupt metadata from earlier.

`pkg audit -F` showed ftp/curl, www/node and x11-servers/xorg-server vulnerable.

x11-servers/xorg-server and www/node both have an update. ftp/curl is waiting for an updated version.

However, fixing the two vulnerabilities breaks www/firefox-esr and it comes up missing linxul.so. Which means it needs rebuilt in my experience and what I'm doing now on that machine and will be doing for some time with the others. Just so you know ahead of time.

I wouldn't have it any other way and love using ports.


----------



## athan (Apr 14, 2021)

Doesn't look fixed yet.

Portsnap fetch used to finish in seconds but now takes an hour (!!) on a 1gb line and ends with metadata corrupt error message.

IMHO they should not have released 13.0 before such problems were solved.


----------



## Deleted member 30996 (Apr 14, 2021)

athan said:


> Doesn't look fixed yet.
> 
> Portsnap fetch used to finish in seconds but now takes an hour (!!) on a 1gb line and ends with metadata corrupt error message.


Once I got the initial update about one hour ago, which was about 23,000 files IIRC, that took its time finishing but everything was back to normal and back up to speed next time I ran `portsnap fetch update`. That only took a few seconds as is usual and there was only one file updated.

I got a full update to the ports tree as described above and am watching the machine to my right rebuilding www/firefox-esr from the terminal as I type on this one sitting beside it.


----------



## richardtoohey2 (Apr 14, 2021)

portsnap is on the way to bit heaven (sadly, it's worked well for me) and this is all to do with the migration to git for ports.  Nothing to do with the release of any version.

You will have to move to something like gitup one day, why not today?

I'm not enthusiatic about having to change but it seems to be working (apart from constrained memory systems).


----------



## Deleted member 30996 (Apr 14, 2021)

richardtoohey2 said:


> portsnap is on the way to bit heaven (sadly, it's worked well for me) and this is all to do with the migration to git for ports.  Nothing to do with the release of any version.
> 
> You will have to move to something like gitup one day, why not today?


I've got 7 more laptops running FreeBSD to update like the one I'm doing now before I do anything. 

I may make the change on one of my machines after that to see how difficult it will be to adapt my Beginners Tutorial, but at this point plan on continue using it like I have been till no longer able.


----------



## athan (Apr 14, 2021)

If they wanted to migrate to git they first had to include some basic git-client in the base before forcing users to switch.

Once they've decided to keep portsnap in R13 it had to work as well as before.

Sorry but that's my humble opinion.


----------



## Deleted member 30996 (Apr 14, 2021)

I got a corrupt metadata message the first time I ran `portsnap fetch update` on the next machine.

I ran the command again and it started the correct process with 29,213 patches to be downloaded to update the ports tree.

Then all is back to normal operating procedure.

When they do a RELEASE that includes the port system update in the base system install I'll change my Tutorial. Right now it isn't broke and I don't fix things till they are.


----------



## drhowarddrfine (Apr 14, 2021)

As said, portsnap had problems at first but it's working fine for all my machines now. 

athan portsnap works as well as before. There were initial problems until eight hours ago or so but not now. In addition, there were notices and instructions how to use git for several months posted and that worked very well, too. It was easy.


----------



## SirDice (Apr 14, 2021)

Maybe it sometimes works, maybe it sometimes doesn't. No word yet of the actual completion. So assume it's not done yet.



			Portsnap restoration after Git migration


----------



## Deleted member 30996 (Apr 14, 2021)

The one machine finished its build of www/firefox-esr, then on hitting the desktop multimedia/audacious had suffered a dependency debacle during the build of Firefox so it had to be rebuilt as well.

Right now that's all done and I'm back on it waiting for the port tree update of ftp/curl. All is going as normal with the other slower machine at work.


----------



## Owosi (Apr 14, 2021)

portsnap fetch extract update
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 02:09:41 CEST 2021 to Wed Apr 14 12:49:06 CEST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... gunzip: (stdin): unexpected end of file


----------



## Owosi (Apr 14, 2021)

Now it's working fine


----------



## SirDice (Apr 14, 2021)

Read post #15.


----------



## macosxgeek (Apr 14, 2021)

```
[/var/db/portsnap]$ portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Mar 31 08:36:39 IST 2021 to Wed Apr 14 15:00:07 IST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open b1b1cb6ba5967e44b1ece8d63f7cb3d01b2c03fcf6691129dcb1cd5b8fe6024f.gz: No such file or directory
metadata is corrupt.
```




I guess it's not the best moment to update the system and upgrade to 13.0  Might wait a while longer.
BTW: Someone did mention that portsnap is going away (sadly) and suggested to use gitup.. Any guidance on how to use the new (Git based) update tool?


----------



## SirDice (Apr 14, 2021)

macosxgeek said:


> Someone did mention that portsnap is going away (sadly)


Not anytime soon. Probably when 14.0 is released, so it's still going be here for a while.


----------



## nedry (Apr 14, 2021)

I tried portsnap fetchtoday and I just got the following crash:




Not good. I really need to use the ports tree.


----------



## SirDice (Apr 14, 2021)

nedry said:


> Not good. I really need to use the ports tree.


portsnap(8) is still being worked on. If you really must have a ports tree NOW then I suggest you use git to checkout a fresh copy. 

But that error doesn't have anything to do with portsnap(8) though. You might be having disk issues too.


----------



## T-Daemon (Apr 14, 2021)

nedry said:


> Not good. I really need to use the ports tree.



Leaving out the possible disk issues SirDice mentioned, if you don't want to use devel/git or one of its flavors, as well not net/gitup, download  freebsd-ports-main.zip snapshot file from https://github.com/freebsd/freebsd-ports (green "Code" button, ~ 100MB).

GitHub freebsd-ports mirror is equal up to date as the official git repository (a few minutes apart probably). The unziped ports directory needs to be renamed before used and is IIRC portsnap(8) compatible.


----------



## richardtoohey2 (Apr 14, 2021)

As SirDice says - mpt is your disk driver - you've got bigger problems than portsnap.






						mpt(4)
					






					www.freebsd.org


----------



## Deleted member 30996 (Apr 14, 2021)

It's working fine for me. I've got another machine working to rebuild www/firefox-esr and this one waiting for the ftp/curl vulnerability update to hit ports.


```
root@bakemono:/ # portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 04:45:16 CDT 2021 to Wed Apr 14 15:54:39 CDT 2021.
Fetching 5 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
Fetching 170 patches. 
(170/170) 100.00%  done.                                     
done.
Applying patches... 
done.
Fetching 2 new ports or files... done.
Removing old files and directories... done.
Extracting new files:
*snip*
Building new INDEX files... done.
root@bakemono:/ #
```


----------



## diego (Apr 15, 2021)

nedry said:


> Not good. I really need to use the ports tree.


While portsnap had problems I will remove "/usr/ports" and do a fresh installation via git (Port Collection)


```
# rm -rf /usr/ports/*
# git clone https://git.FreeBSD.org/ports.git /usr/ports
```
for updating port collection ....

```
# git -C /usr/ports pull
```

Waiting portsnap will be fixed soon


----------



## a6h (Apr 15, 2021)

nedry said:


> Fetching 5 metadata files... /usr/sbin/portsnap: cannot open bc296f38560f99bc3850120ee604754fe95618059f581c280573c208e7c28a89.gz: No such file or directory



"cannot open" -- mainly in Europe, was due to some load issue.

The reason:
Mirrors have expanded from one up to ten instances of m6g.medium, and mirrors are pulling from same Amazon EFS.

Why 10x mirrors:
Commits are removing $FreeBSD$ tags from ports.

Source:
Colin Percival (Security Officer Emeritus, FreeBSD/EC2 maintainer)
https://twitter.com/cperciva/status/1382491128537096193


----------



## macosxgeek (Apr 15, 2021)

portsnap did work fine for me this morning.

(my server is based in Germany)


----------



## athan (Apr 15, 2021)

It works for me too now, both in lab (Greece) and on my remote servers (Germany and UK)


----------



## athan (Apr 16, 2021)

As I said, portsnap seems to be working now though not in sync with cgit.freebsd.org/ports.
Last time portsnap local tree got an update was 26 hours ago although many ports have been updated since then.


----------



## diego (Apr 16, 2021)

FYI.
portsnap is working fine again  -->  server based in UK


----------



## ZFS_HBG (Apr 16, 2021)

athan said:


> As I said, portsnap seems to be working now though not in sync with cgit.freebsd.org/ports.
> Last time portsnap local tree got an update was 26 hours ago although many ports have been updated since then.


That is still happening:


```
portsnap auto
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Latest snapshot on server matches what we already have.
No updates needed.
Ports tree is already up to date.
# date
Fri Apr 16 20:21:28 CEST 2021
```

So the port tree is not yet stable it seams.


----------



## tingo (Apr 17, 2021)

Note that portsnap builds snapshots from the ports tree, at unspecified intervals (at least I could not find any info about it). Implied (by me): portsnap snapshots have a lower update rate than the commit rate for the ports repository.


----------

