# problem fetching ports tree



## fluca1978 (Jan 12, 2012)

I'm getting errors while trying to install the ports tree using `# portsnap`:


```
# portsnap fetch extract
portsnap: Directory does not exist or is not writable: /var/db/portsnap
bsdmag# rm /var/db/portsnap 
bsdmag# rm /usr/ports 
bsdmag# mkdir /var/db/portsnap
bsdmag# portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.
Fetching public key from portsnap5.FreeBSD.org... done.
Fetching snapshot tag from portsnap5.FreeBSD.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Thu Jan 12 00:06:43 UTC 2012:

67204aa8591a39a8121c6c2012129ee50551d7f6065175100% of   64 MB  168 kBps 00m00s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from portsnap5.FreeBSD.org... done.
Fetching snapshot metadata... done.
Updating from Thu Jan 12 00:06:43 UTC 2012 to Thu Jan 12 15:05:00 UTC 2012.
Fetching 4 metadata patches. done.
Applying metadata patches... done.
Fetching 4 metadata files... 

/usr/sbin/portsnap: cannot open a1d3ca465757c636dd8295698fe05e7d3fc4943204e504fee6cd5b1665ba0753.gz: 
No such file or directory
metadata is corrupt.
```

I've already tried to remove and recreate /var/db/portsnap but without any success. My system is:


```
# uname -a
FreeBSD bsdluk 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Fri Feb 18 02:24:46 UTC 2011     
root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
```

Any idea on what is going wrong here?


----------



## SNK (Jan 12, 2012)

Is the problem persistent?


----------



## fluca1978 (May 22, 2012)

I'm experiencing the same problem again. I've tried to do a portsnap(8) which failed due to metadata corrupt, I then recreated the /var/db/portsnap folder and I then tried also to do a new fetch (instead of an update). The result is always the same:


```
19714ed1c2980d1e1a2c44d292a7ccfd84f400a83211f9100% of   67 MB  177 kBps 00m00s
Extracting snapshot... done.
Verifying snapshot integrity... 
done.
Fetching snapshot tag from geodns-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Tue May 22 02:12:15 CEST 2012 to Tue May 22 10:45:23 CEST 2012.
Fetching 4 metadata patches. done.
Applying metadata patches... done.
Fetching 4 metadata files... /usr/sbin/portsnap: 
cannot open 1e7ee6890d8372822978351df474d31be0c7e77f30a5055ccad013360479f9fd.gz: 
No such file or directory
```

The only thing that is different in my environment is that I'm behind a proxy, that portsnap gets from the http_proxy environment variable.
I've tried today and yesterday.
Any idea?


----------



## fluca1978 (Nov 8, 2012)

I'm still fighting with this problem, and now, due to some changes of the proxy I'm behind, again I cannot download the ports update. This time the error is different and is related to phttpget:


```
# portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.                               
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.                    
Fetching snapshot metadata... done.                                                       
Updating from Mon Nov  5 00:03:52 UTC 2012 to Thu Nov  8 11:39:28 UTC 2012.               
Fetching 4 metadata patches. done.                                                        
Applying metadata patches... done.                                                        
[B]Fetching 4 metadata files... phttpget: host = fluca, port = pwd@192.168.1.7:8080: servname not supported for ai_socktype                [/B]                                     
Processing file ee2cf31cffdb8da7ecb6db2dd8d36ed4823fce7cffbb09b55c2754247961a812          
/usr/sbin/portsnap: cannot open ee2cf31cffdb8da7ecb6db2dd8d36ed4823fce7cffbb09b55c2754247961a812.gz: No such file or directory                                                      
metadata is corrupt.                                                                      
/usr/sbin/portsnap: cannot open ee2cf31cffdb8da7ecb6db2dd8d36ed4823fce7cffbb09b55c2754247961a812.gz: No such file or directory                                                      
SHA256 =  e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
```

The wrong line is the one where my proxy authentication information is used as a port (i.e., I've got an env variable _http_proxy_ set as _http://username:password@proxyIP:proxyPort_ - being those shown not my real credentials). So phttpget is not working properly with proxy authentication.
I found this old PR but it mentions also a proxy option for phttpget that seems to me it is no more present.

Anybody has a suggestion on how to solve the problem?


----------



## fluca1978 (Nov 26, 2012)

I've fixed it for my machines making a change to portsnap: now it uses net/wget to download the stuff instead of _phttpget_, and since _wget_ can handle the _http_proxy_ variable well, the process works. Of course, I had to change also the logic that built the files to download as in the following patch. I've named this portsnap_wget on my system, and maybe this can be useful for someone else.


```
268c268,270
< PHTTPGET=/usr/libexec/phttpget
---
> PHTTPGET=/usr/local/bin/wget
>
> echo "WARNING: this version of portsnap is usign $PHTTPGET as Web-HTTP-Get program!"
687,690c689,694
< tr '|' '-' < patchlist |
< lam -s "tp/" - -s ".gz" |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${STATSREDIR} | fetch_progress
---
> # fetch metadata files using wget behind a proxy
> for current_file in `tr '|' '-' < patchlist | lam -s "tp/" - -s ".gz"`
> do
> ${PHTTPGET} ${SERVERNAME}/${current_file} > /dev/null 2>&1
> done
>
721,724c725,732
< echo ${NDEBUG} "metadata files... "
< lam -s "f/" - -s ".gz" < filelist |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${QUIETREDIR}
---
> echo ${NDEBUG} "metadata files..."
>
> # fetch metadata files using wget behind a proxy
> for current_file in `cat filelist`
> do
> ${PHTTPGET} ${SERVERNAME}/f/${current_file}.gz > /dev/null 2>&1
> done
>
763,765c771,776
< tr '|' '-' < patchlist | lam -s "bp/" - |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${STATSREDIR} | fetch_progress
---
>
> # fetch metadata files using wget behind a proxy
> for current_file in `tr '|' '-' < patchlist | lam -s "bp/" - -s ".gz"`
> do
> ${PHTTPGET} ${SERVERNAME}/${current_file} > /dev/null 2>&1
> done
794,796c805,809
< lam -s "f/" - -s ".gz" < filelist |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${QUIETREDIR}
---
> # fetch metadata files using wget behind a proxy
> for current_file in `lam -s "f/" - -s ".gz" < filelist`
> do
> ${PHTTPGET} ${SERVERNAME}/${current_file} > /dev/null 2>&1
> done
```

I was wondering if it is useful to change the adoption of _phttpget_ into portsnap, since this problem could happen to others. In short, the problem was that phttpget does not handle correctly an _http_proxy_ of the form

```
http://user:password@proxy:port
```
considering the _assword@proxy_ as the port specification.


----------



## kpa (Nov 26, 2012)

Make it a configurable option in /etc/portsnap.conf and ask on the mailing lists (current, stable) if this change could be committed.


----------



## fluca1978 (Nov 28, 2012)

Thanks to the guys on the mailing list now this patch exploits fetch(1) to download the stuff. It has also a parameter in the /etc/portsnap.conf file to enable the usage of fetch(1). At the time it is available on a repository of mines.


----------



## rtweeg (Mar 20, 2014)

Good afternoon,

Could guide me on how to apply this patch?
I'm having some doubts:

1) In which of these files 'portsnap' Should I apply the patch?


```
/etc/portsnap.conf
/usr/sbin/portsnap
/usr/share/examples/etc/portsnap.conf
/usr/share/man/man5/portsnap.conf.5.gz
/usr/share/man/man8/portsnap.8.gz
/usr/src/etc/portsnap.conf
/usr/src/share/man/man5/portsnap.conf.5
/usr/src/usr.sbin/portsnap
/usr/src/usr.sbin/portsnap/Makefile
/usr/src/usr.sbin/portsnap/Makefile.inc
/usr/src/usr.sbin/portsnap/make_index
/usr/src/usr.sbin/portsnap/make_index/Makefile
/usr/src/usr.sbin/portsnap/make_index/make_index.c
/usr/src/usr.sbin/portsnap/phttpget
/usr/src/usr.sbin/portsnap/phttpget/Makefile
/usr/src/usr.sbin/portsnap/phttpget/phttpget.c
/usr/src/usr.sbin/portsnap/portsnap
/usr/src/usr.sbin/portsnap/portsnap/Makefile
/usr/src/usr.sbin/portsnap/portsnap/portsnap.8
/usr/src/usr.sbin/portsnap/portsnap/portsnap.sh
```

2) The command syntax would be this one:

`# patch -p1 < portsnap.patch`

Like I said, I do not know if the patch is to be applied in portsnap which is inside the directory /usr/sbin or whether it is to be applied to portsnap which is in one of the directories shown above.

Thank you in advance for any help.

hugs


----------

