# "make install" slow ftp download sites, how to change?



## johnblue (Apr 11, 2009)

When I do the 'make clean install' tango to install stuff from ports, sometimes the tar file is downloaded from slow ftp servers.

ie:  mySQL client

```
=> mysql-5.0.77.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch from ftp://ftp.fi.muni.cz/pub/mysql/Downloads/MySQL-5.0/.
```

I seem to remember some discussion about this on the defunct bsdforums site but I cannot remember what was said.  Is there any particular reason why a server in the Czech Republic was selected to download from?  And if it is selectable, how can I point it to faster servers?

Thanks!


----------



## Lowell (Apr 11, 2009)

If you look at the Makefile for that port, you will see that it uses a variable called "MASTER_SITE_MYSQL" to provide a list of mirrors from which the sources can be downloaded.  You can define that variable yourself, with sites that are faster for you at the top of the list.  

That variable only applies to mysql, of course.
Other ports may use other variables or define the download sites directly.

As to why some particular site was chosen as the top choice, you would need to ask the people who committed the file containing the list.  The usual reason is that certain sites are more authoritative for the project in question than others.


----------



## hedwards (Apr 12, 2009)

johnblue said:
			
		

> I seem to remember some discussion about this on the defunct bsdforums site but I cannot remember what was said.  Is there any particular reason why a server in the Czech Republic was selected to download from?  And if it is selectable, how can I point it to faster servers?
> 
> Thanks!


I wish one could set a minimum transfer rate. Basically abort if it goes below about 5kbps or so.

What I do is temporarily add the URL to my firewalls block list. It can't connect and will pretty quickly switch to another site.


----------



## downloadlands (Dec 7, 2009)

Lowell said:
			
		

> If you look at the Makefile for that port, you will see that it uses a variable called "MASTER_SITE_MYSQL" to provide a list of mirrors from which the sources can be downloaded.  You can define that variable yourself, with sites that are faster for you at the top of the list.
> 
> That variable only applies to mysql, of course.
> Other ports may use other variables or define the download sites directly.
> ...



Thank you very much


----------



## DutchDaemon (Dec 7, 2009)

Try ports-mgmt/fastest_sites.

You can run it from cron like this:


```
0 6 * * *       /usr/local/bin/fastest_sites > /usr/local/etc/ports_sites.conf
```

and reference it from /etc/make.conf like this:


```
.include "/usr/local/etc/ports_sites.conf"
```

That's all.


----------



## SIFE (Dec 10, 2009)

this work for some ports as i know :

```
MASTER_SITE_OVERRIDE= \
ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ fetch
```
Mr DutchDaemon tell me from where you get like this configuration files ,i had not seen it FreeBSD handbook .


----------



## DutchDaemon (Dec 10, 2009)

I don't know what you mean .. the /usr/local/etc/ports_sites.conf gets created by putting the command from post #5 in root's crontab. Every time the cron job runs, that file gets updated with the fastest mirrors.


----------



## downloadlands (Dec 21, 2009)

ee /etc/make.conf
[CMD="MASTER_SITE_OVERRIDE?=http://ports.hshh.org/${DISK_SUBDIR}/"][/CMD]

I often use this method


----------



## downloadlands (Dec 21, 2009)

ee /etc/make.conf

MASTER_SITE_OVERRIDE?=http://ports.hshh.org/${DISK_SUBDIR}/


----------



## chessmaster (Jan 26, 2012)

DutchDaemon said:
			
		

> Try ports-mgmt/fastest_sites.
> 
> You can run it from cron like this:
> 
> ...



Is there anything that has to be configured to enable this? Or does it automatically work after rehash?


----------



## DutchDaemon (Jan 26, 2012)

The above is all you need to do. Hence: "That's all".


----------



## chessmaster (Feb 15, 2012)

DutchDaemon said:
			
		

> Try ports-mgmt/fastest_sites.
> 
> You can run it from cron like this:
> 
> ...



I'm sorry. I'm not familiar with cron jobs. How is the command run in the 
	
	



```
section? I tried 
[code]fastest_sites > /usr/local/etc/ports_sites.conf &
```
then adding this to /etc/make.conf

```
.include "/usr/local/etc/ports_sites.conf"
```

Does this achieve the same results?


----------



## DutchDaemon (Feb 16, 2012)

Yes, but you will only get one result with that. You should run this on a regular basis, which is usually achieved by putting it in a cron job.

As root:
[cmd=]crontab -e[/cmd]

Paste:

```
0 6 * * *       /usr/local/bin/fastest_sites > /usr/local/etc/ports_sites.conf
```

Save.

And it will run each day at 6 AM.


----------



## ankscorek (Jan 4, 2015)

DutchDaemon said:


> Yes, but you will only get one result with that. You should run this on a regular basis, which is usually achieved by putting it in a cron job.
> 
> As root:
> [cmd=]crontab -e[/cmd]
> ...



This error of 0 bad hour is coming:

```
/tmp/crontab.jLAVYGGbZy: 1 lines, 91 characters.
crontab: installing new crontab
"/tmp/crontab.jLAVYGGbZy":0: bad hour
crontab: errors in crontab file, can't install
Do you want to retry the same edit?
```


----------



## DutchDaemon (Jan 4, 2015)

The line I posted is correct. If it doesn't work, there must be an error or a typo (control character, vi command?) in it.


----------



## chrbr (Jan 5, 2015)

Below are the first lines of my /etc/crontab. Here the minutes come first.

```
# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD: stable/10/etc/crontab 194170 2009-06-14 06:37:19Z brian $
#
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
#
#minute hour  mday  month  wday  who  command
#
*/5  *  *  *  *  root  /usr/libexec/atrun
```
I am not sure, but swapping the 0 and the 6 might be better.


----------



## DutchDaemon (Jan 5, 2015)

No, read crontab(5), it's "0 6" for 6:00.

And don't edit /etc/crontab, edit the user's crontab.


----------



## ankscorek (Jan 5, 2015)

DutchDaemon said:


> The line I posted is correct. If it doesn't work, there must be an error or a typo (control character, vi command?) in it.



it is a simple copy and a paste of the line written by you


----------



## DutchDaemon (Jan 5, 2015)

I doubt it. This is a copy and paste of the line, and a copy and paste of the result. 
	
	



```
0 6 * * *       /usr/local/bin/fastest_sites > /usr/local/etc/ports_sites.conf

"/tmp/crontab.khtWpIrhBa" 3 lines, 201 characters
crontab: installing new crontab
```


----------



## chrbr (Jan 6, 2015)

Dear DutchDaemon,


DutchDaemon said:


> No, read crontab(5), it's "0 6" for 6:00.
> 
> And don't edit /etc/crontab, edit the user's crontab.


I have tried `crontab -e` and entered the line you have proposed. I can confirm that the line is accepted. It works as normal user and as root. I should have tried that first. I have been irritated by the error message regarding hour 0. Sorry for the noise.


----------



## ankscorek (Jan 7, 2015)

Strange this time `crontab -e` followed by the line

```
0 6 * * * /usr/local/bin/fastest_sites > /usr/local/etc/ports_sites.conf
```
gave good results

```
/tmp/crontab.iIfg3ylVXj: 2 lines, 2 characters.
crontab: installing new crontab
```

`crontab -l` showed this

```
root@noname:/home/bsdfrankie # crontab -l


root@noname:/home/bsdfrankie #
```

A total blank. Now if I want to edit 0 6 to something else I am unable to do this.


```
$ uname -a
FreeBSD noname.noname.edu 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 22:51:51 UTC 2014  root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386
```


----------



## DutchDaemon (Jan 7, 2015)

See if it shows up under /var/cron/tabs/ somewhere.


----------



## ankscorek (Jan 9, 2015)

DutchDaemon said:


> See if it shows up under /var/cron/tabs/ somewhere.



It does show that installation has taken place

```
root@noname:/var/cron/tabs # cat root
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.iIfg3ylVXj installed on Wed Jan  7 16:53:09 2015)
# (Cron version -- $FreeBSD: releng/10.1/usr.sbin/cron/crontab/crontab.c 239991 2012-09-01 14:45:15Z ed $)
```

How am I supposed to edit it?


----------



## chrbr (Jan 9, 2015)

Dear ankscorek, just in case you are not familiar with vi. `crontab -e` as root is correct. It copies the current crontab into a vi. Press "i" to change to the insert mode. Then insert the line, either by typing or by copy and paste. Then press the key <esc> to leave the insert mode. If you do it more often the computer might beep, but this is ok. Then type ":wq". This should save the file and close the editor. This is very intuitive . The new crontab should appear as the result of `crontab -l`.


----------



## DutchDaemon (Jan 9, 2015)

If that is too involved, run the command `export EDITOR=ee` or `setenv EDITOR ee` (depending on your shell) before running `crontab -e` That'll give you an easier editor.


----------



## ankscorek (Jan 9, 2015)

chrbr said:


> Dear ankscorek, just in case you are not familiar with vi. `crontab -e` as root is correct. It copies the current crontab into a vi. Press "i" to change to the insert mode. Then insert the line, either by typing or by copy and paste. Then press the key <esc> to leave the insert mode. If you do it more often the computer might beep, but this is ok. Then type ":wq". This should save the file and close the editor. This is very intuitive . The new crontab should appear as the result of `crontab -l`.



I am dumb, but, hey I guess I used "insert" copy paste the line followed by ":wq" and then I got



ankscorek said:


> /tmp/crontab.iIfg3ylVXj: 2 lines, 2 characters. crontab: installing new crontab





DutchDaemon said:


> If that is too involved, run the command `export EDITOR=ee` or `setenv EDITOR ee` (depending on your shell) before running `crontab -e`. That'll give you an easier editor.



vi is an easy editor.

My query is that I do not see the line which I copied, pasted, saved and exited the editor. I want to edit the 0 6 to something else.


----------



## DutchDaemon (Jan 9, 2015)

Then just type the line by hand and change what you need..


----------



## ankscorek (Jan 10, 2015)

Ok if I understood correctly, a blank crontab has been created and that I need to create a new one?


----------



## chrbr (Jan 10, 2015)

This is correct. With `crontab -e` you edit the current crontab or create a new one.


----------



## ankscorek (Jan 10, 2015)

I followed the same procedure as earlier and now this is it


```
root@noname:/home/xxxxxx # crontab -l
0 21 * * *  /usr/local/bin/fastest_sites > /usr/local/etc/ports_sites.conf
```

..?

And then this is what happened

```
# /usr/local/bin/fastest_sites > /usr/local/etc/ports_sites.conf
make: "/etc/make.conf" line 10: .include filename must be delimited by '"' or '<'
make: Fatal errors encountered -- cannot continueTraceback (most recent call last):
  File "/usr/local/bin/fastest_sites", line 145, in <module>
  fd = open(sites_mk, "r")
IOError: [Errno 2] No such file or directory: '\nmake: stopped
```

line 10 of /etc/make.conf is

```
.include /usr/local/etc/ports_sites.conf
```


----------



## chrbr (Jan 10, 2015)

Dear ankscorek,
may be now you just need the quotes as in post #5.


DutchDaemon said:


> and reference it from /etc/make.conf like this:
> 
> 
> ```
> ...


But I am not sure about ports-mgmt/fastest_sites. This is just the difference I have noted.


----------



## ankscorek (Jan 11, 2015)

chrbr said:


> Dear ankscorek,
> may be now you just need the quotes as in post #5.
> 
> But I am not sure about ports-mgmt/fastest_sites. This is just the difference I have noted.



Thanks chrbr for telling me to read and understand the error message correctly. The suggestion worked.


----------

