# freebsd-update keeps failing



## balanga (Dec 1, 2022)

I've been having endless problems trying to run `freebsd-update` on various sytems which all seem to end in failure. I'm currently attempting to upgrde 12.2 to 13.1 and have already fetched 1000's of patches. If I run the update again does it start from scratch or continue where it left off?

Can anyone suggest a way to get round my problem? Maybe manually select an update server?..


----------



## SirDice (Dec 1, 2022)

balanga said:


> If I run the update again does it start from scratch or continue where it left off?


Files are cached in /var/db/freebsd-update.



balanga said:


> Can anyone suggest a way to get round my problem?


I would suggest running freebsd-update(8) through a caching proxy server. This will be much faster if you have lots of servers to update.


----------



## gpw928 (Dec 1, 2022)

You don't mention how they fail.  I have problems when I use ksh, with my environment, path, and aliases.  So I always completely sanitise the environment before running the update:
	
	



```
/usr/bin/env - /bin/sh
freebsd-update fetch
freebsd-update install
exit
```


----------



## mer (Dec 1, 2022)

I've not experienced issues, but with what gpw928 says about environment and shell is a good idea.  I've run into issues where bash was assumed and using tcsh caused wierd issues.  That's one thing I dislike about most Linux distros:  they have all kinds of aliases  variables set by default that annoy me.


----------



## BobSlacker (Dec 1, 2022)

gpw928 said:


> You don't mention how they fail.  I have problems when I use ksh, with my environment, path, and aliases.  So I always completely sanitise the environment before running the update:
> 
> 
> 
> ...


I do exactly that with a shell script.


```
#!/usr/bin/env bash

if [[ "${UID}" != "0" ]] ; then echo "UID != 0" ; exit ; fi

printexec() {

    printf "\e[91;1m * ${*}\e[0m\n"
    "${@}"

}

#if [[ $(date +%u) -eq 3 ]] || [[ $(date +%u) -eq 7 ]] ; then

    printexec freebsd-update fetch install
    printexec pkg update
    printexec pkg upgrade
    printexec portsnap fetch update
    printf "\e[91;1m * Done\e[0m\n"
    exit 0

#fi
```

Works like a charm every time.


----------



## Alain De Vos (Dec 1, 2022)

Or can do "su -"


----------



## balanga (Dec 1, 2022)

gpw928 said:


> You don't mention how they fail.



Sometimes it just says fail, sometimes I get 'file' has incorrect hash when the last of say 1054 files have been fetched.


----------



## mer (Dec 1, 2022)

Have you tried to verify a failed hash by hand?  A failed hash is "computed hash does not match what someone else says the hash should be".
I'm not sure where the one to compare against is stored (maybe locally on the system, maybe stored on a remote site).
It's also possible that the stored value should have been updated but wasn't for some reason;  that would be a false negative.
Have you tried running just the freebsd-update fetch command by hand in a clean environment, just to eliminate everything else?
It looks like freebsd-update has an "-s" which should let you pick a specific server to use, maybe try that when running it by hand.


----------



## balanga (Dec 1, 2022)

SirDice said:


> Files are cached in /var/db/freebsd-update.



Are you saying that if I just keep re-running the update I will eventually get all the files I need? For example when fetching 801 files and I get an 'incorrect hash' with the last file, have the previous 800 been retrieved correctly? 

In /var/db/freebsd-update/files/ I have almost 40,000 files going back to 2019. Is there any way of identifying what they relate to and would I be able to copy them to a 'freebsd update server' if/when I manage to set one up?


----------



## SirDice (Dec 1, 2022)

balanga said:


> For example when fetching 801 files and I get an 'incorrect hash' with the last file, have the previous 800 been retrieved correctly?


It should, yes.



balanga said:


> In /var/db/freebsd-update/files/ I have almost 40,000 files going back to 2019. Is there any way of identifying what they relate to and would I be able to copy them to a 'freebsd update server' if/when I manage to set one up?


No idea. You don't need to set up your own update server. Just set up a caching proxy for this. It's all 'standard' HTTP traffic and the files are easily cached with a caching proxy server. So the next server you run freebsd-update(8) on will get the files from the proxy's cache. Which will be much faster than constantly trying to fetch these files from update.freebsd.org.


----------



## balanga (Dec 1, 2022)

mer said:


> Have you tried to verify a failed hash by hand?  A failed hash is "computed hash does not match what someone else says the hash should be".
> I'm not sure where the one to compare against is stored (maybe locally on the system, maybe stored on a remote site).
> It's also possible that the stored value should have been updated but wasn't for some reason;  that would be a false negative.
> Have you tried running just the freebsd-update fetch command by hand in a clean environment, just to eliminate everything else?
> It looks like freebsd-update has an "-s" which should let you pick a specific server to use, maybe try that when running it by hand.


The only time I see a filename shown is when an 'incorrect hash' is reported. I suspect that in such a case that file is not stored on the system.

If I want to use the '-s' option, how do I find which servers are available?

I think I will need to deploy script() to keep track of my attempts. 

When running 'fetch' as a separate step, does that retrieve all the required files? And if there has been a previous fail, does it make use of previously retrieved files?


----------



## VladiBG (Dec 1, 2022)

Is this physical server or virtual machine?


----------



## balanga (Dec 1, 2022)

VladiBG said:


> Is this physical server or virtual machine?


I'm updating from update1.freebsd.org


----------



## astyle (Dec 1, 2022)

Funny, I updated the 13-RELEASE kernel (in a VirtualBox VM) to 13.1-RELEASE just following the Handbook... I think about 20k patches got downloaded, that part took up about half an hour... For me, it boiled down to keeping track of the steps:

```
# freebsd-update -r 13.1-RELEASE upgrade
...
(Stuff happens. In my case,  I think I first had patches downloaded, and then I had to edit /etc/groups because somehow the update would leave me out of the video group... :/ )
...
# freebsd-update install
# reboot
```
(after reboot)

```
# freebsd-update install
# uname -a
(should reflect the update)
```

That's it. Sometimes, keeping it simple yields the best results.


----------



## VladiBG (Dec 1, 2022)

balanga said:


> I'm updating from update1.freebsd.org


Is the server that you are updating is physical server or virtual machine? I'm not asking from where you are updating.


----------



## mer (Dec 1, 2022)

balanga  To the best of my knowledge, freebsd-update fetch should pick up where it left off.  If it needs to get 100 files, fails at file 51, when you rerun it I think it starts at file 51.  It may do a quick check of the others (maybe a hash check) but I think it picks it up.  It's been a while since I've run into a failure so I'm going by memory.


----------



## astyle (Dec 1, 2022)

VladiBG said:


> Is the server that you are updating is physical server or virtual machine? I'm not asking from where you are updating.


Why would the difference  between physical and virtual matter? If you do steps correctly (as Handbook shows you), that physical/virtual difference really shouldn't matter. Not for update server, and not for machine being updated... 

Edit: FWIW, that update from my earlier post - that was just a quick chore I did last night...


----------



## balanga (Dec 1, 2022)

VladiBG said:


> Is the server that you are updating is physical server or virtual machine? I'm not asking from where you are updating.


I'm just upgrading a laptop. It isn't running a virtual machine.


----------



## balanga (Dec 1, 2022)

astyle said:


> Funny, I updated the 13-RELEASE kernel (in a VirtualBox VM) to 13.1-RELEASE just following the Handbook... I think about 20k patches got downloaded, that part took up about half an hour... For me, it boiled down to keeping track of the steps:
> 
> ```
> # freebsd-update -r 13.1-RELEASE upgrade
> ...


I know the theory and it has worked without problems on numerous occasions in the past but I seemed to have had a lot of issues with upgrading to 13.1-RELEASE. I guess I should assume that I simply have a unreliable connection.

As I understand it all the fetches are cumulative and I just need to run the update over and over until I get all the required files.


----------



## VladiBG (Dec 1, 2022)

astyle said:


> Why would the difference  between physical and virtual matter? If you do steps correctly (as Handbook shows you), that physical/virtual difference really shouldn't matter. Not for update server, and not for machine being updated...
> 
> Edit: FWIW, that update from my earlier post - that was just a quick chore I did last night...


In some virtual machines the hardware based checksum(Offloading) of the RX/TX for the NIC need to be disabled so it can be done on software level.
The problem with the receiving a corrupted (damaged) file which is catch by the checksum verification of freebsd-update means that he has some sort of connection issue. I would reset the TCP counters with `netstat -sz` then perform a freebsd-update and after receiving again an  incorrect hash error will check the TCP statistic with `netstat -ss -p TCP`


----------



## astyle (Dec 1, 2022)

balanga said:


> I know the theory and it has worked without problems on numerous occasions in the past but I seemed to have had a lot of issues with upgrading to 13.1-RELEASE. I guess I should assume that I simply have a unreliable connection.
> 
> As I understand it all the fetches are cumulative and I just need to run the update over and over until I get all the required files.


I think the command  `# freebsd-update -r 13.1-RELEASE upgrade` will actually let you know when everything is done and ready for `# freebsd-update install`... The important part is to keep re-running the first command until it finishes successfully.


----------



## astyle (Dec 1, 2022)

VladiBG said:


> In some virtual machines the hardware based checksum(Offloading) of the RX/TX for the NIC need to be disabled so it can be done on software level.
> The problem with the receiving a corrupted (damaged) file which is catch by the checksum verification of freebsd-update means that he has some sort of connection issue. I would reset the TCP counters with `netstat -sz` then perform a freebsd-update and after receiving again an  incorrect hash error will check the TCP statistic with `netstat -ss -p TCP`


Last time virtual NICs had that kind of problem was back in 2010s, when virtualization was a newcomer to the market. Since then, codebases have improved, so ending up with a corrupted download is less of an issue.  Or was there a recent event where an up-to-date VM ended up with a corrupted download (preferably involving `freebsd-update`)?


----------



## xinqu (Dec 5, 2022)

balanga said:


> Sometimes it just says fail, sometimes I get 'file' has incorrect hash when the last of say 1054 files have been fetched.



I have the same issue on every update. Cause is a *Sophos UTM* firewall which does *transparent http-proxying*. I solved this by ssh-tunneling http-traffic to a different squid-proxy on the internet / outside the LAN. Works flawlessly (and much *much* more faster for subsequent hosts).

I also tried to use a squid proxy inside the LAN, but this only cached the 'incorrect hash' and I had to clean up the cache manually all the time.

Besides that, I can confirm that using a squid proxy can speed up updates by factor 100x, especially if you have to update multiple machines on the same LAN.


----------



## astyle (Dec 5, 2022)

xinqu said:


> I have the same issue on every update. Cause is a *Sophos UTM* firewall which does *transparent http-proxying*. I solved this by ssh-tunneling http-traffic to a different squid-proxy on the internet / outside the LAN. Works flawlessly (and much *much* more faster for subsequent hosts).
> 
> I also tried to use a squid proxy inside the LAN, but this only cached the 'incorrect hash' and I had to clean up the cache manually all the time.
> 
> Besides that, I can confirm that using a squid proxy can speed up updates by factor 100x, especially if you have to update multiple machines on the same LAN.


Wouldn't that mean that you gotta have a ton of storage space for the proxy cache?


----------



## bob2112 (Dec 5, 2022)

freebsd-update relies on the proxy server string in the environment to seed the random selection of mirrors. If this isn't set consistently across installations,  it leads to a low hit rate in the cache. With an intercepting proxy (which is what people usually mean when they say transparent proxy) freebsd-update makes no attempt to use consistent URIs. In that case it may be possible to go directly to the proxy by setting the  environment variable.

One other thing is that freebsd-update relies heavily on HTTP pipelining. If a proxy can't handle that efficiently it can make the "fetch" stage very slow.


----------



## SirDice (Dec 5, 2022)

astyle said:


> Wouldn't that mean that you gotta have a ton of storage space for the proxy cache?


Not that bad, as far as I can tell.

Using Apache as a caching proxy, for the sole purpose of caching freebsd-update(8):

```
# du -sh /var/cache/freebsd-update/
1.3G    /var/cache/freebsd-update/
```

It's been a while since I did a major version upgrade using this cache. I did do a round of patch updates recently. This cache was set up several major and minor version upgrades ago and I never bothered to check how much it actually uses. Things may have timed out eventually, I do have htcacheclean(8) running. 

```
htcacheclean_cache="/var/cache/freebsd-update/"
htcacheclean_enable="YES"
```


----------

