# freebsd-update improvement idea



## tux2bsd (Sep 27, 2021)

A simple method to improve freebsd-update with little effort for those with the coder-fu and necessary server side access.

e.g.  current release = 13.0-RELEASE-p4


```
HTTP GET https://signal.update.freebsd.org/13.0-RELEASE-p4 if response is 200
then
  do nothing
else
  get back a 302 (or whatever, e.g. points to 13.1-RELEASE-p8 (because it's been a while in this fictional example...)
  do the normal freebsd-update fetch(/install), i.e. Inspecting system etc...

# server side burst rate limit N requests per hour, please yourself.
```

Then all of the crunching (Fetching/Inspecting...) only happens only if there is an update.  Saving client side and server side resources markedly.


----------



## tux2bsd (Sep 27, 2021)

It can also handle multiple versions, an endpoint for 13.x and one for 14.x, rinse and repeat as versions move through time.

Over all the main goal to achieve is the reduction in time to process.  freebsd-update dives into an intense self inspection prematurely, that should only happen AFTER a trivial preliminary test i.e. "is this self inspection necessary".


----------



## tux2bsd (Sep 27, 2021)

Or, it could simply be a helper script e.g. `freebsd-update-signal` but the server side component would need to be there to work...

This is probably the better approach.  I chose the word signal as in traffic signal.  Output and exit codes reflect outcome of `freebsd-update-signal` green no updates, orange point release update, red full version version.

Something like that...

`freebsd-update-signal` = green && echo yay
`freebsd-update-signal` = orange && freebsd-update fetch install (with env PAGER=cat)
`freebsd-update-signal` = red && freebsd-update cron && email_admin "wake up work to do"


----------



## jbo (Sep 27, 2021)

Which benefits do you see that would be worth the costs?


----------



## SirDice (Sep 27, 2021)

Note that /usr/sbin/freebsd-update is a (rather large) shell script, so you can easily look at the code to see what it does.


----------



## zirias@ (Sep 27, 2021)

This would be a misuse of HTTP. If you model a specific patch release as a resource, a redirect would mean this same resource is now moved, which obviously isn't the correct semantics.

You could model a release (without patch level) as a resource and do a conditional get, using the patchlevel as an Etag, for example. But why so complicated at all? I don't know what exactly freebsd-update(8) is doing right now, but fecthing e.g. a document like this

```
{
   "12.3": "p8",
   "12.4": "p6",
   "13.0": "p4"
}
```
would allow for a simple local check already. I assume something like this is done anyways?


----------



## tux2bsd (Sep 27, 2021)

jbodenmann said:


> Which benefits do you see that would be worth the costs?





tux2bsd said:


> Saving client side and server side resources markedly.


Client side: It takes too long to run freebsd-update, so long it's actually rather absurd.  `freebsd-update fetch` ~1m20s (rpi3+, maybe that's halved on fast hardware) but it's still extremely high for each run, just to check!  The method I'm proposing would be a single web request that would take less than ~100ms from most 1st world countries (if there's nothing to do - see above, method already explained).

Server side:  The update servers would no longer serve the metadata each time, only when it is necessary.  I don't work for FreeBSD so I'm not privy to how much benefit but it would definitely reduce load.

Of the two the significant benefit is seen on the client side.


----------



## SirDice (Sep 27, 2021)

freebsd-update.sh « freebsd-update « usr.sbin - src - FreeBSD source tree
					






					cgit.freebsd.org


----------



## sidetone (Sep 27, 2021)

`freebsd-update updatesready`


----------



## SirDice (Sep 27, 2021)

Yeah, I looked at that too. But it has to have fetched updates first. And it's the fetch stage tux2bsd seems to have problems with.

You can speed things along a bit (especially if you have a bit of dodgy connection to the update servers) by using a caching proxy solution. That's particularly useful if you have to update/upgrade multiple machines. Besides only having one outgoing connection (only the proxy server has to have access to update.freebsd.org) the meta data and patch files themselves can be cached locally. This will speed up the downloads for all the other systems. But there's still quite a lot checking happening locally once the meta data has been downloaded. And to be honest that's a good thing, you do want to get all the right patch files that actually apply to your system. And if you only look at the version string there's a risk you're going to miss patches (especially if the previous update wasn't entirely completed).

Could this be improved, I'm sure it can. But I much rather have something running slow, steady and thorough then something that's fast and miss things.


----------



## kpedersen (Sep 27, 2021)

It is my understanding that you need to fetch the metadata before your system can even work out whether you need to update anything.

Many times it detects the metadata has updated, so it fetches it, only to subsequently process it and realize that nothing that *I* have installed needs updating.

Yes, this can take a while tux2bsd is correct here. I am not quite getting >1min but often it is longer than fetching the actual packages I tend to install / update.

Potentially unless you need such up-to-date packages, the release or quarterly_[1,2,3,4] repos should have less of these metadata updates. My personal (perhaps less popular) solution is just make an offline mirror every 6 months. (only around ~98GB)


----------



## tux2bsd (Sep 28, 2021)

Zirias said:


> This would be a misuse of HTTP...


That is your opinion.  Making a small software utility decide what to do based on a controlled exchange of information is entirely achievable using HTTP(S), not every wheel needs reinventing.


SirDice said:


> And it's the fetch stage _*tux2bsd*_ seems to have problems with.


The "fetch" is unnecessary (in my method).  And it's actually the Inspection (of the fetch stage) that is most bizarre (if no updates do Inspection anyway... ???)


kpedersen said:


> It is my understanding that you need to fetch the metadata before your system can even work out whether you need to update anything.


From above: freebsd-update dives into an intense self inspection prematurely, that should only happen AFTER a trivial preliminary test i.e. "is this self inspection necessary".


tux2bsd said:


> it could simply be a helper script e.g. `freebsd-update-signal`


This was the route I decided on.  I will prototype it when I get time (that is a made up script name, it doesn't exist).


SirDice said:


> But I much rather have something running slow, steady and thorough then something that's fast and miss things.


It is a misunderstanding of computing to think that a quick result cannot be accurate.  But with the (hypothetical) helper script you are free to a) not use it b) use it but perform freebsd-update by hand or however you always have - the onus is on the user.


----------



## zirias@ (Sep 28, 2021)

tux2bsd said:


> That is your opinion.


No, this is a fact. An URL in HTTP identifies a _resource_.


----------



## tux2bsd (Sep 28, 2021)

sidetone said:


> `freebsd-update updatesready`


  "updatesready -- Check if there are fetched updates ready to install"

Unfortunately not, because the (slow) fetch still has to occur.


----------



## tux2bsd (Sep 28, 2021)

Zirias said:


> No, this is a fact. An URL in HTTP identifies a _resource_.


Take your hairsplitting elsewhere and stop detracting from the subject at hand.


----------



## zirias@ (Sep 28, 2021)

Some people, if they don't immediately understand what's wrong with their idea, why it's wrong and how it could be improved, do a sane thing: Reflect, research, maybe ask.

Some other people, not so much 

On topic: If you really want to leverage HTTP itself for a version check, do it correctly (conditional GET with an Etag). Still, as described above, having a simple resource that contains the current patchlevels of all supported releases would serve the same purpose (a quick check with little network overhead), is simpler to implement and more flexible (e.g. also answers the question whether there's a new minor release, whether the current release is EOL, etc).


----------



## SirDice (Sep 28, 2021)

tux2bsd said:


> Unfortunately not, because the (slow) fetch still has to occur.


It's not the fetching of the files from the server that's slow. It's the fact the Raspberry Pi is rather slow to read from an SD card.


----------



## tux2bsd (Sep 29, 2021)

SirDice said:


> It's the fact the Raspberry Pi is rather slow to read from an SD card.


15s with SSD.  It is slow, period.


----------



## tux2bsd (Sep 29, 2021)

I modified freebsd-update directly so that it performs a comparison and skips the "Inspection" *if there are no new updates.*  It now takes between 0.7s and 2.1s, down from ~15s (this is on PC with SSD mentioned in the immediately previous post) the main variation being network induced (far away in NZ).


----------



## jbo (Sep 29, 2021)

jbodenmann said:


> Which benefits do you see that would be worth the costs?


I would still like to get back to my initial question here: What benefits do you see that make it worth the efforts? I fully understand that there is room for improvement, especially if looking only at a very specific metric such as "speed". However, usually updating the base system is a comparably rare occasion. It's not something that runs every 5 minutes periodically. I'd be really interested to know the situation which makes it necessary or worth addressing this.

Other than that, I think that SirDice already provided other solutions: Have a proxy/mirror nearby. That's a very easy, flexible & scalable approach.


----------



## tux2bsd (Sep 30, 2021)

jbodenmann said:


> initial question / Sirdice other solutions



Your initial question was answered.  SirDice didn't provide any solution, certainly not one I was looking for but what he suggested may be a useful idea for someone else.  SirDice comments regarding freebsd-update's code and the RPI SD card were appreciated and helpful.



jbodenmann said:


> speed



It isn't about "speed", it is about reducing unnecessary inefficiency.  It's that simple.

My underlying goal was to get `freebsd-update fetch` be more efficient *when there are no updates*.  Originally I proposed a grandiose idea and that was just an idea (grandiose in that I never had any expectation anyone else would come to the party).

So.

What happened in reality:  I did some experimenting with freebsd-update and found I could achieve most of my goal with no additional external aspect.

*When there are no updates *I've taken `freebsd-update fetch` from 15s down to 0.7s (that's on SSD, I'm yet to see what it does on the Raspberry Pi 3 (my prediction 1m20s down to ~5s).

I've reduced unnecessary inefficiency.  I'm happy.


----------



## tux2bsd (Sep 30, 2021)

tux2bsd said:


> I'm yet to see what it does on the Raspberry Pi 3 (my prediction 1m20s down to ~5s).



Copied my change over...  1m20s down to 1.5s

A touch better than I expected.


```
root@freebsd-dns:/usr/sbin # date ; time freebsd-update.old fetch ; date
Thu Sep 30 21:31:23 NZDT 2021
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p4.
70.123u 12.270s 1:19.25 103.9%  17+171k 0+35io 0pf+0w
Thu Sep 30 21:32:42 NZDT 2021

#original above: freebsd-update.old
#modified below: freebsd-update

root@freebsd-dns:/usr/sbin # date ; time freebsd-update fetch ; date
Thu Sep 30 21:32:53 NZDT 2021
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
debug: Thu Sep 30 21:32:54 NZDT 2021
Already running 13.0-RELEASE-p4, no updates to perform.
0.421u 0.931s 0:01.51 89.4%     81+164k 0+8io 0pf+0w
Thu Sep 30 21:32:54 NZDT 2021
```


----------



## zirias@ (Sep 30, 2021)

Improving performance by (simple?) client-side changes definitely sounds like a good thing! Instead of showing your measurements, the actual change would be much more interesting though. Looking at the change, people could review it (and also double-check it doesn't break anything).

Of course, submitting it on bugzilla or phabricator would be best, cause that would have the best chance to receive meaningful reviews. If the change doesn't have unintended side-effects, it could be merged, so everyone can profit from it.


----------



## tux2bsd (Sep 30, 2021)

Zirias said:


> Instead of showing your measurements, the actual change would be much more interesting though.


You need to work on your communication skills.  If I did not take the time to make the change and show the measurements you would not be interested...

The change is simple.

I have yet to test the obvious case of when an update is available which I will do at some point with a fresh install.


----------



## zirias@ (Sep 30, 2021)

tux2bsd said:


> You need to work on your communication skills.





tux2bsd said:


> If I did not take the time to make the change and show the measurements you would not be interested...


It's kind of difficult to be interested in something that doesn't exist.


----------



## tux2bsd (Sep 30, 2021)

I intended to share it after I have the chance to test it more.  There is no use if I've made a mistake and blocked updates from happening, for example.


tux2bsd said:


> I have yet to test the obvious case of when an update is available which I will do at some point with a fresh install.


I expect it to work without issue but need to verify this first.


----------



## dd_ff_bb (Sep 30, 2021)

Summer is over, winter is almost here so why the "Raging Penguin Syndrome"


----------



## D-FENS (Sep 30, 2021)

tux2bsd said:


> Client side: It takes too long to run freebsd-update, so long it's actually rather absurd.  `freebsd-update fetch` ~1m20s (rpi3+, maybe that's halved on fast hardware) but it's still extremely high for each run, just to check!  The method I'm proposing would be a single web request that would take less than ~100ms from most 1st world countries (if there's nothing to do - see above, method already explained).
> 
> Server side:  The update servers would no longer serve the metadata each time, only when it is necessary.  I don't work for FreeBSD so I'm not privy to how much benefit but it would definitely reduce load.
> 
> Of the two the significant benefit is seen on the client side.


How many times per day do you run freebsd-update? It's actually meant to be run every few weeks or so, 1 minute or 100 ms it does not make any practical difference. Also, if you run it frequently it's actually faster because it has to apply only few patches. Try it!


----------



## jbo (Oct 1, 2021)

roccobaroccoSC said:


> How many times per day do you run freebsd-update? It's actually meant to be run every few weeks or so, 1 minute or 100 ms it does not make any practical difference. Also, if you run it frequently it's actually faster because it has to apply only few patches. Try it!


This is why I initially asked about the OP's motivation for this. A response was provided by OP:



tux2bsd said:


> It isn't about "speed", it is about reducing unnecessary inefficiency. It's that simple.
> 
> My underlying goal was to get `freebsd-update fetch` be more efficient *when there are no updates*. Originally I proposed a grandiose idea and that was just an idea (grandiose in that I never had any expectation anyone else would come to the party).
> 
> ...



While I overall agree with roccobaroccoSC (hence I asked the question initially) I do think that OP's point is valid. Might just be a fun project for the sake of personal satisfaction - with the potential of leaving a mark if anything valuable comes out of it!


----------

