# Damned Upgrades!



## Profighost (Aug 20, 2020)

Anybody knows, upgrades barely improve anything but only make things worse - almost everytime something is not working anymore properly after an update - where is the sense in that? (I could use Windows if I want that #'*+$%&§/$'!!!)

I just did a "pkg upgrade", now my keyboard ain't working anymore. It actually reacts, but if I press a key e.g. something like 0$%&/(/ appear or I switch to another desktop, virtual console or so - X and xterm is doing unexpected, unwanted and not funny things - I don't have no shell!! That is NOT FUNNY!. And that is only what I figured out so far yet.
(fyi I need to use ubuntu from another machine to come here, because my FreeBSD is not working anymore since those amazing upgrades...#*§&!"§#!!!)

How can I downgrade again to the former stable, properly working state? Is there anything like "redo all upgrades" or "fall back to"?

Maybe at least it's only one package that disturbs the keyboard - but how can I figure that out? (remember: I cannot do any fancy things because my system only works very rudimentary at the moment!)

Is there a list/logfile which packages have been upgraded, and where do I find it?

thanks in advance


----------



## Mjölnir (Aug 20, 2020)

`zfs rollback [-rRf] <snapshot>`
Consider to file in a request (via bug reports, see @top of this forum) for ports-mgmt/pkg that the following would be reasonable:

add a command line switch to take a snapshot before upgrades
add a command `pkg history` à la `zpool history`
add a command `pkg rollback` à la `zfs rollback`


----------



## kpedersen (Aug 20, 2020)

I can't quite help you with your current issue but as someone who also dislikes continuous updates I can certainly recommend downloading *all* packages from the FreeBSD pkg mirror (~40 gigs) and making a snapshot in time. I do this around once a year and upgrade in a controlled manner.

Mostly because the problem is, even if you manage to downgrade that package, if you try to install another package, it may rely on a newer version of a dependency and upgrade it as part of the installation of that original package.

Disclaimer: This may or may not be bad advice


----------



## SirDice (Aug 20, 2020)

How long ago since you updated? It sounds like you're hitting the EVDEV change. 



Profighost said:


> Is there a list/logfile which packages have been upgraded, and where do I find it?


Look in /var/log/messages.


----------



## Profighost (Aug 20, 2020)

Thanks for your advices, guys.

After the first anger cooled off, there are several choices (at least it's FreeBSD: there is always a way  - and you added some additional ones. Thanks!

mjollnir Please don't shout at me because of my stupidity - I actually run a zfs, but didn't made a snapshots ... for the next time, I will keep that in mind; it's a really good advice, thanks!

kpedersen
app. 40GB are not that much ... to have a package repository in the background could be a good idea, thanks.

SirDice, well guessed,Sir... a long time ago (a couple of months{I know this could cause problems, because inconsisties can occur - but until now I had no serious ones...})... 
I have no problems with upgrades in general; especially network related stuff needs to be at current state to be safe; but I simply don't like it when the look and feel of software is changed (I run fvwm - simple, clear, easy, all one needs, nothing superfluid, (no xml-config-file!!{people who do this should be tarred and feathered!}) and it stays as it is - heaven! - ...."and keep you sticky fingers of MY settings, dam&%$)'!!!"...

First I try to fix/reinstall the xkeyboard (xkbd?) - as far as I figured it out, that's the point that crashed - since in console I do have a working keyboard.

EVDEV change?


----------



## Emrion (Aug 20, 2020)

Profighost said:


> EVDEV change?


Look at this thread: https://forums.freebsd.org/threads/new-xorg-switch-from-devd-to-udev.74144/
It's just some settings to do.

From my personal experience, more often you upgrade all packages, less you will have problem.

And, actually, if you have a zfs on root installation, create a Boot Environnement before each update / upgrade. See sysutils/beadm. Search on the forum if you don't know enough about BE.


----------



## Mjölnir (Aug 20, 2020)

Emrion said:


> Look at this thread: https://forums.freebsd.org/threads/new-xorg-switch-from-devd-to-udev.74144/
> It's just some settings to do.
> 
> From my personal experience, more often you upgrade all packages, less you will have problem.


Yes.


> And, actually, if you have a zfs on root installation, create a Boot Environnement before each update / upgrade. See sysutils/beadm. Search on the forum if you don't know enough about BE.


No! beadm(8) (& I guess also bectl(8)?) only does a recursive snapshot below zpool/ROOT, so /usr/local (where the ports & packages live) are not snapshot'ed (*).  You have to do that yourself, or automated periodically by some port (sysutils/*).
EDIT *: /usr/local & /var/db/pkg are not _necessarily_ snapshot'ed; see below.


----------



## scottro (Aug 20, 2020)

I'm going to say use beadm, not bectl right now. There seems to be a bug in bectl which is supposed to be fixed, but affected me recently. See this thread from Dec 2019-Jan 2020 (and I added to it today, as I ran into it)

https://forums.freebsd.org/threads/cannot-destroy-a-boot-environment.73296/

Anyway it's really easy. Before doing your upgrade, make a beadm snapshot. (Install beadm from package or port it's about 15K). 
Before your upgrade


```
beadm create workinginstall
```
Then do your upgrade. If it goes south, you will be able to pick the workinginstall from the menu (option 7, I think). 
Or if  you can get into the system but it's not working right, then just 
	
	



```
beadm activate workinginstall
```

I should add I also ran into this, an upgrade after a long period, where mouse and keyboard didn't work. I think I fixed it by adding xf86-input-libinput and possibly xf86-input-evdev.   (For what it's worth, in Fedora Linux, right now, all you should need is xorg-x11-drv-libinput, you don't even need evdev with Fedora, and quite possibly other versions of Linux).  FreeBSD is using evdev, you can look at the 20200320 entry in /usr/ports/UPDATING


----------



## Emrion (Aug 20, 2020)

mjollnir said:


> No! beadm(8) (& I guess also bectl(8)?) only does a recursive snapshot below /ROOT, so /usr/local (where the ports & packages live) are not snapshot'ed. You have to do that yourself, or automated periodically by some port (sysutils/*).


I ever reversed some bad upgrades with BE (including xorg-server-1.20 for a time). I ain't a specialist of the thing, but it works in all cases.


----------



## Alain De Vos (Aug 20, 2020)

In general it's easier to do a frequent upgrade if it is no server.


----------



## a6h (Aug 20, 2020)

Next time even for pkg(8): `cat /usr/ports/UPDATING`


----------



## scottro (Aug 20, 2020)

Oh yes, definitely. I should have mentioned that.  Thanks vigole


----------



## a6h (Aug 21, 2020)

scottro Reading /usr/ports/UPDATING  saved me last January. I postponed update for weeks, after xorg problem (devd/udev) in January.


----------



## richardtoohey2 (Aug 21, 2020)

This:





Emrion said:


> From my personal experience, more often you upgrade all packages, less you will have problem.


And this:





vigole said:


> Next time even for pkg(8):  cat /usr/ports/UPDATING


I don't use FreeBSD on desktop (yet!) but if I let upgrading a server go too long, then instead of having one or two quirks/extra upgrade steps to watch out for, I might end up with four or five, and it might be difficult to work out which one caused any issue that _might_ arise (usually it's all good!)

And UPDATING (and MOVED) good places to check first - anything a bit scary in there, postpone updates or try on a test VM/machine.


----------



## Mjölnir (Aug 21, 2020)

Emrion said:


> I ever reversed some bad upgrades with BE (including xorg-server-1.20 for a time). I ain't a specialist of the thing, but it works in all cases.


By default, /usr/local & /var/db are on the root dataset (zpool/ROOT/default).  But an innocent user/admin can override that.  So yes, if you did not put /usr/local & /var/db on it's own dataset, backing up a BE will be enough to rollback.  _BUT_, e.g. an admin may decide to have a mostly-read-only system, and thus put /var on it's own dataset.  Then a standard BE backup will fail to provide a proper rollback, because the contents of /usr/local (rolled back) do not match the contents of /var/db/pkg (updated, not rolled back).


----------



## Emrion (Aug 21, 2020)

mjollnir said:


> By default, /usr/local & /var/db are on the root dataset (zpool/ROOT/default).  But an innocent user/admin can override that.  So yes, if you did not put /usr/local & /var/db on it's own dataset, backing up a BE will be enough to rollback.  _BUT_, e.g. an admin may decide to have a mostly-read-only system, and thus put /var on it's own dataset.  Then a standard BE backup will fail to provide a proper rollback, because the contents of /usr/local (rolled back) do not match the contents of /var/db/pkg (updated, not rolled back).


So, that works unless you made a change that ruins the interest of BEs (which isn't smart, for the less...).


----------



## Mjölnir (Aug 21, 2020)

Emrion said:


> So, that works unless you made a change that ruins the interest of BEs (which isn't smart, for the less...).


I gave an example that is not so uncommon.  If you want to secure your system, it's reasonable to set it up as _mostly-read-only_.  Of course, it is possible to place /var under zpool/ROOT/default, but the default is to have zpool/var.  I wanted to point out some possible pitfalls: The defaults do not allow to secure your system by making it _mostly-read-only_ out-of-the-box.


----------



## a6h (Aug 21, 2020)

richardtoohey2 said:


> anything a bit scary in there, postpone updates or try on a test VM/machine.


Good approach. I always have 3 VM/FreeBSD for testing: a RELEASE from ISO,  an updated STABLE and a testing CURRENT. And doesn't matter where: bhyve, vbox,, vmware, etc.


----------



## Emrion (Aug 21, 2020)

mjollnir said:


> I gave an example that is not so uncommon.  If you want to secure your system, it's reasonable to set it up as _mostly-read-only_.  Of course, it is possible to place /var under zpool/ROOT/default, but the default is to have zpool/var.  I wanted to point out some possible pitfalls: The defaults do not allow to secure your system by making it _mostly-read-only_ out-of-the-box.


That wasn't your initial remark about this... You thought the BE system isn't working to recover from pkg or port upgrade in general.


----------



## Mjölnir (Aug 21, 2020)

Emrion said:


> That wasn't your initial remark about this... You thought the BE system isn't working to recover from pkg or port upgrade in general.


I did not _think_, but remembered that I ran into this issue on a previous installation where I had separate /var and /usr/local datasets.  I edited my post accordingly.  Yes, on a default installation, creating a BE before an upgrade is sufficient.  A recursive snapshot on the root zpool(s) works with any setup, and usually it doesn't hurt since it can be deleted after a few days.


----------



## Profighost (Sep 7, 2020)

"I am back, baby!" - Problem solved
Or, to be more precisely: worked around aka new installation

First things first: I like to thank all participants in this thread for advices, tipps and the discussion (I neither wanted nor expected to start such, but it seems there are issues or at least points worth to be discussed and be looked closer at (that's what I in fact did - thanks!))

I could not solve the problem in "repairing/fixing" my system, didn't figured out where or what went wrong with my x-keyboard (xkbd seemed to be okay...), but I also figured out I had several additonal probs like my x-windows was slowed down badly... - however.. since I knew I had some kind of patchworksystem made as a wild mix of compiled ports and installed packages (yes, yes,.. I know) I decided to make a fresh start, new HDDs, backups, install...- the procedure.
This took me app. 5..6 hours, now having a fresh 12.1, up-to-date, running smoothely, because after all I had learned one or two things about FreeBSD (besides the consent again it's a exemplary OS), reinstallation was no dirty (...if I think back to old Windows-Times (a Win98 installation cost a whole weekend!  - that was when I started to make complete HDD-copies as back-ups and fallbacks) but in fact a quick job - if one have the old system, files and a bit knowledge, what where...

Even if the hints did not directly helped to solve my originally problem directly you guys gave me lots of ideas to think of and do it better from now on and in the future.

The next thing is, to think about how keep on (pkg upgrades in cron?....).
snapshots are for sure something I will not overkip anymore (That's something I will do in a cron job.)
I also will take a closer look to some other things that came up in this thread

Thanks! Thanks a lot!


----------



## Mjölnir (Sep 7, 2020)

Don't reinvent the wheel, instead look into /usr/ports/sysutils for a variety of ports to choose from for _automagic_ ZFS snapshots.
`psearch -s snapshot` or `portfind -dr snapshot` will give an overview. `pkg install` one of them or both.


----------



## astyle (Sep 5, 2021)

I hope this is not necroposting, but still... The fact that it's so incredibly difficult to downgrade just one port if you're having issues with an upgrade is exactly why I've been working so hard all this time to understand ZFS and bectl... knowing how and when to use those tools will help save DUMP TRUCKLOADS of time and data. Reading the manuals led me to conclude that pkg and ports work only one way when installing. And un-installing a problematic upgrade requires a LOT of effort and know-how. With FreeBSD, it's at least possible to figure that out, and maybe even automate that, even if it takes different tools. But still, here's a basic list:

Snapshot a working system with ZFS.
Do your upgrade # *Lots of ways to fine-tune that*, with Poudriere, bectl, testing and what not.
If you like your upgrade, snapshot your system again.
If you don't, just roll back to the snapshot created in Step 1.


----------



## grahamperrin@ (Sep 5, 2021)

Mjölnir said:


> … beadm(8) (& I guess also bectl(8)?) only does a recursive snapshot below zpool/ROOT, …



I imagine that what's at the head of <https://forums.FreeBSD.org/threads/81525/> is the result of option `-r`


----------

