# I need help understanding pkg upgrade



## pschmehl (Aug 13, 2021)

I'm tired of running portmaster -ad and taking two or three days to build all the ports, resolving so many issues that pop up along the way. But pkg scares the hell out of me. I manage two servers. I have no backup or test systems to try things out before going into production. Both of these systems are live on the internet and in production. So, it's critical that I get updates right.

But, when I run pkg upgrade, it wants to uninstall ports that I absolutely need.

For example, one server is the mail server. The other is the webserver. Since web is more critical, and mail can be down for five days before we start losing mail, I always run upgrades on the mail server first. That helps me iron out any bugs, discover which major ports (like perl, php, python, etc.) need to be upgraded (and upgrade all the dependent ports) before moving to the web server and risking taking it down.

Tonight, I ran pkg upgrade, and it wanted to uninstall postfix. Why would it do that? So, I locked postfix, and ran it again. Then it wanted to uninstall 10 apps. I really don't understand what's going on. Can someone explain it to me?

This is the first run. Obviously, I don't want to uninstall postfix, so I canceled the upgrade. But why on earth does pkg want to uninstall postfix? That's the most important app on this server. I also run mailman on this server, and pkg wants to reinstall it because its options have changed. How can I find out what options have changed so I can determine if I have to lock it as well?


> Installed packages to be REMOVED:
> postfix: 3.5.6,1





> New packages to be INSTALLED:
> freeglut: 3.2.1
> jpeg-turbo: 2.0.6
> libGLU: 9.0.2_1
> ...


----------



## grahamperrin@ (Aug 13, 2021)

`pkg info -x courier`

Is mail/courier installed?



pschmehl said:


> uninstall postfix. Why would it do that?



Try <https://www.freshports.org/mail/postfix/> ▶ *Conflicts*


----------



## richardtoohey2 (Aug 13, 2021)

So just to be clear - these are machines that USED to be using ports/portmaster and you want to try and move to binary packages from that old set-up?

They are not clean set-ups with binary packages already?


pschmehl said:


> I manage two servers. I have no backup or test systems to try things out before going into production. Both of these systems are live on the internet and in production. So, it's critical that I get updates right.


Save yourself a world of pain and change this situation.

I don't have the answer for you, but I'm going through the same migration process - but I'm leaving old machines on portmaster, and starting (on test machines and then a new production machine) using binary packages.  Trying the migration live is a recipe for maxium stress.  Find an old machine, cheap VM, whatever - there must be something you can experiment on!

And sort out backups.  You'll feel better for it.


----------



## astyle (Aug 13, 2021)

OP, this is part of the experience. Even big companies are highly reluctant to upgrade their Linux servers precisely because of the issues you describe. So they end up stuck in the morass "I am so scared of upgrading, if I don't do it right, everything will break!" 
I'm still in the process of setting up Poudriere - but even lining up the details so that  only KDE gets properly upgraded (just fresh tarballs downloaded, compiled with same settings as before) - even that is a lengthy process for me.

One suggestion I have - Get another server, install up  to date stuff on it, and then replicate the functionality you need. yeah, that will take more time, but that's a safer way to do it. I did exactly that on my first IT job at a small shop. The shop's boss didn't want to buy another server, though, so I scrounged up a discarded PC from the back room, and set it up as a dedicated email server running Dovecot and FreeBSD 6.0 (yeah, it was a long time ago).  The move was very clean. And, I used pkg at the time.


----------



## pschmehl (Aug 13, 2021)

grahamperrin said:


> `pkg info -x courier`
> 
> Is mail/courier installed?
> 
> Try <https://www.freshports.org/mail/postfix/> ▶ *Conflicts*


Yes, I've been using courier with postfix for years. But I only use the imaps part along with authlib. Is that now a problem?


```
]# pkg info -x courier
courier-authlib-base-0.71.0
courier-authlib-userdb-0.71.0
courier-imap-5.0.11,2
courier-unicode-2.1_2
```


----------



## pschmehl (Aug 13, 2021)

richardtoohey2 said:


> So just to be clear - these are machines that USED to be using ports/portmaster and you want to try and move to binary packages from that old set-up?
> 
> They are not clean set-ups with binary packages already?
> 
> ...


Yes, these are machines that have always used ports/portmaster. They are not clean set-ups with binary packages. The reason for that is that pkg doesn't install the correct options for certian ports that I use. But 99% of the ports installed could use packages with no problem. In fact, it's been my practice, when a port fails to upgrade with portmaster, to install the package instead, where possible.

I don't understand all the talk about not mixing binary packages with portmaster-built packages. What is the difference? Portmster actually builds packages.

Backups are a separate issue. There is no money for software. This is a strictly volunteer, bandaid and bailing wire setup.


----------



## pschmehl (Aug 13, 2021)

astyle said:


> OP, this is part of the experience. Even big companies are highly reluctant to upgrade their Linux servers precisely because of the issues you describe. So they end up stuck in the morass "I am so scared of upgrading, if I don't do it right, everything will break!"
> I'm still in the process of setting up Poudriere - but even lining up the details so that  only KDE gets properly upgraded (just fresh tarballs downloaded, compiled with same settings as before) - even that is a lengthy process for me.
> 
> One suggestion I have - Get another server, install up  to date stuff on it, and then replicate the functionality you need. yeah, that will take more time, but that's a safer way to do it. I did exactly that on my first IT job at a small shop. The shop's boss didn't want to buy another server, though, so I scrounged up a discarded PC from the back room, and set it up as a dedicated email server running Dovecot and FreeBSD 6.0 (yeah, it was a long time ago).  The move was very clean. And, I used pkg at the time.


The problem with this suggestion is that I'm trying to do LESS work, not MORE. I'm 74 and retired. I don't get a dime for this work. The last thing I want to do is scrounge up a cheap server and try to replicate the setup we have. There are no old PCs laying around.


----------



## grahamperrin@ (Aug 14, 2021)

pschmehl said:


> not mixing binary packages with portmaster-built packages.



Defocusing from portmaster: if you build packages for yourself, then you should avoid mixing with packages from _quarterly_. 

So, for example, I have: 


```
% uclcmd get --file /etc/pkg/FreeBSD.conf FreeBSD.url
"pkg+http://pkg.FreeBSD.org/${ABI}/latest"
%
```

– _latest_.



pschmehl said:


> … Is that now a problem?
> 
> 
> ```
> ...



From what's listed, those four, I don't see a conflict. 

Consider the other possible conflicts.


----------



## pschmehl (Aug 14, 2021)

grahamperrin said:


> Defocusing from portmaster: if you build packages for yourself, then you should avoid mixing with packages from _quarterly_.
> 
> So, for example, I have:
> 
> ...


So, if I'm understanding you correctly, the problem with mixing portmaster-built ports and pkg-installed ports is that the pkg-installed ports are possibly older than the portmaster-built ports?


----------



## astyle (Aug 14, 2021)

pschmehl said:


> The problem with this suggestion is that I'm trying to do LESS work, not MORE. I'm 74 and retired. I don't get a dime for this work. The last thing I want to do is scrounge up a cheap server and try to replicate the setup we have. There are no old PCs laying around.


In that case, try to learn about ZFS, creating snapshots and rolling them back. And also, ZFS boot environments. They make it really easy to recover from mistakes. FWIW, I'm in the process of teaching myself to do that.


----------



## richardtoohey2 (Aug 14, 2021)

So why exactly do you want to move from portmaster?  If you just want to keep this fragile set-up going and there's no time or budget for anything else - carry on with portmaster.

Why is it taking 2 days, though?  Are the machines very old/limited?  Or you have a bucket-load installed?  Doesn't look like a huge list from your first post.

I don't think there's going to be an easy way to go from what you have to a trouble-free low maintenance set-up without spending time & money and you don't appear to have either.  So you're a bit stuck.


----------



## pschmehl (Aug 14, 2021)

astyle said:


> In that case, try to learn about ZFS, creating snapshots and rolling them back. And also, ZFS boot environments. They make it really easy to recover from mistakes. FWIW, I'm in the process of teaching myself to do that.


I'm going to be frank with you. ZFS is beyond my understanding or capability. I simply don't understand how it works. (And I've read a lot about it.)


----------



## Vull (Aug 14, 2021)

One has to weigh the cost of having a solid backup and recovery strategy against the cost of not having it, factoring in the cost of allowing the whole system to fail irrecoverably and become unusable. Everything has a cost.


----------



## pschmehl (Aug 14, 2021)

richardtoohey2 said:


> So why exactly do you want to move from portmaster?  If you just want to keep this fragile set-up going and there's no time or budget for anything else - carry on with portmaster.
> 
> Why is it taking 2 days, though?  Are the machines very old/limited?  Or you have a bucket-load installed?  Doesn't look like a huge list from your first post.
> 
> I don't think there's going to be an easy way to go from what you have to a trouble-free low maintenance set-up without spending time & money and you don't appear to have either.  So you're a bit stuck.


Every time I run portmaster (which is infrequently) I run into problems with ports that won't build for one reason or another. Then I have to google to chase down the problem and get it fixed so I can find the next port that won't build. The equipment isn't that old, but it's definitely not new. The mail server is a dual core opteron with 8 MB of memory and 125GB hard drive. The web server is newer.

pkg is much faster, but you're stuck with the options someone else chose.

When I run portmaster, I use screen. I get it started and watch it for errors until I go to bed. Then the next day I reattach and I do some more. Finally, but the third day I'm usually done. I'm just looking for a quicker way to get up to date without blowing up the main apps I run. PHP is always a problem, because pkg uninstalls extensions that I need.

Frankly, I'm beginning to think it's time to hand all this off to a hosting company and let them handle all the backend work. I just hate leaving FreeBSD because it works so well.


----------



## pschmehl (Aug 14, 2021)

Vull said:


> One has to weigh the cost of having a solid backup and recovery strategy against the cost of not having it, factoring in the cost of allowing the whole system to fail irrecoverably and become unusable. Everything has a cost.


Well, sometimes people can't afford to go the dentist, so their teeth rot. Not everyone can afford the best of everything. For backups, I use scripts that upload the files to Dropbox and delete them after seven days. So, I always have one week's worth of the most recent backups. Yet, it's not ideal, but it also doesn't cost much.


----------



## Vull (Aug 14, 2021)

pschmehl said:


> Well, sometimes people can't afford to go the dentist, so their teeth rot. Not everyone can afford the best of everything. For backups, I use scripts that upload the files to Dropbox and delete them after seven days. So, I always have one week's worth of the most recent backups. Yet, it's not ideal, but it also doesn't cost much.


Sounds to me like a backup and recovery strategy. Here's hoping your upgrade goes well.


----------



## VladiBG (Aug 14, 2021)

Keep using ports only. Make a full backup, restore this backup on your computer inside virtual machine and test the upgrade process there while reading /usr/ports/UPDATING and taking notes of everything during the test update. Then perform the actual update on your production server.


----------



## mark_j (Aug 14, 2021)

I feel your pain. At work we have a build server and our own package repository and builds can very take a long time on a system with 16 cpus!

I also know that the ports system is broken. How to fix it, I do not know, it's above my station and frankly I don't care.

Let me give you an example: I recently wanted to build doxygen and so issued make config-recursive. After about 20 minutes of answering questions for over 50 ports I abandoned it when it said it needed to build gcc!! Seriously, it wanted to build gcc!. This is by no means an isolated or unique incident.

 I then downloaded doxygen, fixed two things with the CMakesfiles.txt to get it to build  and built it in 15 minutes, no 50+ dependencies except flex & bison & maybe 1 other. (This was on an Armv7 device as well).

So, I'm convinced on this: ports is broken. Use pkg only. Always!


----------



## Menelkir (Aug 14, 2021)

mark_j said:


> I feel your pain. At work we have a build server and our own package repository and builds can very take a long time on a system with 16 cpus!
> 
> I also know that the ports system is broken. How to fix it, I do not know, it's above my station and frankly I don't care.
> 
> ...


Probably one of the deps wasn't able to build without gcc for whatever reason (some packages have this issue, being there, done that) or maybe a more recent version is able to build without it and the maintainer didn't tested again if it builds without gcc.


----------



## mark_j (Aug 14, 2021)

Yes probably. It still is insane. A small build taking 15 minutes would quite possibly take days using ports. Use pkg.


----------



## VladiBG (Aug 14, 2021)

You are forced to use ports when you need a custom option which is not build by default configuration of the port which is valid only for prebuild pkg.


----------



## drhowarddrfine (Aug 14, 2021)

mark_j If you are building from ports then, if gcc is a dependency, don't be surprised when it needs to be built. It doesn't mean ports is broken. I also question your need to answer questions when you were using `config-recursive`.


----------



## grahamperrin@ (Aug 14, 2021)

pschmehl said:


> … a strictly volunteer, bandaid and bailing wire setup.





pschmehl said:


> … no old PCs laying around.





pschmehl said:


> … 125GB hard drive. …



Could a small drive be added?

A 16 GB USB flash drive might be ample, although in your volunteer scenario you might find someone to donate an old spare hard drive (small HDDs are throwaway items, these days).

You could:

use gpart(8) or whatever to create a single ZFS partition e.g. /dev/da0p1
create a ZFS pool e.g. `zpool create portsdrive da0p1`
install ports-mgmt/poudriere-devel
change three lines in a preconfigured poudriere.conf
`poudriere ports -c`
`poudriere jail -c -j thirteen -v 13.0-RELEASE`
`poudriere ports -u`
`poudriere bulk -v -j thirteen mail/courier-imap`
That's `mail/courier-imap` as one example; build as few or as many ports as you want. The speed with which poudriere can build things into a repository will be a breath of fresh air to you.

Steps (1)–(6) can be one-off, need not be repeated. Step (7) whenever you want to update the ports tree.

For step (4), these three lines in your /usr/local/etc/poudriere.conf:


```
ZPOOL=portsdrive
DISTFILES_CACHE=/usr/ports/distfiles
PACKAGE_FETCH_BRANCH=latest
```

Your /usr/local/etc/pkg/repos/poudriere.conf:


```
{
    "poudriere": {
        "url": "file:///usr/local/poudriere/data/packages/thirteen-default",
        "enabled": true
    }
}
```

As a side note, that's ZFS at its simplest; no learning curve. (Just be careful about specifying the partition to give to ZFS. `da0p1` above is just an example.)

*PS* corrected a handful of typos above.




pschmehl said:


> … pkg-installed ports are possibly older than the portmaster-built ports? …



True. Compare, for example, the _latest_ and _quarterly_ columns at <https://www.freshports.org/lang/gcc/#packages>.

Whilst things such as pkg-install(8) are reasonably good at working with dependencies, mixing latest with quarterly *will* make things unnecessarily difficult. If you're accustomed to using _any_ utility to build and install from ports – from latest – then (on the same computer) constraining /etc/pkg/FreeBSD.conf to quarterly is unlikely to add value; change it to latest.


----------



## jmos (Aug 14, 2021)

pschmehl said:


> Backups are a separate issue. There is no money for software.


Backup is a concept, not a piece of software. There's nothing more needed than disk space on another machine. Same for a virtual machine to play araound with those updates (and maybe step back an try another way again). Otherwise you're playing russian roulette with a live systems - may cost much more than some time and disk space.

I've seen far to many companies with not the smallest two-digit million euro turnovers without backups and without development machine losing complete servers. Said "hey, we have raid", and the technician took out the wrong of the two remaining disks after noticing all other disks died a long time ago (and the last two also sounding … weird). Or crashing raid controllers - can be fun, too. And I never get the "why". No backup, no mercy.

You've (or your company) decided to go high risk. So you're even going high risk on updates.


----------



## pschmehl (Aug 14, 2021)

mark_j said:


> Yes probably. It still is insane. A small build taking 15 minutes would quite possibly take days using ports. Use pkg.


Mark, I would use package if it would install the bits I need. It doesn't.


----------



## pschmehl (Aug 14, 2021)

jmos said:


> Backup is a concept, not a piece of software. There's nothing more needed than disk space on another machine. Same for a virtual machine to play araound with those updates (and maybe step back an try another way again). Otherwise you're playing russian roulette with a live systems - may cost much more than some time and disk space.
> 
> I've seen far to many companies with not the smallest two-digit million euro turnovers without backups and without development machine losing complete servers. Said "hey, we have raid", and the technician took out the wrong of the two remaining disks after noticing all other disks died a long time ago (and the last two also sounding … weird). Or crashing raid controllers - can be fun, too. And I never get the "why". No backup, no mercy.
> 
> You've (or your company) decided to go high risk. So you're even going high risk on updates.


It's not a company. It's a small group of hobbiests, all volunteer, that subsists on donations from its participants. The $477 quarterly that they pay for hosting and DNS services is just about all they can afford. I'm trying to convince them to go to a dedicated solution, but that ups their costs by one-third. In case anyone is interested, the website is https://www.stovebolt.com/.

I worked professionally in IT (security) for 20 years. I know all about robust backup systems and the cost of not backing up. I've seen professors lose twenty years of data because they didn't have backups. I'm doing the best that I can with an extremely limited budget.

In the fifteen years that I've been doing this, I've had to restore from backup once. We lost maybe half a day's worth of traffic on the forum. The domain owners are well aware of the limitations of their system and quite prepared to have it all go away with no recourse. When I have fretted about being down for a couple of days, they had said, Don't worry about it. Take all the time you need.

IF I lost a server, they would have to run a fundraiser to pay for it. Then I would have to install a fresh FreeBSD and all the apps, and then restore the data from the backups I have. It wouldn't be pretty, and it would take a lot of my time, but it's doable in my current setup. The only things we would lose are whatever changed since the last backup.

I get what you're saying. It doesn't apply to me.


----------



## pschmehl (Aug 14, 2021)

grahamperrin said:


> Could a small drive be added?
> 
> A 16 GB USB flash drive might be ample, although in your volunteer scenario you might find someone to donate an old spare hard drive (small HDDs are throwaway items, these days).
> 
> ...


Thanks for this, Graham. You've given me an idea. I could buy a USB flash drive twice the size of the mail server's hard drive for less than $25 and use that to do what you're suggesting. If it craps out, I just buy another drive. Hmmmm...


----------



## Vull (Aug 14, 2021)

pschmehl said:


> It's not a company. It's a small group of hobbiests, all volunteer, that subsists on donations from its participants. The $477 quarterly that they pay for hosting and DNS services is just about all they can afford. I'm trying to convince them to go to a dedicated solution, but that ups their costs by one-third. In case anyone is interested, the website is https://www.stovebolt.com/.
> 
> I worked professionally in IT (security) for 20 years. I know all about robust backup systems and the cost of not backing up. I've seen professors lose twenty years of data because they didn't have backups. I'm doing the best that I can with an extremely limited budget.
> 
> ...


Nice site. Nice forum. May I suggest you start a topic on that forum, asking for one of the forum members to donate an old inexpensive used laptop, or other i386 compatible box, just for forum maintenance purposes? Then you might be able to postpone your upgrade just long enough to take your time in replicating the software configuration(s) you need on this "new" machine. You might even get lucky and wind up getting more than one such machine; lots of people might have old hardware lying around that they've replaced and aren't using anymore. Your forum members seem like very resourceful people.

It wouldn't need to be a server-grade piece of hardware, but rather, just good enough to serve as a software configuration model, and possibly (but not necessarily), just good enough that it might serve as a temporary replacement in a pinch, in case one of your live server upgrades fail.

In such a case, or any case of live server hardware or software failure, you might then be able to restore your most recent backup on the model machine, and go live with it, while you could then take your time and duplicate the software configuration you've just prepared and documented on the model machine on the primary server hardware. It would alleviate a lot of the stress and uncertainty you're dealing with now.

In future, once you have the software config on the model machine synch'ed up with the config on the live machine, you'd be able to test future software upgrades on the model machine without risking the stability of the live server or servers. And, if the hard drive on the model machine is large enough, you might even be able to use it as a multi-boot system, to model all of your server configurations. It seems like a lot of work, I know, but it also seems like you're already working overtime dealing with this situation right now, plus all the stress and uncertainty that goes with it. Just my humble opinion, take it for whatever you think it's worth.


----------



## grahamperrin@ (Aug 14, 2021)

pschmehl said:


> Thanks for this,



You're welcome, NB there were some typos in my original post. Now corrected …

… hopefully nothing else wrong there <https://forums.freebsd.org/goto/post?id=527130> but corrections are welcome.

*PS* there's an assumption of FreeBSD 13.0-RELEASE. 

Also, I didn't mention port options. For a simple setup with a single poudriere jail, as far as I know you can put most of what's required in
/usr/local/etc/poudriere.d/make.conf – here's mine: 


```
# <https://github.com/freebsd/poudriere/issues/867>
MAKE_JOBS_NUMBER=3

# <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252099#c5>
# <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252099#c18>
WITH_DEBUG_PORTS += multimedia/webcamd sysutils/bsdisks

ICA_CERTS=QuoVadisEuropeEVSSLCAG1.crt

# <https://forums.FreeBSD.org/threads/71438/post-517873>
LICENSES_ACCEPTED += commercial

WITHOUT_LLVM_TARGET_ALL=
```

(Probably not an entirely sane example … I don't know where I got the notion of a debug build of sysutils/bsdisks because <https://www.freshports.org/sysutils/bsdisks/#config> there's no such option.)


----------



## astyle (Aug 14, 2021)

mark_j said:


> I feel your pain. At work we have a build server and our own package repository and builds can very take a long time on a system with 16 cpus!
> 
> I also know that the ports system is broken. How to fix it, I do not know, it's above my station and frankly I don't care.
> 
> ...


I build gcc early on. And doxygen - only after graphviz and ruby. Learned my lesson on how to do it right, even if it takes time. Well, I do have the hardware, too - the Ryzen 5 1400 is no slouch.


----------



## mark_j (Aug 14, 2021)

drhowarddrfine said:


> mark_j If you are building from ports then, if gcc is a dependency, don't be surprised when it needs to be built. It doesn't mean ports is broken. I also question your need to answer questions when you were using `config-recursive`.


Gcc is not a dependency, that's the problem. At least not directly. With a completely new system, all that's required is cmake, a c++ compiler, flex and bison and, sigh, python. So if I built python or flex or bison or cmake via ports it would eventually lead to needing to build gcc10, possibly just to build one module used by one of the aforementioned applications.

It's broken.

Use pkg.


----------



## astyle (Aug 15, 2021)

mark_j said:


> Gcc is not a dependency, that's the problem. At least not directly. With a completely new system, all that's required is cmake, a c++ compiler, flex and bison and, sigh, python. So if I built python or flex or bison or cmake via ports it would eventually lead to needing to build gcc10, possibly just to build one module used by one of the aforementioned applications.
> 
> It's broken.
> 
> Use pkg.


you only end up with gcc if you build webcamd, which has a dependency on some Linux libs. Linux libs depend on being built by gcc...  That's why I build GCC as early as practical. No, ports are not broken, but they do have the risk of getting into circular deps if you specify too many deps in your makefiles. Took me a few tries to figure out how to break out of that vicious circle - and it worked, because I was paying attention.


----------



## mark_j (Aug 15, 2021)

It matters not. The fact you have to build GCC anyway is just ridiculous. 
Unless you're compiling the linux kernel which uses some hacky-shit only provided by GCC, there's no reason for it to be even used. Clang does it all. If it doesn't then hack the code to fix the hack and stop being a lazy port maintainer. If GCC is providing some interface to Fortran or something then yes it's a valid requirement. In this case, as I said, use pkg. 
That's what I am trying to re-inforce to the OP. Dump ports, it's a slow grind into boredom. Recursively building tool after tool with potential for failure at any point, why bother? It's a waste of your time.

I cannot recall what it was a few weeks ago, but experimenting with a port that suddenly wanted Clang10 on an installed system with Clang11. Get out!

Use pkg for everything and only where you cannot use the binary package, use the port and customise. This is an absolute on smaller computing power machines and those who cannot be bothered answering 1000 prompts. 

Do what you want if you have a build server; that's not the OPs issue.


----------



## pschmehl (Aug 15, 2021)

mark_j said:


> Do what you want if you have a build server; that's not the OPs issue.


Mark, if I'm understanding you correctly, you seem to be saying use pkg for everything you can, and build the custom ports yourself. Is that right?

Here's a couple of problems that I ran into over the past couple of days getting the mail server updated.

1) I had to install and configure Roundcube because Squirrelmail is EOL. So, I ran pkg install roundcube. Later, I was running portmaster, and it wanted to update roundcube. So, clearly, pkg is "behind" ports (for roundcube at least.)
2) I locked several important apps and than ran pkg upgrade. It upgraded/installed 121 packages. I then ran portmaster. It wanted to update a bunch more. (I have 349 ports installed on that machine. I don't even know why 2/3rds of them are needed.) So, there is obviously a difference between pkg and building ports with portmaster.

How do you reconcile that? I require certain php extensions for some of the stuff we're running. But pkg removes them because they're not on its list of options. How do I resolve that using pkg?

I'm open to new ideas. I just need to understand how it works and what the risks are.

For those who have expressed concern about backups, here's what I'm doing. I wrote a script that creates .tgz files from the bits that need to be backed up (including a mmddyyyy.all.sql file that backups up the mysql dbs), then writes them to /var/backup/. The filenames use the pattern mmddyyyy.filename.tgz. Then they are uploaded to my Dropbox folder. Each day, when the script runs, it deletes the previous day's file from /var/backup/ and the previous 7th-day file from Dropbox. This keeps the /var partition from filling up while keeping the most recent backup handy on the hard drive and keeps the previous 7 day's backups on Dropbox in case I have a disaster that requires an older backup.

Kludgy, I know, but it's the best I can do with no money for backup software. If you need me to restore something from last month or last year, you're out of luck.


----------



## grahamperrin@ (Aug 15, 2021)

grahamperrin said:


> assumption of FreeBSD 13.0-RELEASE.



Was that a false assumption?

pschmehl which version of FreeBSD do you use?



pschmehl said:


> 𡀦… pkg is "behind" ports (for roundcube at least.) …



Apparently _not_ behind for `roundcube-php74` on `FreeBSD:13:amd64`:






<https://www.freshports.org/mail/roundcube/#packages>


```
% grep -A 2 1.4.11 /usr/ports/mail/roundcube/Makefile
DISTVERSION=    1.4.11
PORTREVISION=   1
PORTEPOCH=      1
%
```


----------



## astyle (Aug 15, 2021)

OP: For backup software, you can always try something from ports: sysutils/bacula11-server. This is enterprise-grade stuff, available for free. Or, you can put together a patchwork of simple file copy utilities available in the base install, write a script to back everything up, and put it into cron so that your backup is on a schedule. All you need is enough disk space, and a willingness to study how those systems work. Just take a look at what FreeBSD even has to offer! Even uploading to DropBox if you want!

Remarks from mark_j are exactly the reason to NOT mix packages and ports. If you picked packages at the start, stay with them. If you picked ports, stay with them. The best way to handle upgrading this one port (that you needed compiled with your own custom options) is to have a separate build server that will build your package. Then, AFTER the package is built, you install it on the main machine. If that solution doesn't work, then I'm afraid you're stuck.


----------



## pschmehl (Aug 15, 2021)

grahamperrin, I'm running 11.4-RELEASE on both servers. I had a bad experience upgrading to 13.x-RELEASE on another server, so I've opted to stay with 11.4-RELEASE for now.


----------



## grahamperrin@ (Aug 16, 2021)

Thanks. I see mostly gaps for FreeBSD:11:amd64 at <https://www.freshports.org/mail/roundcube/#packages> but neither roundcube-php73 nor roundcube-php74 are _behind_ latest ports.


----------



## grahamperrin@ (Aug 16, 2021)

astyle said:


> … have a separate build server …



No need for separation with poudriere.


----------



## Deleted member 30996 (Aug 16, 2021)

pschmehl said:


> I'm tired of running portmaster -ad and taking two or three days to build all the ports, resolving so many issues that pop up along the way.


I've been using ports-mgmt/portmaster for years but never updated any programs unless portmaster pulled them in during a program build.

That came back to haunt me recently and I had a genuine mess consisting of over 120 ports to clean up that required the use of pkg and about 3 days for me to reslove.

I'm old too, not quite as old as you, but neither of us too old to learn from mistakes. Yesterday I used `portmaster -a` only a couple weeks after last using it. This time there were only 53 ports that needed updating. I started in the morning and was done before the day had passed.

The only program that caused a stop was updating multimedia/vlc and that was near the very end. When it stopped I ran `portsnap fetch update`. Then ran `portmaster -a` again to get a good look at what I was gong to have to do to set thing right.

During the time that had passed since morn the problem child file that caused portmaster to balk had been updated and portmaster listed it in the plan of what it was going to install. I scrolled up to restart the build with the same command initially used and was finished an hour or so later with all programs up to date.

I don't run a server, only laptops, so downtime only concerns me and this time used in what I see as wisely. If I was running a server I might update more frequently to keep downtime blocks to a minimum. But that's me.


----------



## pschmehl (Aug 16, 2021)

Trihexagonal, I try not to disrupt service operations any more than I have to, so I update infrequently. But there are some ports that take forever to update. llvm80, llvm90, and cmake are three that make me groan every time they show up on the list (which is often.) There are some others, but those three always take a very long time to build. I don't even know what those ports are needed for.


----------



## Tieks (Aug 16, 2021)

pschmehl said:
			
		

> I don't even know what those ports are needed for.



From pkg-descr: "The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.".

These are compilers. They are used for building ports, not for running ports. If you have to keep operations going while updating ports you might want to install these once and keep 'em. See `pkg help set` to mark these so that `pkg autoremove` does not remove these compilers.


----------



## astyle (Aug 16, 2021)

pschmehl said:


> Trihexagonal, I try not to disrupt service operations any more than I have to, so I update infrequently. But there are some ports that take forever to update. llvm80, llvm90, and cmake are three that make me groan every time they show up on the list (which is often.) There are some others, but those three always take a very long time to build. I don't even know what those ports are needed for.


devel/cmake has a truckload of dependencies, but once those are satisfied, it takes less than 10 minutes to compile (Well, I do have a Ryzen 5 1400, and it's surprisingly capable). I'd suggest NOT cleaning out those deps, either. Otherwise, ports will pull them right back in when it's time to upgrade something.


----------



## richardtoohey2 (Aug 16, 2021)

pschmehl said:


> But there are some ports that take forever to update. llvm80, llvm90, and cmake are three that make me groan every time they show up on the list (which is often.) There are some others, but those three always take a very long time to build.


Oh, yes, cmake is a beast (and there have been a good few updates and new versions in the last few months).  And llvm versions are x2 or x3 worse!  If you are pulling in llvm versions then generally it's worth taking the time to figure out what's pulling them in - there will be some build config option that you've ticked and if you don't need that option then you can save yourself a lot of build tome.  But figuring that out can be time-consuming (but worth it in the long run if you can avoid building llvm.  For a time on 13.0 if you used MySQL it would pull in llvm90 but that wouldn't be the case on 11.4).

The longer you leave updates, the bigger the mess when you do get around to doing them, so I tend to try and run updates at most monthly (this is running servers, so there's no so much in terms of desktop applications and all those dependencies).  Then if any glitches, it's obvious what updates caused them and a _bit_ easier to work around. But then we come back to the volunteer nature of the work etc. so not so clear-cut for you.


----------



## Deleted member 30996 (Aug 17, 2021)

pschmehl, I ran `portmaster -a` after updating my ports tree today to see what it reported, 2 days after updating all my programs, and it showed graphics/libepoxy and graphics/mesa-libs as being the only ports with an update. I may start running that on a daily basis to keep the time down factor to a minimum. richardtoohey2 talks about it in his post preceding mine.

Yes, some of them can take several hours to compile depending on your machine specs. If you're updating your server it probably doesn't have a number of ports my laptops do.

There's no doubt pkg is quicker and there have been several times I've had to mix pkg and ports to get past a fail point. I haven't ran into a problem from mixing them but have dealt with ports long enough I can deal with them. 

That's where people who do not yet have that experience become frustrated and think about giving up on FreeBSD. And why I recommend people new to FreeBSD don't mix pkg and ports. But that's where experience comes from, solving problems.

So you _may_ be damned if you do, and damned if you don't. Running a server adds more heat to that fire for you. It's up to you to decide how much you can take. 

If I wanted to switch a machine I'd been using ports on to pkg I'd start doing it without rebuilding the machine. This is one I've mixed them on, am still using ports on a regular basis and it's running smooth as can be.


----------



## rigoletto@ (Aug 17, 2021)

TL;DR

If you have a powerful machine available somehow, use it to build the packages with ports-mgmt/poudriere. POUDRIERE will create a repository with your custom packages and then you just need to configure the server to use that repository as its pkg repository. If the eventual powerful machine isn't running FreeBSD, you can ever do it on VirtualBox or something.


----------



## forquare (Aug 17, 2021)

I wonder if ports-mgmt/synth is a good solution here?  I used to use it on a few machines in a previous company, but IIRC you can build packages that require specific options, but for other things it will just use the package.  Saved me a lot of time building, but did require some reading and a little trial and error.


----------



## mark_j (Aug 17, 2021)

pschmehl said:


> Mark, if I'm understanding you correctly, you seem to be saying use pkg for everything you can, and build the custom ports yourself. Is that right?


Absolutely. BUT, I am only talking within this perspective:

1. You don't want to bother answering 100 questions to get some port to build, and/or
2. You have a limited capacity machine on which to run ports.

Nowadays, the complexity is enormous. Every port seems to have its own build system, where it needs to pull in python, perl, ruby, go, java and (God help me!) Rust to run a test that outputs "hello world". It's just gone mad. (I use cmake a lot nowadays, but, seriously, autotools did the job. Now there's more build tools than I can count; ninja, meson,automake,cmake,gmake,scons,sbuild, etc, etc, etc).

Why torture yourself when a build server has already done the hard yards and built a package for you?




pschmehl said:


> Here's a couple of problems that I ran into over the past couple of days getting the mail server updated.
> 
> 1) I had to install and configure Roundcube because Squirrelmail is EOL. So, I ran pkg install roundcube. Later, I was running portmaster, and it wanted to update roundcube. So, clearly, pkg is "behind" ports (for roundcube at least.)



Why? Didn't you just install roundcube? What makes you think updating via ports will make it better? I don't use portmaster (never have) but can't it be used to pull in packages in lieu of building from source to satisfy dependencies? Just what I'm suggesting.

Are packages behind ports? Well it depends on the package repository. Is it pointing to latest or quarterly? Compared to ports? Ports are, with hesitation, always ahead of packages.
If you're going to mix and match, you best get those two synchronised before you go updating.



pschmehl said:


> 2) I locked several important apps and than ran pkg upgrade. It upgraded/installed 121 packages. I then ran portmaster. It wanted to update a bunch more. (I have 349 ports installed on that machine. I don't even know why 2/3rds of them are needed.) So, there is obviously a difference between pkg and building ports with portmaster.



This sounds like I'm picking on you, but I am not. However, you raised the spectre so I will address it.
Why are you upgrading? What are you hoping to achieve? Is some software you're using broken? Have security issues?
Install the software, configure it and leave it. You'll thank me for that sage advice. 



pschmehl said:


> How do you reconcile that? I require certain php extensions for some of the stuff we're running. But pkg removes them because they're not on its list of options. How do I resolve that using pkg?



Build the port. Off hand, I would suspect some package that has hard-coded requirements for php extensions is probably bad anyway. Regardless, build from ports. However, you need to ensure your port tree is synced as close to the quarterly as possible. If you're using latest, then you're in for a world of hurt.

I know the consensus is don't mix/match ports and packages, and yes, it's probably good advice. I don't know your level of programming knowledge so this also might be bad advice from me. All I can say is for many years I have been doing this and NOTHING has ever happened I can't fix quite quickly (usually versions of libraries).

YMMV.



pschmehl said:


> I'm open to new ideas. I just need to understand how it works and what the risks are.



Don't update unnecessarily. If you want the latest/greatest/flashiest feature, then stick to the latest branch of ports and update every day. I hope you like wasting time. 

If you stick to quarterly a lot more packages are stable. Security fixes and that's about all. No need to update, just install your package and relax.




pschmehl said:


> For those who have expressed concern about backups, here's what I'm doing. I wrote a script that creates .tgz files from the bits that need to be backed up (including a mmddyyyy.all.sql file that backups up the mysql dbs), then writes them to /var/backup/. The filenames use the pattern mmddyyyy.filename.tgz. Then they are uploaded to my Dropbox folder. Each day, when the script runs, it deletes the previous day's file from /var/backup/ and the previous 7th-day file from Dropbox. This keeps the /var partition from filling up while keeping the most recent backup handy on the hard drive and keeps the previous 7 day's backups on Dropbox in case I have a disaster that requires an older backup.
> 
> Kludgy, I know, but it's the best I can do with no money for backup software. If you need me to restore something from last month or last year, you're out of luck.



How big are the backups? I'd buy two 128GB+ USB flash drives and use them in rotation for, say, a month plus your off-site.

In summary. If you're scared that mixing ports and packages will make your system unusable, that you don't feel capable to deal with an odd situation should it arise and you're worried about the impact on others, then you should probably stick to packages (where customisation is out) or ports (where customisation is in but is tedious) and never mix the two.


----------



## pschmehl (Aug 18, 2021)

forquare said:


> I wonder if ports-mgmt/synth is a good solution here?  I used to use it on a few machines in a previous company, but IIRC you can build packages that require specific options, but for other things it will just use the package.  Saved me a lot of time building, but did require some reading and a little trial and error.


Thanks for that. I'll take a look.


----------



## pschmehl (Aug 18, 2021)

mark_j, thanks for your insight. I'll try to answer your questions frankly.

Yes to both - don't want to answer 100 questions and have limited capacity (but sufficient) to build ports. Honestly? I'm getting older, and tired of wrestling with ports.
I'm upgrading because I'm a big believer in keeping applications current, for security reasons. One never knows when an app might open a hole in your system
I don't want flashy, latest/greatest. I do want secure. I don't run any non-ssl/tls stuff on my servers except for port 25 for mail. All websites are ssl, mail is imaps, no ports are open that aren't being currently used, and mysql only listens on localhost
I'm not scared of running into problems. I'm just tired of dealing with them. I haven't had a problem yet that I wasn't able to fix, but there's always a first time. I'm not a programmer, but I have some basic knowledge - enough to read and understand code and figure out the cryptic instructions that programmers typically provide for configuring their apps. I used to be a port maintainer for FreeBSD but gave it up when I retired.
I kind of like the idea of using USB sticks. The only downside is then I have to drive down to the colo to insert them. Not a major pain, but still....


----------



## grahamperrin@ (Aug 19, 2021)

pschmehl said:


> … tired of wrestling with ports. …



I understand, but it need not be a fight.

ports-mgmt/poudriere-devel is our friend; "… most people will find it useful to bulk build ports …".

Fears of mixing packages and ports are often unjustified. Here:


```
-b name  Specify the name of the binary package branch to use to prefetch
              packages.  Should be "latest", "quarterly", "release_*", or url.
              With this option poudriere will first try to fetch from the
              binary package repository …
```

– that's *mixture by design*, and it's not heresy. It works.



pschmehl said:


> … 11.4-RELEASE on both servers. I had a bad experience upgrading to 13.x-RELEASE on another server, so I've opted to stay with 11.4-RELEASE for now.



Is there a record of the experience and if so, can you share a link? Thanks.

After 11.4-RELEASE dies: the longer you defer an upgrade, the greater the risk of you encountering a wrestler.


----------



## Deleted member 30996 (Aug 19, 2021)

mark_j said:


> I don't use portmaster (never have) but can't it be used to pull in packages in lieu of building from source to satisfy dependencies?


When you start a build with ports-mgmt/portmaster it checks to see what dependencies are going to be have to pulled in for the build and displays a list for you to peruse before You start the build.

You can choose not to, and it will give you a message telling you that if you don't want to update everything it lists you can use the `portmaster -i`. It will interactively go through the list and ask whether or not you want to update that particular port or not.

When you've gone through every port in the list and issued the command to begin the build it will update only the ports you've chosen.


----------



## pschmehl (Aug 19, 2021)

grahamperrin said:


> Is there a record of the experience and if so, can you share a link? Thanks.
> 
> After 11.4-RELEASE dies: the longer you defer an upgrade, the greater the risk of you encountering a wrestler.


No, there's no record of it, except in my head, and the site is down (likely permanently), so there's no url either.

First, I have done multiple OS upgrades with no problems encountered. But the upgrade to 13.x-RELEASE was a PITA.
1) After rebooting, I could not login. Turns out the /etc/passwd db had gotten corrupted somehow. The hosting party had to fix it and then create a new password for me, after which I could login and change my password.
2) Updating ports was a major disaster. I tried to update mysql 5.6 to mysql 5.7 and the dbs were corrupted. By the time I realized what was going on, my backups were all corrupted as well. That site is now down and has been for over five months. (Long story - the length of the outage is not my fault. The parties involved are still trying to decide what to do.

Unfortunately, I did not backup the dbs immediately before the upgrade. I assumed my backups would be fine. They were not. (Yes, I know about assuming things.) The entire system was db-based, so it's all gone now.


----------



## grahamperrin@ (Aug 20, 2021)

Thanks,



pschmehl said:


> … /etc/passwd …



If it was the bug with which I'm familiar – and if you encountered it when upgrading from FreeBSD 11.3 or less:

the bug was not specific to FreeBSD 13.0-RELEASE
there was a sense of randomness, like, it was difficult to understand why one upgrade from x.z to y.y failed when an apparently equal upgrade from x.z to y.y succeeded
the bug should not recur for you with any upgrade from 11.4-RELEASE.
<https://cgit.freebsd.org/src/commit/?id=2ca137b4306dea2dbe1db31c44102060caedb19a&h=releng/11.4> committed to _releng/11.4_ 2021-02-24.


----------



## grahamperrin@ (Aug 20, 2021)

pschmehl said:


> … tried to update mysql 5.6 to mysql 5.7 and the dbs were corrupted. …



FreeBSD on UFS, yes? There are edge cases where the file system is not as it should be following an interruption (but let's not jump to any conclusion).

Worth noting: at least one of the bugs that were fixed by the commit above was known to affect the _mysql_ user – see for example <https://www.google.com/search?q="pw:+user+'mysql'+disappeared+during+update"&tbs=li:1#unfucked>. (I don't know enough about MySQL to tell whether corruption, in your case, was a consequence of disappearance of the user during freebsd-update(8) with 11.3-RELEASE; I imagine not.)

<https://www.freshports.org/databases/mysql57-server/#message> there's the hint to run `mysql_upgrade`, is it possible that something related failed (and caused corruption) in the absence of the mysql user? (Again, I'm not educated but I imagine not.)

<https://cgit.freebsd.org/ports/tree/UPDATING> nothing recent re: MySQL.

<https://bugs.freebsd.org/bugzilla/buglist.cgi?component=Individual Port(s)&list_id=436365&product=Ports & Packages&query_format=advanced&resolution=---&short_desc=databases/mysql57-server&short_desc_type=allwordssubstr> for 5.7, at a glance I don't see anything matching.

<https://bugs.freebsd.org/bugzilla/buglist.cgi?component=Individual Port(s)&list_id=436364&product=Ports & Packages&query_format=advanced&short_desc=databases/mysql56-server&short_desc_type=allwordssubstr> (all closed) for 5.6, no mention of _corrupt_ on the page.


----------



## richardtoohey2 (Aug 20, 2021)

FreeBSD on UFS, with MySQL, and started back in the FreeBSD 7.x days (can't remember the MySQL version - might have been 4.x in those days).

So far never encountered upgrade corruption issues (as soon as I post this, I will probably be punished!)  I've seen the missing user error - seems to be addressed in 13.x.  Don't think I ever encountered it in production - just when setting up development servers or pushing things or testing upgrades.

Your story definitely a reminder to have backups, rotational backups, offsite backups, etc.  And backup before upgrades.  And test environments that you can trash and rebuild and re-test on.  I understand these things weren't applicable in your case/environment and doesn't help you, just saying the above for people who might read this thread.


----------



## grahamperrin@ (Aug 20, 2021)

richardtoohey2 said:


> … seems to be addressed in 13.x. …



Not just 13. Please see previously linked <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232921#c20>


----------



## pschmehl (Aug 20, 2021)

grahamperrin said:


> Thanks,
> 
> 
> 
> ...


Well, that's comforting. The system I upgraded was 11.3, but both of the ones we've been discussing are 11.4.


----------



## pschmehl (Aug 20, 2021)

grahamperrin said:


> FreeBSD on UFS, yes? There are edge cases where the file system is not as it should be following an interruption (but let's not jump to any conclusion).
> 
> Worth noting: at least one of the bugs that were fixed by the commit above was known to affect the _mysql_ user – see for example <https://www.google.com/search?q="pw:+user+'mysql'+disappeared+during+update"&tbs=li:1#unfucked>. (I don't know enough about MySQL to tell whether corruption, in your case, was a consequence of disappearance of the user during freebsd-update(8) with 11.3-RELEASE; I imagine not.)
> 
> ...


Yes, FreeBSD on UFS. Frankly, I don't understand ZFS (especially the snapshot stuff), and those servers were UFS before there was a ZFS.

I really don't know what happened to mysql. I used the word corruption because the server wouldn't start and couldn't read any of the dbs. I changed back to 5.6 and it still wouldn't start. I don't understand mysql well enough to overcome a problem like that. Someone else may have been able to restore the dbs, but the folks responsible for the server didn't want to spend the money for a pro to fix it.

Historically, when a mysql instance went sideways, I simply wiped it, reinstalled it, and then recreated the dbs, which mysql was then able to read the files on the hard drive, and everything was back to normal. That didn't work this time, and I have no idea why. The files are still there. But, if they won't load, I assume something got corrupted.

I did something really stupid with my backups (since corrected). As I mentioned earlier, I wrote a script that writes tar.gz files to the /var/backup partition and also uploads a copy to Dropbox (using dropbox-uploader.sh), but the script deleted the previous day's file both on the hard drive and on dropbox. So, when the system crapped out, and the backup script ran, poof. All gone.

That has since been corrected. The local file is deleted each time the script runs, but the Dropbox duplicates are kept for seven days. That at least gives me a chance to gather my thoughts and preserve good copies before it's too late. Live and learn. The reason for the daily file deletion was space on the hard drives, but space isn't a problem on Dropbox. I should have thought of that, but I didn't.

It's all water under the bridge now. The folks that "owned" that server clearly didn't care that much, because it's been down for five months and they've not done anything to correct the problem or even to start over from scratch.


----------



## richardtoohey2 (Aug 21, 2021)

pschmehl said:


> It's all water under the bridge now. The folks that "owned" that server clearly didn't care that much, because it's been down for five months and they've not done anything to correct the problem or even to start over from scratch.


Lost count of the number of times I've fretted about something and the people who should have cared didn't.  But like you say, learn what you can and move on.  When you've got restricted resources, just have to do the best you can and sometimes you lose.


----------



## astyle (Aug 21, 2021)

richardtoohey2 said:


> Lost count of the number of times I've fretted about something and the people who should have cared didn't.  But like you say, learn what you can and move on.  When you've got restricted resources, just have to do the best you can and sometimes you lose.


Sometimes, in big organizations, it's possible to lose track of servers you actually 'own'. And then they get forgotten about - until it's time to either upgrade or they're so broken it's easier to just get rid of them. I've seen quite a few madhouses over that - but after doing some thorough homework, I just sit at my desk, drink tea, and watch a burning dumpster barge float past me (figuratively, of course).


----------



## grahamperrin@ (Aug 21, 2021)

pschmehl said:


> … all water under the bridge now. …



☑



pschmehl said:


> … wouldn't start and couldn't read any of the dbs. I changed back to 5.6 and it still wouldn't start. …



Someone might like to test whether removal of the _mysql_ user has this effect.


----------



## Deleted member 30996 (Aug 29, 2021)

pschmehl said:


> Trihexagonal, I try not to disrupt service operations any more than I have to, so I update infrequently. But there are some ports that take forever to update. llvm80, llvm90, and cmake are three that make me groan every time they show up on the list (which is often.) There are some others, but those three always take a very long time to build.


I ran `portmaster -a` on my Thinkpad T400 yesterday morning and it returned 60 ports to update.

This being the machine I used last time with the 2nd Round of the Online Turing Test. I had to keep a Firefox interface for my bot up on i for 24 straight or be disqualified. That was 3 months ago and it could still be up running now if I had stayed connected, my .W520 .mp3 player is at over 4 months.

But I wanted to stay current with System patches and keep my programs up to date for stability, so I went ahead and started it running yesterday morning. It finished this morning, I issued `# rehash`, updated security/rkhunter, my ports tree and ran `portmaster -a` again this morning. Overnight lang/rust and www/firefox-esr got an update, and both take a long time to compile.

But the Test date isn't till next Saturday. So it's toiling away with the only oPolar gaming fan I have keeping it cooled and will be done in a matter of a few hours. That fan was well worth the $30 or so it cost and I'm going to get another one, or I'd be doing this T61 along with the T400.


----------

