# Deprecating base system ftpd



## eternal_noob (Apr 7, 2021)

Our favourite programmers are discussing wether to deprecate the ftpd in FreeBSD base or not.
What do you think? Keep it or toss it? Do you still use it?


----------



## kpedersen (Apr 7, 2021)

If the OpenSSH upstream deprecates scp and our guys deprecate ftpd then I will be a little annoyed. I don't like http and I especially despise https.

I vote keep. It is tiny. If it really troubles them, rip it out from the rest of the system and keep it as a single binary that an unprivileged user can just run on a port above 1000 temporarily to transfer files.

Whats left?

`cat somefile.txt | nc -l 1987`
and
`nc host1 1987 > somefile.txt`

I know I would probably end up using that more than I want rather than install an additional port.


----------



## zirias@ (Apr 7, 2021)

I'm surprised something like that is still there. IMVHO, base should (well, simplified) contain anything absolutely necessary for a self-contained "Unix" system. But that doesn't mean to keep ancient stuff around. A means to access files on a remote machine definitely belongs into a Unix system, but even Unix changes over time, and `ftp` isn't what you normally use any more. Most people probably use `scp` or `sftp` instead, and both are present in base.

So, IMHO, yes, time to get rid of old cruft. Anyone who wants to operate an `ftp` service will find a matching port/package.


----------



## olli@ (Apr 7, 2021)

There is no reason to keep ftpd in the base system. You can install it from ports / packages without problems.
 
Basically, the long-time goal should be to remove _everything_ from the base system that is not required for installing or booting FreeBSD. ftpd certainly isn’t.
Everything that _can_ be a package _should_ be one, IMHO.


----------



## zirias@ (Apr 7, 2021)

olli@ said:


> Everything that _can_ be a package _should_ be one, IMHO.


I think that might be a bit too "radical". To give an example: I'd be pretty annoyed if cu(1) would vanish from base, although it would work perfectly as a package. The reason is, you need such a tool for administration of headless machines (well, short of a real-life serial terminal in hardware, I guess they are extinct, hehe).

So, my opinion is: keep "base" a self-contained, fully functional Unix system. It will always cause some debates whether tool x or tool y is required for that. As a rule of thumb, I'd say keep everything directly needed for administration and disaster recovery.

But, in general, I agree with "keep base small"


----------



## Jose (Apr 7, 2021)

Zirias said:


> I think that might be a bit too "radical".


No SSH in base would be kind of a bummer.


----------



## ralphbsz (Apr 7, 2021)

OpenSSH is not planning to remove the existing scp until they have a replacement; they're working on one that has the same command line format for common (simple) use cases.

Personally, I use scp a lot (but only in very simple fashions), rsync a lot (but usually also in a simple fashion, for whole directory trees with attributes). I never use ftp. Given that both ftp and the traditional scp are too easily abused (either intentionally or innocently), I'm in favor of removing them, as long as a simple and safe replacement is available.

As to Olli's opinion that nearly everything that's not vital to get booted should be in a package: For larger stuff, that makes a lot of sense. But there are a lot of small utilities that are highly useful, and that I don't want to go hunting for a package for. Zirias already suggested cu, and I agree. I would add both bc and dc, which I use regularly. And these utilities are so small (a single dozens of kB executable), it's not worth creating a separate package for each. Having several hundred commands in /usr/bin is not a big problem in my mind.


----------



## zirias@ (Apr 7, 2021)

Jose said:


> No SSH in base would be kind of a bummer.


Incidentally, I build mine with `WITHOUT_OPENSSH=yes` 

But I agree, nowadays, SSH is another perfect example for a functionality expected from a self-contained Unix system. My only reason is that I'm installing it from ports anyways. Perhaps, some time in the future, pkg-base will cover such "special needs". But nevertheless, it should stay in base.


----------



## olli@ (Apr 7, 2021)

Jose said:


> No SSH in base would be kind of a bummer.


Don’t worry, ssh will stay in base (at least the client, but not necessarily the sshd daemon) because it may be required for certain ways of installing FreeBSD, e.g. to set up a tunnel for downloading the distribution sets and packages.
 
The same was once true for cu(1), by the way.


----------



## Jose (Apr 7, 2021)

olli@ said:


> Don’t worry, ssh will stay in base (at least the client, but not necessarily the sshd daemon) because it may be required for certain ways of installing FreeBSD, e.g. to set up a tunnel for downloading the distribution sets and packages.


No SSH daemon in base would be a real bummer


----------



## olli@ (Apr 7, 2021)

Jose said:


> No SSH daemon in base would be a real bummer


Why?


----------



## Jose (Apr 7, 2021)

olli@ said:


> Why?


Most of my Freebsd machines are headless, many are remote. It would be a pain to have to install a port or package as part of the install.


----------



## olli@ (Apr 7, 2021)

Jose said:


> Most of my Freebsd machines are headless, many are remote. It would be a pain to have to install a port or package as part of the install.


I’m not sure I understand. What is the problem with adding packages during installation? I do that all the time, including on remote servers. It doesn’t matter if they’re headless or not.


----------



## Jose (Apr 7, 2021)

olli@ said:


> I’m not sure I understand. What is the problem with adding packages during installation? I do that all the time, including on remote servers. It doesn’t matter if they’re headless or not.


I guess it wouldn't matter if whatever install media you used had some packages on it. I often don't enable public networking while installing because I want to vet my firewall rules first. Maybe I'm scarred by the bad old days when the mean time to compromise of a fresh install of Windows was minutes even on some (large) private networks.

Going along this path, which packages would you provide in the install media? Clearly not all of them, that would be too much, so some small subset of essential packages. But haven't you just redefined what "base" means?


----------



## zirias@ (Apr 7, 2021)

I think the discussion about what belongs into base and what not will stay forever 

I personally think it's outright weird to still have sendmail in base, and it's somewhat questionable to have tcsh (cause, all that's needed is a POSIX shell, and for something comfortable for interactive use, you pick whatever you like from ports/packages).

But then, going the "radical" way and removing everything not strictly needed during installation has other drawbacks.

If pkg-base is finally there and enabled by default, the discussion might shift a bit, towards only "what makes sense to _maintain_ in base". Let's see where this leads


----------



## George (Apr 7, 2021)

They already decided to keep it in base.

No need to discuss this any further.


----------



## olli@ (Apr 7, 2021)

Jose said:


> Going along this path, which packages would you provide in the install media? Clearly not all of them, that would be too much, so some small subset of essential packages. But haven't you just redefined what "base" means?


Yes, that’s right – that’s what the pkg-base project is all about.
 
Actually the desire to “package the base system” is rather old. The current way of installing FreeBSD with so-called “distribution sets” is old-fashioned and has some disadvantages. It makes a lot of sense to convert it to the package format used by pkg, and make it more fine-grained. This will make the installation procedure simpler and more robust, and it will also make it easier to tailor it for your own needs, e.g. when you do scripted installs for a computing center or similar things. For example, if you don’t need tftp, then just drop the tftp package from the installation. There will probably be a “default base package set” that basically covers much of what is in the base system today.

Regarding installation media: Yes, SSH (both client and server) should certainly be available from standard installation media.


----------



## zirias@ (Apr 7, 2021)

Elazar said:


> No need to discuss this any further.


There's no need to discuss it _here_ anyways, but I'd expect this topic to come up again, so, still interesting.


----------



## eternal_noob (Apr 7, 2021)

Elazar said:


> No need to discuss this any further.


Sorry, but this is hilarious. Why shouldn't we discuss about the discussion? This is a forum meant for discussion after all.
I am well aware that our opinions don't matter but they are still interesting.


----------



## kpedersen (Apr 7, 2021)

olli@ said:


> There will probably be a “default base package set” that basically covers much of what is in the base system today.


I personally don't like this "DIY Linux" approach. The base is there so that programs can be managed and maintained in a monolithic and unified way. Splitting it up achieves very little and just makes it a modular mess reducing consistency. I think it is a bad move. I don't believe it has worked particularly well with Xorg at all.

We will end up with a fairly useless initial install. Great for car set-top boxes probably but not UNIX-like and I am not sure it will even conform with POSIX / SUS etc.

But I do feel this is an approach that interests the developers and so will probably be taken. I strongly believe they would be wrong here though.

Of course providing installation .img/.iso with 40G+ of packages will solve this. A nice "modern" solution


----------



## Alain De Vos (Apr 7, 2021)

Zirias said:


> I think the discussion about what belongs into base and what not will stay forever
> 
> I personally think it's outright weird to still have sendmail in base, and it's somewhat questionable to have tcsh (cause, all that's needed is a POSIX shell, and for something comfortable for interactive use, you pick whatever you like from ports/packages).
> 
> ...


I think sendmail & tcsh are there for historical reasons. And freebsd does not like to give up it's history even if it's became obsolete.
If you have sendmail in base you can as well have postgresql in base 
Personally I think any server does not belong in base, including time-server ntpd or dns-server local_unbound , mailserver sendmail.


----------



## olli@ (Apr 7, 2021)

kpedersen said:


> I personally don't like this "DIY Linux" approach. The base is there so that programs can be managed and maintained in a monolithic and unified way. Splitting it up achieves very little and just makes it a modular mess reducing consistency. I think it is a bad move. I don't believe it has worked particularly well with Xorg at all.
> 
> We will end up with a fairly useless initial install. Great for car set-top boxes probably but not UNIX-like and I am not sure it will even conform with POSIX / SUS etc.


No, the initial install will not be much different by default. You will get the same things that you get today when you select “default install” in the installer.


----------



## Jose (Apr 7, 2021)

olli@ said:


> Yes, that’s right – that’s what the pkg-base project is all about.
> 
> Actually the desire to “package the base system” is rather old. The current way of installing FreeBSD with so-called “distribution sets” is old-fashioned and has some disadvantages.


I personally really like the approach Openbsd takes, which is installation = unpack some tarballs. Super simple and not error-prone.


olli@ said:


> It makes a lot of sense to convert it to the package format used by pkg, and make it more fine-grained. This will make the installation procedure simpler and more robust...


I dunno. I believe Gentoo Linux pioneered this everything-is-a-package approach and their install was (is?) a hot mess. They still wound up with a package called "baselayout", BTW.


olli@ said:


> ...and it will also make it easier to tailor it for your own needs, e.g. when you do scripted installs for a computing center or similar things. For example, if you don’t need tftp, then just drop the tftp package from the installation. There will probably be a “default base package set” that basically covers much of what is in the base system today.


Do you need this crutch if you're sophisticated enough to be doing scripted installs on a large fleet of machines? Also, I find that Mfsbsd suits these needs quite nicely.


----------



## msplsh (Apr 7, 2021)

Would be interesting to enumerate all the programs capable of network file transfer in base and figure out which ones are plain redundant...

Sendmail is in base to presumably provide some MTA.


----------



## kpedersen (Apr 7, 2021)

olli@ said:


> No, the initial install will not be much different by default. You will get the same things that you get today when you select “default install” in the installer.


But for how long until people start pecking at things.

"Oh, I don't think users would find it shocking if ed was removed from the default install" (semi-quote from Debian mailing list)

"Oh, we don't really need two shells. Lets remove one"

"Do we really need a C/C++ compiler in base in 2021? Rust is so _*modern*_"

And it could also work the other way.

"Lets just tuck in a Python interpreter to the base meta-package. It is so convenient to have"

"Actually, lets just grab a Python 2.7 and 3.x package in the base meta-package to cover all angles"



Jose said:


> I personally really like the approach Openbsd takes, which is installation = unpack some tarballs. Super simple and not error-prone.


I agree. I actually find it strange that so many other operating systems aren't seeing the elegance of that. I would also suggest that OpenBSD is the last remaining free operating system that is suitable for an entirely offline lab (No online-centric repo infrastructure, no 300+ packages just for a single package (Xorg). Site specific firmware clearly isolated, a responsible collection of default packages and servers like httpd, sshd).

It provides all of this and yet still ends up smaller than most Linux distros and FreeBSD oddly enough. This must be because they are less relaxed on optional extras being dragged in via packages.


----------



## olli@ (Apr 7, 2021)

Well, they can remove ed today, too, if they want. Or one of the shells. I don’t think the decision for doing that would be no different.

And I highly doubt Python will go into the default install. Convenience for some people alone is not a valid reason, not today and not when pkg-base exists.


----------



## kpedersen (Apr 7, 2021)

olli@ said:


> Well, they can remove ed today, too, if they want. Or one of the shells. I don’t think the decision for doing that would be no different.
> 
> And I highly doubt Python will go into the default install. Convenience for some people alone is not a valid reason, not today and not when pkg-base exists.


Whilst that is absolutely true, once they are just dealing with a meta-package, I believe this will be a lot more volatile than our current base approach.

ed is required for POSIX as far as I know. If the developers want to break this, then that is certainly their choice and they can turn FreeBSD into a mess. However I prefer UNIX-like operating systems. 

I am also not entirely sure if they will put base packages in /usr or just regress to spamming everything in /usr/local. It does open up for some really dodgy future decisions to be made.


----------



## olli@ (Apr 7, 2021)

kpedersen said:


> ed is required for POSIX as far as I know. If the developers want to break this, then that is certainly their choice and they can turn FreeBSD into a mess. However I prefer UNIX-like operating systems.


Well, I assume there will be several package sets to choose from, for example a “default set” for most normal users, a “minimal set” for embedded systems, and a “POSIX conformance set” for you that contains ed.  And it will be very easy to create your own installation set.


----------



## kpedersen (Apr 7, 2021)

olli@ said:


> Well, I assume there will be several package sets to choose from, for example a “default set” for most normal users, a “minimal set” for embedded systems, and a “POSIX conformance set” for you that contains ed.  And it will be very easy to create your own installation set.


Heh I do hope so. I guess my concern comes from the fact that no Linux distro has done this and at this point FreeBSD is basically following them in this regard to little ratty packages.

Though on a positive note, we might see some sort of return of x-base again.


----------



## ct85711 (Apr 7, 2021)

Well, I know that removing the openssh daemon from base would really be a real pain; as it would really make it significantly more difficult to do a headless install.  Right now my server is headless, so the only way to get access is through ssh in; unless I scrounge up and drag a monitor and keyboard over to connect to the machine.  Even then, the alternative is the installer have to somehow configure and install a copy of pkg tree and have the openssh pkg that installs too.  That isn't any better than having openssh deamon as part of base.

As far as ftpd goes, I don't mind so much if gets removed from base, as long as it is available as a package.  My thinking on that, is when you are initially configuring the machine, ftp isn't as big of a need (though helpful to grab a copy of the configurations from a central location).


----------



## ralphbsz (Apr 7, 2021)

olli@ said:


> What is the problem with adding packages during installation?


Convenience. I want the base system to contain exactly those things that I happen to use, no more and no less. Without the hassle of installing or maintaining packages. Ideally, there should be a single command "freebsd-update" that does all the updates, and that is easiest if all the packages I need are in the base.

If you didn't notice, that was meant as humor. Because the next sentence is: And I don't care about any of the other people.


----------



## zirias@ (Apr 7, 2021)

Grinning started exactly here:


ralphbsz said:


> I want the base system to contain exactly those things that I happen to use, no more and no less.



And yes, this is the perfect description for almost any discussion about contents of base. And the perfect explanation for the direction it always takes. Heated polemics instead of technical arguments.


----------



## kpedersen (Apr 7, 2021)

In all fairness, I am fine with ralphbsz's suggestion. Even if all the packages in base are specifically tailored for him, I would rather that even for my use-case. So long as base *is* a base rather than a scatty ever changing collection of packages.


----------



## hruodr (Apr 7, 2021)

Alain De Vos said:


> I think sendmail & tcsh are there for historical reasons. And freebsd does not like to give up it's history even if it's became obsolete.



*BSD is what it is for historical reasons. Also UNIX is what it is for historical reasons. They could be much
better, but they are *BSD and UNIX. That "obsolete" programs could be better, but are good enough. And
much better things are not being done.

The idea of a meager base system, in which people select what they want from packages, may sound good,
but it means to renounce to have a standard complete system. The experience with Linux shows to
what chaos this leads. Yes, you can install packages in a Linux distribution to get something similar to traditional Unix, but there are now people that not even know how a Unix system looks like.

In many ocasions it is a nightmare not to have some expected small programs at hand.

To have a standard is an advantages.



Alain De Vos said:


> Personally I think any server does not belong in base, including time-server ntpd or dns-server local_unbound , mailserver sendmail.



Absurd! Why to erase a program only because it is not the part that initiates the connection?




olli@ said:


> Well, they can remove ed today, too, if they want.



ed is very useful, if not as a normal editor, at least inside of scripts. But I know, ed is obsolete,
cool people prefer to put emacs in their scripts, emacs is better. And desktop users do not write 
scripts, then better delete ed anyway.


----------



## shkhln (Apr 7, 2021)

hruodr said:


> The idea of a meager base system, in which people select what they want from packages, may sound good,


"meager" never sounds good, you serial word abuser.


----------



## kpedersen (Apr 7, 2021)

shkhln said:


> "meager" never sounds good, you serial word abuser.


Hah, in that link it defines "meager" as:

"having little flesh"

Sounds like a typical "base" Linux distro to me. So I think it is fair 

If you install Debian that is pretty much what you have. I don't believe even wpa_supplicant is there unless you install a random meta package called "standard system utilities".


----------



## mark_j (Apr 7, 2021)

When is this sort of stuff going to stop, seriously? Are we to see the demise of dd? mkfile? clang?

When there's a base system that takes 10 minutes to install then 30 minutes of adding packages you've basically become Gnu/systemd/linux.

Core needs to pull their head in. Focus on preventing big stuff ups like wireguard and  less on trimming freebsd down to just a kernel.

They seem to have lost the plot.


----------



## zirias@ (Apr 7, 2021)

Zirias said:


> Heated polemics instead of technical arguments.


Getting hotter 

And FWIW, there are always people panicking about any attempt to _improve_ things. This can get very exhausting.

pkg-base won't change anything about having a stable base with release engineering. It won't change anything about having all base integrated in one repository. It won't even stop you from building it all at once from the source tree (or, installing it that way).

All it _will_ do is enable to build packages from base, thus bringing a feature to users of binary releases I'm using for a long time from source: Leave out stuff I _personally_ don't need.


----------



## olli@ (Apr 7, 2021)

mark_j said:


> When is this sort of stuff going to stop, seriously? Are we to see the demise of dd? mkfile? clang?


Seriously – dd is required by the standard boot procedure (e.g. stuff in /etc/rc.d), so it can’t be optional. The same holds true for many other standard utilities.
 
mkfile I don’t know; I don’t have a program with that name on my FreeBSD machine.
 
And clang – well, why shouldn’t it be optional? If a non-developer installs FreeBSD for normal use, clang would be a waste of space. In fact, when I install FreeBSD on a space-constrained machine, I remove the  compiler tool chain, static libraries, include files and several other things. It would make sense to be able to de-select these in the installer if you don’t need them.

As far as the famous ed(1) is concerned, nothing in the current base system requires it. The existing scripts use sed, awk, grep, cut, tr etc.

By the way, note that the source-level build system of FreeBSD (a.k.a. “make world”) already supports selecting / de-selecting several components of the base system via src.conf(5). For example, it supports de-selecting clang, so it will build and install a base system that does not contain clang.


----------



## kpedersen (Apr 8, 2021)

olli@ said:


> And clang – well, why shouldn’t it be optional? If a non-developer installs FreeBSD for normal use, clang would be a waste of space.


A system compiler installed is dictated by the POSIX and SUS standard.

That and ed


----------



## mark_j (Apr 8, 2021)

Right, so strip EVERYTHING out that isn't used for the boot process and then watch your small userbase dwindle even further? Idiocracy is the new method of running FreeBSD?

It's absurd and reduces FreeBSD to a kernel. People advocating this need to move to Linux.


----------



## zirias@ (Apr 8, 2021)

olli@ said:


> And clang – well, why shouldn’t it be optional?


When discussing pkg-base, we must be careful not to mix up things. Clang should (IMHO) *never* be removed from base, it wouldn't be self-contained any more regarding building. But, of course, it should be optional to _install_ it


----------



## kpedersen (Apr 8, 2021)

mark_j said:


> Right, so strip EVERYTHING out that isn't used for the boot process and then watch your small userbase dwindle even further? Idiocracy is the new method of running FreeBSD?
> 
> It's absurd and reduces FreeBSD to a kernel. People advocating this need to move to Linux.


Agreed. It basically would be Debian kFreeBSD at this point. Just pull in the "pkg-base" via apt-get instead and be done with it.


----------



## zirias@ (Apr 8, 2021)

mark_j said:


> Right, so strip EVERYTHING out that isn't used for the boot process and then watch your small userbase dwindle even further? Idiocracy is the new method of running FreeBSD?
> 
> It's absurd and reduces FreeBSD to a kernel. People advocating this need to move to Linux.


Understanding things isn't really your strength…

Do you have more unfounded, silly and totally non-technical "arguments"?


----------



## ShelLuser (Apr 8, 2021)

eternal_noob said:


> Our favourite programmers are discussing wether to deprecate the ftpd in FreeBSD base or not.
> What do you think? Keep it or toss it? Do you still use it?


I still use _and_ enjoy using ftpd and to be honest I'm not bothered with this at all. For several reasons...

First the obvious... how often do you install FreeBSD?  I'm convinced that those who do have long customized the procedure and if they didn't then maybe now would be a good time to learn how to do so.

Second... I'm _already_ "unhappy" with the base system! (_naah_ ).  Not unhappy as in displeased or such, but the current base system already doesn't cut it for me. All my FreeBSD servers need at the very least the addition of: Midnight commander, Git, GnuPG, Korn shell (pdksh), docproj, lynx (because of said docproj ), portmaster, pwgen, screen, rar (I even have this officially licensed, *awesome* archiver IMO), chkrootkit, java (though I'm somewhat moving away from this a bit) and I also rely on the PostgreSQL server.

Probably needless to say but I also don't rely on packages for this ("sorta") so these get build with custom options.

In other words.. I'm already spending plenty of time to set everything up and adding one more package to this list isn't going to change much for me.

And the final reason...  I always update my servers by using the source (and not just for the geek sound of it ). So...  how long has it been since GCC was dumped and replaced by Clang? Coincidence has it that I just updated the source tree this evening so.. let's take a look here...


```
peter@vps:/usr/src $ man -m share/man src.conf | grep -i gcc
             CCACHE_CPP2 option is used for Clang but not GCC.
             phase of the build.  To be able to build the system, either gcc
             Set to install the GCC compiler as /usr/bin/cc, /usr/bin/c++ and
             Set to not build and install gcc and g++ as part of the normal
             Set to build and install gcc and g++.
             Set to not build gcc and g++ as part of the bootstrap process.
             You must enable either gcc or clang bootstrap to be able to build
             Set to build gcc and g++ as part of the bootstrap process.
             platforms where gcc is the system compiler.
             Set to use GCC's stack unwinder (instead of LLVM's libunwind).
             Set to use LLVM's libunwind stack unwinder (instead of GCC's
```
Surely I must be seeing things here...  hmm, nope: WITH_GCC is still a thing.

How long has it been?  Well, I did a `git log` and looked around then found this:


```
commit 52b42bace1338f8146d7dd5d1a7f6e41d5f5f80d
Author: David Chisnall <theraven@FreeBSD.org>
Date:   Fri Sep 6 20:08:03 2013 +0000

    On platforms where clang is the default compiler, don't build gcc or libstdc++.
    To enable them, set WITH_GCC and WITH_GNUCXX in src.conf.
    Make clang default to using libc++ on FreeBSD 10.
    Bumped __FreeBSD_version for the change.
```
Sounds about right to me because it was around 10 when the transition started, I still remember because I moved away from GCC manually and started using Clang long before it became official.

So 8 years later and GCC is still in the base system?  I'm not worried.

Just my 2 cents here of course.

(edit)



Jose said:


> I personally really like the approach Openbsd takes, which is installation = unpack some tarballs. Super simple and not error-prone.


FreeBSD does the same thing though. Why else do you think base.txz and kernel.txz exist? I always use those 2 to set up a new jail. Painless, only needs a default configuration added to /etc.


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> It basically would be Debian kFreeBSD at this point. Just pull in the "pkg-base" via apt-get instead and be done with it.


Utter nonsense. It's just a more flexible way for distributing the same old thing, which is STILL a complete and stable base system developed as a whole. You're just not forced to build it yourself in order to leave out things in your install any more.

*edit:* BTW, have a look e.g. here: https://cgit.freebsd.org/src/tree/bin/ed/Makefile?h=releng/13.0#n5
This kind of infrastructure for actually packaging base components has been there for a long time already. In this case, it tells you that `ed` will go in a `runtime` package.


----------



## mark_j (Apr 8, 2021)

Zirias said:


> Understanding things isn't really your strength…
> 
> Do you have more unfounded, silly and totally non-technical "arguments"?


Your ad hominems are tiresome.


----------



## zirias@ (Apr 8, 2021)

mark_j said:


> You're ad hominems are tiresome.


Look at your dumb posts.


----------



## kpedersen (Apr 8, 2021)

Zirias said:


> Utter nonsense. It's just a more flexible way for distributing the same old thing, which is STILL a complete and stable base system developed as a whole. You're just not forced to build it yourself in order to leave out things in your install any more.


I honestly don't see the difference between:

`# pkg install pkg-base`

and

`# apt-get install pkg-base`

Both grab the same files. Both presumably populate /usr/local. Really the filesystem should end up identical apart from some random GNU programs lying around in /usr (and maybe grub and some other Linux crap).

Especially if you start ricing with the install and leaving out things like clang or... ed.


----------



## olli@ (Apr 8, 2021)

kpedersen said:


> A system compiler installed is dictated by the POSIX and SUS standard.
> 
> That and ed


FreeBSD isn’t POSIX / SUS compliant anyway.  Seriously, a user shouldn’t be forced to install things he don’t need, no matter if some standard dictates it or not. I prefer freedom of choice to dictatorship.

Apart from that, I am _not_ saying that FreeBSD shouldn’t ship with a compiler. It certainly should. I’m just saying that it would be useful to have an option to not install it if you don’t want it.


----------



## zirias@ (Apr 8, 2021)

kpedersen, you only proved that you have NO idea how pkg-base works. It doesn't change *anything* in the structure of base, neither in the source nor in the installed version. It just enables to build individual binary packages. See also my edit above.


----------



## kpedersen (Apr 8, 2021)

olli@ said:


> FreeBSD isn’t POSIX / SUS compliant anyway.


No but it certainly doesn't seem good to be moving in the wrong direction either.


olli@ said:


> I’m just saying that it would be useful to have an option to not install it if you don’t want it.


If you want to rice with things, then


```
rm /usr/bin/clang
rm /bin/ed
```

And just remember to mention that on the mailing lists if you ever run into issues and need troubleshooting.

Whats next? Split development libraries from packages? aka zlib, zlib-dev


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> Whats next? Split development libraries from packages? aka zlib, zlib-dev


Given some packages pull in a whole gcc10, just to have a few runtime libraries, a mechanism to build more than one package from one port is desperately needed.*) But that's a different topic.

*) and FWIW, I've seen this issue discussed on IRC and mailing lists.


----------



## kpedersen (Apr 8, 2021)

Zirias said:


> Given some packages pull in a whole gcc10, just to have a few runtime libraries, a mechanism to build more than one package from one port is desperately needed. But that's a different topic.


Possibly a different topic. But I am again of a different idea when it comes to packages, less is better. I.e a gtk+ package should bundle all the gtk+ cruft such as pango, gobject, gconf into one package called gtk+. They never get used individually anyway. This is just modular monolith GNU crap.

As for the pkg-base, I am not entirely certain I have misunderstood the design. Just because programs are grouped into larger meta-packages with a different name doesn't mean they wont be fragmented and split at a later date.

Are you ever going to install the i.e compiler packages and not the runtime packages? What is the point of modularising them when a base is a monolithic concept?


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> Possibly a different topic. But I am again of a different idea when it comes to packages, less is better. I.e a gtk+ package should bundle all the gtk+ cruft such as pango, gobject, gconf into one package called gtk+. They never get used individually anyway.


Except when they are, which is especially true for a few libraries out of a huge package with everything. The assumption that the machine running the software is also the build machine is flawed.


kpedersen said:


> As for the pkg-base, I am not entirely certain I have misunderstood the design. Just because programs are grouped into larger meta-packages with a different name doesn't mean they wont be fragmented and split at a later date.


Then again, from what you write here, I'm _pretty_ sure you misunderstand the concept. I'll try again: nothing will change about base being self-contained, one repository, developed and release-engineered as a whole. And then, infrastructure for building packages is already there. Once this is officially used, all that will change is instead of distributing one huge base.txz, distribute a lot of small .txz packages with (together) exactly the same content. This just means you don't have to compile yourself any more in order to choose to NOT install some component. And that's it…

(I guess one "blocker" for pkg-base to go "live" could be that you need to revisit ALL the ports, so they specify what they need from base. I'd even consider _that_ a valid argument against it, as it's more work for port maintainers – although I think it's manageable once initially started…)



kpedersen said:


> Are you ever going to install the i.e compiler packages and not the runtime packages? What is the point?


The usecase is the OTHER way around. Software compiled with GCC needs GCC runtime libraries. This is an extreme example, but one that's relevant for many packages. Just to _run_ a software compiled with GCC, you need to install the whole _huge_ GCC package to have the few libraries available.
See here: https://cgit.freebsd.org/ports/tree/Mk/bsd.gcc.mk#n148
This is added except for GCC-based architectures.


----------



## kpedersen (Apr 8, 2021)

Zirias said:


> Except when they are, which is especially true for a few libraries out of a huge package with everything. The assumption that the machine running the software is also the build machine is flawed.


So for the few gcc projects I had, to eliminate the dependency on gcc package at runtime, I just statically link the runtime in. Yes, not available for everyone but clang *is* the FreeBSD system compiler. If people go against that, that is a little bit their problem 

If you extract all those gtk+ packages, it comes to about ~5 megs. The package meta-data from all those packages in cache probably takes up more space than the packages themselves. Gtk+ is one thing.



Zirias said:


> Once this is officially used, all that will change is instead of distributing one huge base.txz, distribute a lot of small .txz packages with (together) exactly the same content. This just means you don't have to compile yourself any more in order to choose to NOT install some component. And that's it…


Thats fine. So long as you can cat those ratty small files together and rename it base.txz I am happy. Otherwise, why are they fragmenting those files?

What you are saying is that if they were made into .deb files and installed via apt-get it would work the same way right?


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> So for the few gcc projects I had, to eliminate the dependency on gcc package at runtime, I just statically link the runtime in. Yes, not available for everyone but clang *is* the FreeBSD system compiler. If people go against that, that is a little bit their problem


So, it's your fault as an end user wanting to run a software that only compiles with GCC? How could you be so stupid wanting to use it? Then live with tons of "crap" installed you never need.


kpedersen said:


> Thats fine. So long as you can cat those ratty small files together and rename it base.txz I am happy. Otherwise, why are they fragmenting those files?


I answered this very question twice. Calling it "fragmentation" is already (at best!) misleading. Do you have any argument for forcing anyone to always install the whole base, or otherwise compile himself? How does it hurt you?


----------



## kpedersen (Apr 8, 2021)

Zirias said:


> The usecase is the OTHER way around. Software compiled with GCC needs GCC runtime libraries.


Yes, I know. I was being annoying to try to show that it is slightly of limited use.

So OpenBSD does a little bit similar with their base.tgz and comp.tgz packages. However they basically also say if you don't install comp.tgz (even on a server) you are being weird and you are on your own.

Ricing is something that should not be normalized or you will just end up with crazy fragmentation with users doing mad things.



Zirias said:


> So, it's your fault as an end user wanting to run a software that only compiles with GCC? How could you be so stupid wanting to use it? Then live with tons of "crap" installed you never need.


Or get someone to fix that broken crap so it compiles with Clang. We do this for MSVC++ centric code all the time right? It is broken and needs fixing.


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> Ricing is something that should not be normalized


More strawman nonsense. It's just about not installing stuff you don't need.


----------



## kpedersen (Apr 8, 2021)

Zirias said:


> More strawman nonsense. It's just about not installing stuff you don't need.


So will the installer also provide a list of kernel modules / firmware the user probably wont need and they can pluck them away?

Honestly I see what you mean if we were talking about serious size but for the sake of ~10 megs it is *not* a good use of time.


----------



## zirias@ (Apr 8, 2021)

I don't care how exactly an installer making use of pkg-base looks like. Most probably, it will just install the whole base anyways by default. The relevant thing is you _can_ remove things you don't want/need (even later).

Again, you can already right now, but you have to compile yourself to do so. My reason for leaving out SSH is I always use the one from ports, linked against libressl. I don't want to have a second unused variant installed, and this is also for minimizing the risk that it is, accidentally, used. For other stuff I leave out, the motivation is more to save unnecessary build time than disk space, but again: How does it hurt you when users of binary releases have this freedom as well?


----------



## olli@ (Apr 8, 2021)

kpedersen said:


> No but it certainly doesn't seem good to be moving in the wrong direction either.


Are you misunderstanding me on purpose? Again, I’m not saying that FreeBSD should ship without clang.
 


> If you want to rice with things, then
> 
> 
> ```
> ...


No, the correct approach would be something like `pkg delete ed` (similarly for clang) so its (non-)presence is properly recorded in the package database, so other things that depend on it can react accordingly.
 


> And just remember to mention that on the mailing lists if you ever run into issues and need troubleshooting.


It’s unlikely that there will be any issues. What kind of issues would you expect? If a 3rd-party package requires ed, it has a dependency recorded and pulls in the ed package if needed, like with any other dependency.
 


> Whats next? Split development libraries from packages? aka zlib, zlib-dev


We already do that for quite some time.


----------



## kpedersen (Apr 8, 2021)

olli@ said:


> Are you misunderstanding me on purpose? Again, I’m not saying that FreeBSD should ship without clang.


I am sure there are many who think that it should. And now (or in the future) we will have to deal with their broken installs.

What I am saying is that we should disallow:

`# pkg remove compiler`

Not because I am mean but because it would generate a mess of different installs. Freedom is great for those who don't need to provide support to others.


olli@ said:


> It’s unlikely that there will be any issues. What kind of issues would you expect? If a 3rd-party package requires ed, it has a dependency recorded and pulls in the ed package if needed, like with any other dependency.


For deployment it is nice to have a known base. It means in some cases, you don't even need to make a port or package. The fact that this base can now be minced up reduces the guarantee and either a big list of packages needs to be listed as a requirement or I need to make a formal port.

Consider if on Windows it was possible to randomly uninstall conhost or user32.dll. You would expect all 3rd party software to gracefully handle that? FreeBSD really isn't much different. Known configurations are nice.



olli@ said:


> We already do that for quite some time.


That is news to me. I'm running FreeBSD where if I install i.e zlib, I get the lib and includes. What OS are you running?


----------



## sidetone (Apr 8, 2021)

When I looked into pkg base, it looked like I had to compile my world to get it. It seemed to be useful for when someone wants to distribute that to similar machines. Maybe future use was intended what I thought it was, for base packages to be direct packages from FreeBSD, without the user needing to recompile world.

If that's the case, the install wouldn't be just the kernel. It would have many of the tools of a small system rescue CD.

I see LLVM and Clang staying in it, unless they're base packages. Everything is in C. Rust is a cool idea, but it didn't turn out as well as I thought it did for a system that's written in C. Rust makes sense for being in Redox. On FreeBSD, Rust programs run faster than I thought possible, but they have a parallel package build to FreeBSD's ports. I don't know if it could ever turn out, that there would be a slim FreeBSD in C without a Clang compiler, then there be two directions to go, to install Rust and only have Rust programs in packages (optionally bypassing ports), or to install Clang and go with ports, or have both. But then, you'd want Clang in base for everyone to at least recompile the kernel in C, if not everything else in base, or base packages.

The benefit of having a base package, would be, we have to wait for a new release to get a newer LLVM and Clang, when ports already require the latest versions. This would be a benefit of not having LLVM and Clang in base, and having it in base packages. There was an implication of having LLVM in base, but not Clang, but some ports may require that both be upgraded.

If base packages work like that, it would be good, as that system can be much more better than ports, and without Linuxism dependencies. For instance, base packages relying on bmake instead of gmake, or having OpenBSD's, NetBSD's or DragonFly's implementation of programs instead of GNU's or FreeDesktop's. Then, the rest in ports including Linuxisms can sit on top of a more efficient system, which ports will itself be structured more efficiently as a result.

Linux emulation would be better or more appealing to more users too, For emulation, I don't understand why there are two sets of program dependencies in ports: the one for everyone and the Linux emulated version. Shouldn't the emulation layer use everything it can from the ports and base that are used for everyone, and only have additions for what's lacking from the one FreeBSD offers in ports for everyone?


----------



## ralphbsz (Apr 8, 2021)

olli@ said:


> And clang – well, why shouldn’t it be optional?


Some of us are old enough to remember what happened when Sun for the first time shipped SunOS without a compiler, and then required paying of extra license fees to get a compiler. Actually, the correct term is not "remember", but "still feel the pain and are still mad". So removing the compiler from the most basic OS is likely to create a strongly emotional response.



> In fact, when I install FreeBSD on a space-constrained machine, I remove the  compiler tool chain, static libraries, include files and several other things. It would make sense to be able to de-select these in the installer if you don’t need them.


Completely agree. I'm running a few RPi0, and on those I would never think of doing a major compile, and disk space is at a premium, so not having the compiler and all the development libraries is sensible.

So I just gave a strong argument for keeping the compiler as mandatory, and one for making it optional. Resolving such contradictions is hard. Similar sets of arguments can be made for and against removing ftpd.

Ultimately, this will be a decision that's based on judgement. Part of that judgement is knowing history, knowing the personalities involved, knowing how people feel. For example: Berkeley without a campanile is not Berkeley. Should the campanile fall down in the next earthquake, it would need to be rebuilt. Not because it is useful, but because it defines what "Berkeley" means, together with other things (like Sproul plaza and Zellerbach hall). Similar arguments go for the Hoover tower, quad and memorial church and Stanford. And in the same fashion as a Usenix conference without Kirk and Eric is not a Usenix conference, a BSD without UFS and Sendmail is not BSD. That's not a technical argument (ZFS works just fine, and most people's need for a MTA is better met by other software), but a psychological or political one.


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> Not because I am mean but because it would generate a mess of different installs.


So, what problems exactly would you expect? We already have working dependency mechanisms in ports and packages.


kpedersen said:


> For deployment it is nice to have a known base. It means in some cases, you don't even need to make a port or package. The fact that this base can now be minced up reduces the guarantee and either a big list of packages needs to be listed as a requirement or I need to make a formal port.


Did you ever create a port? Adding a dependency isn't a huge thing to do, and they work transitively.


kpedersen said:


> Consider if on Windows it was possible to randomly uninstall conhost or user32.dll. You would expect all 3rd party software to gracefully handle that? FreeBSD really isn't much different. Known configurations are nice.


So, right now, FreeBSD is just like Windows regarding the base system, but unlike Windows has a well-working package management. Then your argument is, it should still keep the drawbacks of Windows?

I still didn't see a single valid argument what the problem would be. Maybe I have to repeat that a few more times: I already have a long list of `WITHOUT_XXX` knobs for my base. It doesn't create any problem.



sidetone said:


> The benefit of having a base package, would be, we have to wait for a new release to get a newer LLVM and Clang, when ports already require the latest versions. This would be a benefit of not having LLVM and Clang in base, and having it in base packages. There was an implication of having LLVM in base, but not Clang, but some ports may require that both be upgraded.


Not sure if that's what you're implying here, but individually upgrading packages in base isn't the point of pkg-base and shouldn't (and, I'm pretty sure, WON'T) be supported.


----------



## zirias@ (Apr 8, 2021)

ralphbsz said:


> Some of us are old enough to remember what happened when Sun for the first time shipped SunOS without a compiler, and then required paying of extra license fees to get a compiler. Actually, the correct term is not "remember", but "still feel the pain and are still mad". So removing the compiler from the most basic OS is likely to create a strongly emotional response.


You're really beating this strawman to death.
Making installation default, but optional, is something completely different than removing it from base.


----------



## sidetone (Apr 8, 2021)

ralphbsz said:


> Some of us are old enough to remember what happened when Sun for the first time shipped SunOS without a compiler, and then required paying of extra license fees to get a compiler. Actually, the correct term is not "remember", but "still feel the pain and are still mad". So removing the compiler from the most basic OS is likely to create a strongly emotional response.


I doubt anything like this will happen. It's more of a fear on something that happened from somewhere else. That wouldn't fit FreeBSD's business model anyway. Also, the base has a set standard to use certain types of licenses, pkg base will be like this too or more or less like this.

It would be cool, if the opensource compiler is always available in pkg base, then allow others especially companies to choose to use that or use a proprietary one. Of course, a BSD-like license would apply, so it would remain a choice.

If FreeBSD went that way of no longer offering the compiler in base packages or base, it would fall apart after losing lots of followers, and it would go against what FreeBSD was based on. It wouldn't be going with what everyone knows FreeBSD to be.




Zirias said:


> Not sure if that's what you're implying here, but individually upgrading packages in base isn't the point of pkg-base and shouldn't (and, I'm pretty sure, WON'T) be supported.



Maybe not. The only useful utility for that to happen would be for when LLVM or Clang or something like that needs to be updated before the next release comes about. If base pkg is set to remain at one version, which makes sense for most programs, it would be nice if the compiler and toolchain is the exception.

They have to work on multiple versions of the compiler as it is, and wait for new releases. It would be better, if they worked on one, and let this update as time goes on, instead of having incompatibilities from building different ports.


----------



## zirias@ (Apr 8, 2021)

Well, after years of work on it (and, maybe, more years to come), some day someone will have to admit: pkg-base, killed by FUD…

IIRC, pkg-base was worked on even before 12.0-RELEASE. It was never about doing that Linux distro crap of individual source packages for everything, but always just about building individual binary packages from the one large source tree. All it does is give the option to NOT install something – without having to build yourself.


----------



## kpedersen (Apr 8, 2021)

ralphbsz said:


> Completely agree. I'm running a few RPi0, and on those I would never think of doing a major compile, and disk space is at a premium, so not having the compiler and all the development libraries is sensible.


I'm not sure but don't a number of Perl, Python, Node.js dependencies compile up their native binaries during install / first use. Just installing something from PIP / CPAN will likely fail unless you have a C compiler installed.



Zirias said:


> Well, after years of work on it (and, maybe, more years to come), some day someone will have to admit: pkg-base, killed by FUD…


I wish. Sounds like it is going ahead. I personally think it is a bad decision. That said, I am still not impressed by the entirety of pkgng which seems to be an enabler of this.



Zirias said:


> Making installation default, but optional, is something completely different than removing it from base.


Really its not. At least if it is removed you know where it stands and you can make more educated guesses about peoples setups. Either way you will now have to explicitly state "make sure to have ABC installed"



Zirias said:


> Did you ever create a port? Adding a dependency isn't a huge thing to do, and they work transitively.


Do you add a dependency on xz to extract the package itself? Can I perhaps just add one dependency called "base"? This whole situation is very disappointing.



Zirias said:


> All it does is give the option to NOT install something – without having to build yourself.


Time will tell. It sounds like a clusterfsck to me


----------



## zirias@ (Apr 8, 2021)

kpedersen said:


> Do you add a dependency on xz to extract the package itself?


Yes, see https://cgit.freebsd.org/ports/tree/Mk/Uses/tar.mk#n11
It's as simple as `USES= tar:xz`.


kpedersen said:


> Can I perhaps just add one dependency called "base"? This whole situation is very disappointing.


I'm still convinced you just don't understand how it works. For the example above, nothing would have to change in the port. `USES= tar:xz` would just automatically add that dependency on the required base package.


----------



## Jose (Apr 8, 2021)

ralphbsz said:


> Some of us are old enough to remember what happened when Sun for the first time shipped SunOS without a compiler, and then required paying of extra license fees to get a compiler. Actually, the correct term is not "remember", but "still feel the pain and are still mad". So removing the compiler from the most basic OS is likely to create a strongly emotional response.


Sunsite! I forget the full URL, but that's where you had to go to make your shiny new Sun box useful.


----------



## Jose (Apr 8, 2021)

kpedersen said:


> In all fairness, I am fine with ralphbsz's suggestion. Even if all the packages in base are specifically tailored for him, I would rather that even for my use-case. So long as base *is* a base rather than a scatty ever changing collection of packages.


Devil's advocate argument. It already is. It's just hidden from you by the fact that a bunch of sources are downloaded and checked into the src repo.

Is it better to do it that way or to have a bunch of packages that ultimately do the same thing? I honestly don't know. I see the value of being able to remove the compiler toolchain on memory-, storage-, and CPU- constrained environments, but I've also felt the frustration of not having a compiler installed by default.

My weakly held opinion is that it is better to have something like the src repository that is compiled, integrated, and tested as a unit to provide a sane foundation for doing computer things. I see the value of tweaking what's in that src repository to suit your needs too.

Addressing the meta-argument - yes, we're bikeshedding here, but it's fun bikeshedding, dammit!


----------



## PMc (Apr 8, 2021)

Do we get `gitlite` (or `igitt`) in base?
From how it appears to me, we cannot build a kernel without, because we do not get a version string into `uname`. But then, git requires 89 prerequiste ports - and I am not really into installing all these into a bhyve before building a kernel...


----------



## mark_j (Apr 8, 2021)

PMc said:


> Do we get `gitlite` (or `igitt`) in base?
> From how it appears to me, we cannot build a kernel without, because we do not get a version string into `uname`. But then, git requires 89 prerequiste ports - and I am not really into installing all these into a bhyve before building a kernel...


Don't know them but git won't because of the license, unless core  reverse their approach on gpl.


----------



## VladiBG (Apr 8, 2021)

What next? Removing "ifconfig" like in other OS and making it part of net-tools package? Or remove the "ping" you don't need it to boot. I personally prefer some tools to be preinstalled on the OS like sshd, ifconfig, telnet, ftp,trace,ping and so on... you never know when you going to need them but it's better to have them as part of the OS.


----------



## shkhln (Apr 8, 2021)

VladiBG said:


> prefer some tools to be preinstalled on the OS like sshd, ifconfig, telnet, ftp,trace,ping and so on... you never know when you going to need them but it's better to have them as part of the OS.


Can we get aircrack-ng in this list?


----------



## VladiBG (Apr 8, 2021)

shkhln said:


> Can we get aircrack-ng in this list?


Right after IEEE 802.11ax


----------



## zirias@ (Apr 8, 2021)

PMc said:


> From how it appears to me, we cannot build a kernel without, because we do not get a version string into `uname`.


Nonsense, the only thing that's unavailable for uname is a commit count and hash. Now why is _that_ a problem?



PMc said:


> But then, git requires 89 prerequiste ports - and I am not really into installing all these into a bhyve before building a kernel...


Ever heard of a tiny (sic) something called flavors?


----------



## zirias@ (Apr 8, 2021)

VladiBG said:


> What next? Removing "ifconfig" like in other OS and making it part of net-tools package?


Does anyone actually _read_ anything before spitting out polemics?

pkg-base has _nothing_ to do with _removing_ things from base. And btw:


			
				https://cgit.freebsd.org/src/tree/sbin/ifconfig/Makefile?h=releng/13.0#n6 said:
			
		

> PACKAGE=runtime



Not installing the base "runtime" package would effectively mean not to install base at all.


----------



## VladiBG (Apr 8, 2021)

You know which Linux Distribution i refer to...


----------



## zirias@ (Apr 8, 2021)

VladiBG said:


> You know which Linux Distribution i refer to...


No. But does it really matter? pkg-base has nothing to do with splitting base in parts – it will always be the single project it is today. Building separate binary packages from this one source tree is a very different thing (and nothing you will ever find with any Linux system).


----------



## kpedersen (Apr 8, 2021)

Jose said:


> Addressing the meta-argument - yes, we're bikeshedding here, but it's fun bikeshedding, dammit!


Oh, very likely! 



Jose said:


> I see the value of tweaking what's in that src repository to suit your needs too.


Agreed, but this already does exist. You can mince with the build scripts as you compile a custom FreeBSD userland. It sounds like Zirias already does this. Yes, granted it is currently harder than just rinsing some random packages from the install but honestly you will not want a beginner doing this anyway.

At the very least when I list all my installed packages, I don't want the output to be clogged up with packages I know are there because... well of course they are, they are in base.



Zirias said:


> It's as simple as `USES= tar:xz`.


And will you now need to do similar for sh? To be able to run the ./configure script?
You may find out that this list of USES you need to spam in the port will get quite long.

CMake at runtime uses a lot of hidden things in base you may not realize. Your port will now have to cater for those mad men who decided not to install a proper base.


----------



## olli@ (Apr 8, 2021)

kpedersen said:


> That is news to me. I'm running FreeBSD where if I install i.e zlib, I get the lib and includes. What OS are you running?


These are the distribution sets of FreeBSD 13.0-RC5 amd64:

```
-rw-r--r--  1  ftp  ftp   192282016  Apr 02 07:23  base-dbg.txz
-rw-r--r--  1  ftp  ftp   189152220  Apr 02 07:23  base.txz
-rw-r--r--  1  ftp  ftp    89978588  Apr 02 07:23  kernel-dbg.txz
-rw-r--r--  1  ftp  ftp    42371308  Apr 02 07:23  kernel.txz
-rw-r--r--  1  ftp  ftp    19242368  Apr 02 07:23  lib32-dbg.txz
-rw-r--r--  1  ftp  ftp    69986184  Apr 02 07:23  lib32.txz
-rw-r--r--  1  ftp  ftp    41361196  Apr 02 07:23  ports.txz
-rw-r--r--  1  ftp  ftp   160365292  Apr 02 07:23  src.txz
-rw-r--r--  1  ftp  ftp    13861380  Apr 02 07:23  tests.txz
```
The ones with `-dbg` in the name contain libraries for development that have debug symbols and profiling information, the others don’t. The installer asks you which of the distribution sets you want. Only the “kernel” and “base” sets are mandatory, the others are optional (and this won’t change with pkg-base).
FreeBSD has this distinction since day one (although the naming and layout changed over time).


----------



## zirias@ (Apr 8, 2021)

olli@ said:


> These are the distribution sets of FreeBSD 13.0-RC5 amd64:


Ah, yes. I think the confusion here is: it's currently not possible to do for ports (short of workarounds like slave ports). Still, the need is there, especially if a full library (or even compiler!) package is a _lot_ larger than just the set of runtime libraries actually needed.


----------



## shkhln (Apr 8, 2021)

Zirias said:


> Still, the need is there, especially if a full library (or even compiler!) package is a _lot_ larger than just the set of runtime libraries actually needed.


Currently stuck at https://reviews.freebsd.org/D16457.


----------



## hruodr (Apr 8, 2021)

olli@ said:


> Seriously – dd is required by the standard boot procedure



We read above: standard. I am not a developer as olli@, but I have made some non-standard
installations that would have been much more difficult not having standard tools, in which it would
have been a dream to be able to just type `pkg add xxx`. I think it was an error to remove `slattach`,
although a lot of people would say it was obsolete, insecure, primitive, etc.



ralphbsz said:


> And in the same fashion as a Usenix conference without Kirk and Eric is not a Usenix conference, a BSD without UFS and Sendmail is not BSD.


And UFS is a good filesystem and sendmail not worse than other MTAs. And sure, sendmail is used out
of custom, perhaps only because it is there as default, but this is not a problem, it is an advantage: one
puts energy to learn an MTA and then it is easy to use and configure. One puts energy to learn a
standard FreeBSD. FreeBSD becomes then a concept of a full OS, not of a pkg-supermarket with products
for every taste. Learning, the beginning, needs always some energy. Those that want other MTA may install it.

Those that are creative and really want to do something better, should write an new OS and leave FreeBSD
as FreeBSD. Then we have more, not less. The Unix creators did that with plan9, and plan9 is a wonderful
OS, conceptually better than Unix.


----------



## kpedersen (Apr 8, 2021)

olli@ said:


> FreeBSD has this distinction since day one (although the naming and layout changed over time).


Ah I was referring to -dev packages not -dbg. So for example on Linux you install zlib but you don't get the C headers. You would need to install zlib-dev or zlib-devel for that.

In FreeBSD we have the base.txz which includes development files as well. For example we don't need base-dev.txz


----------



## PMc (Apr 8, 2021)

Zirias said:


> Nonsense, the only thing that's unavailable for uname is a commit count and hash. Now why is _that_ a problem?


That's about the equivalent of travelling without passport.


----------



## zirias@ (Apr 8, 2021)

So, saying nothing with a broken analogon?

Uname will always have the exact release name (13.0-RC5-p1 …), the build host and the build timestamp. What exactly do you need a commit hash and count for?

The only _possible_ use case would be on a "stable" branch or on main/current. If you follow one of _these_, it always made sense to get your source directly from the repository, git didn't change anything about that.


----------



## PMc (Apr 8, 2021)

Zirias said:


> So, saying nothing with a broken analogon?


It's precisely the matching analogon. As some people are indeed promoting the opinion that identification is superfluous, and anybody should be free to travel anywhere.



Zirias said:


> Uname will always have the exact release name (13.0-RC5-p1 …), the build host and the build timestamp. What exactly do you need a commit hash and count for?


Currently I dont see neither build host nor timestamp (I don't want to see these, anyway):

```
# uname -a
FreeBSD  12.2-RELEASE-p6 FreeBSD 12.2-RELEASE-p6#9f00856c417 GENERIC  amd64
```
Before, it used to look like this:

```
$ uname -a
FreeBSD  12.2-RELEASE-p5 FreeBSD 12.2-RELEASE-p5 #0 r369523M#N1250: Sat Mar 27 [etc.etc.]
```
The thing _starting at the last hashmark_ is a local add-on and reports the local patchset (you don't find that one in the official repo)! Without that the uname would now probably look like this:

```
# uname -a
FreeBSD  12.2-RELEASE-p6 FreeBSD 12.2-RELEASE-p6 GENERIC  amd64
```



Zirias said:


> The only _possible_ use case would be on a "stable" branch or on main/current.


Yes, then it gets really bad. Then you will gather bugreports from people and you will have no idea what code they might actually be running. And probably not even an easy way to figure that out after the fact (there are no $Id: anymore going into the code).
Oh, I forgot: you have "reproducible builds" now. So you can simply build a kernel from every commit, and compare `cksum` until you find the matching one.

But that's all not a problem with me, this is just fine along the Golgafrincham storyline. What really annoys me is that there now `12.2-RELEASE-p6#9f00856c417` is no space in between. That is ugly. What I would want to see is this:
`FreeBSD  12.2-RELEASE-p6 FreeBSD 12.2-RELEASE-p6 81526c74d9c#9f00856c417 GENERIC  amd64`


----------



## zirias@ (Apr 8, 2021)

PMc said:


> It's precisely the matching analogon. As some people are indeed promoting the opinion that identification is superfluous, and anybody should be free to travel anywhere.


No, it's silly. 12.2-RELEASE-p6 is all the identification you need, there's only one commit linked to it, short of


PMc said:


> local patchset


yeah, right – then add whatever you want to mark that. So, where exactly would be the problem with NOT using git? You didn't get a revision number in uname either when not using svn before.


PMc said:


> Yes, then it gets really bad. Then you will gather bugreports from people and you will have no idea what code they might actually be running.


I don't think anyone who isn't able to _either_ use git on a default branch _or_ track what exactly he built himself would be qualified submitting any helpful bugreport about CURRENT. Same thing, nothing substantial changed compared to svn.

This all looks like a lot of "I don't like change" complaints.

Just a side note, I wonder how you build your system to get `uname -a` output like this. Mine for example looks like this (and always did, on 12.x, built from subversion, ...)

```
FreeBSD nexus.home.palmen-it.de 13.0-RC5 FreeBSD 13.0-RC5 #14
config-n244728-2f274dd5c94: Fri Apr  2 15:52:15 CEST 2021
root@builder.home.palmen-it.de:/usr/obj/usr/src/amd64.amd64/sys/DESKTOP
amd64
```
Newlines added for readability, the only thing that changed with GIT is the `config-n244728-2f274dd5c94` part. There was only a revision number before, now I have `<branch>-<count>-<hash>`. Yes, `config` is a local branch here, containing my kernel configurations.


----------



## olli@ (Apr 8, 2021)

Very interesting.
For example, this is one of my boxes (yes, I still have 32bit machines):

```
$ uname -a
FreeBSD myhost.mydomain 12.2-STABLE-20210131 FreeBSD 12.2-STABLE-20210131 #0: Tue Feb  2 21:13:06 CET 2021     olli@myhost.mydomain:/usr/obj/usr/src/i386.i386/sys/CANTARO  i386
```
The date stamp `-20210131` is a local addition of mine (via `$BRANCH_OVERRIDE` that is picked up by `make kernel`). The rest is standard. When FreeBSD used Subversion, I also had the SVN revision number in there because it was useful for me. Unfortunately, Git doesn’t support revision numbers, and the Git hashes are useless, so I don’t put them in uname.
 
Most of the time, the information from `uname -r` is all I need; `uname -a` is much too verbose.

```
$ uname -r
12.2-STABLE-20210131
```
Actually I have a shell function that calls `uname -rsm` when I enter just `uname` without arguments. Output looks like this:

```
$ uname
FreeBSD 12.1-STABLE-20200612 amd64
$ grep uname ~/.zshrc
uname () { if (( $# == 0 )); then set -- -rsm; fi; command uname "$@"; }
```


----------



## zirias@ (Apr 8, 2021)

olli@ said:


> Git doesn’t support revision numbers, and the Git hashes are useless, so I don’t put them in uname.


Actually, they serve the same purpose. E.g. assuming you have a hash `b7fbdb5042c` in your uname output (a commit that actually exists on releng/13.0), you could inspect exactly what changed since then with `git diff b7fbdb5042c`.

FreeBSD also adds a "commit count", seemingly just to have this increasing number ppl were used to, but _that_ I'd consider somewhat useless, as it can't identify a commit 

*edit*: BTW, _your_ `uname -a` output looks like mine. So, still wondering what PMc is doing here 



olli@ said:


> Actually I have a shell function that calls `uname -rsm` when I enter just `uname` without arguments.


THAT looks like a nice idea! I'd personally add `-i` flag, having multiple kernel configs, I might want to verify the correct one is installed.


----------



## olli@ (Apr 8, 2021)

kpedersen said:


> Ah I was referring to -dev packages not -dbg. So for example on Linux you install zlib but you don't get the C headers. You would need to install zlib-dev or zlib-devel for that.
> 
> In FreeBSD we have the base.txz which includes development files as well. For example we don't need base-dev.txz


Ok, I see, so we were talking about slightly different things.
 
Actually, the naming conventions are somewhat unfortunate. Strictly speaking, the *-dbg* sets are intended to be used by developers (that’s why I called them development sets). They are not required for normal users who just want to build stuff (e.g. /usr/src or /usr/ports). In contrast, the *-dev* packages are required when building things, so they’re not only for developers, but also for normal users, unless you install binary packages only and never build anything yourself.
 
By the way, there are a few nasty things. For example, there are binary packages that compile some stub files during installation. These require a compiler, even though they are binary packages. If the compiler is optional (at some point in the future, _not_ yet with pkg-base), then such packages will have a dependency on the compiler package. We already have this today for ports that require a certain version of the compiler that might be different from the version in base.


----------



## olli@ (Apr 8, 2021)

Zirias said:


> Actually, they serve the same purpose. E.g. assuming you have a hash `b7fbdb5042c` in your uname output (a commit that actually exists on releng/13.0), you could inspect exactly what changed since then with `git diff b7fbdb5042c`.


I know that, but my use case for SVN revision numbers depended on the fact that they are consecutive. That’s not the case for Git hashes.
 


> FreeBSD also adds a "commit count", seemingly just to have this increasing number ppl were used to, but _that_ I'd consider somewhat useless, as it can't identify a commit


Oh, I didn’t know about that. I’ll have to look how to get at that count. Thanks for the hint.


----------



## zirias@ (Apr 8, 2021)

olli@ said:


> my use case for SVN revision numbers depended on the fact that they are consecutive.


So, there actually _is_ a use case for some 

Well, in my output above, in `config-n244728-2f274dd5c94`, `n244728` is the revision count. It counts locally, so it will only work correctly with a full clone. I guess that's one of the reasons shallow clones aren't officially recommended


----------



## PMc (Apr 8, 2021)

Zirias said:


> No, it's silly. 12.2-RELEASE-p6 is all the identification you need, there's only one commit linked to it, short of


In this case, yes.



Zirias said:


> yeah, right – then add whatever you want to mark that. So, where exactly would be the problem with NOT using git? You didn't get a revision number in uname either when not using svn before.


svn was in base.

But then, even *IF* git were in base (which, as was mentioned here, does not work because it comes from smelly fish-eating bird), you would not easily get a useful revision number from it.

Because: it considers a checkout containing local patches as just fine for the purpose, and consequentially switches to "reproducibe build". *IF* git were installed, it might just spit out the 9f00856c417 commit-id and be done with it - which is a local one, and not to be found in the official repo!



Zirias said:


> I don't think anyone who isn't able to _either_ use git on a default branch _or_ track what exactly he built himself would be qualified submitting any helpful bugreport about CURRENT. Same thing, nothing substantial changed compared to svn.


It concerns STABLE also.



Zirias said:


> This all looks like a lot of "I don't like change" complaints.


Do that to whomever, but not to me. It doesn't work here.

I spent time getting that git thing to do something useful, and it's troublesome enough.



Zirias said:


> Just a side note, I wonder how you build your system to get `uname -a` output like this. Mine for example looks like this (and always did, on 12.x, built from subversion, ...)


What's different between mine and that? I have no hostname configured, so 2nd field is empty. git is not isntalled, so there is no commit-id from there - but my own add-on fetches and inserts that.

And it seems You have set `WITHOUT_REPRODUCIBLE_BUILD` in /etc/src.conf, so that the date-stamp+email+localpath gets added even without the checkout being dirty.


----------



## zirias@ (Apr 8, 2021)

PMc said:


> svn was in base.


So what? git-tiny is a tiny package. Complaint-mode?


PMc said:


> But then, even *IF* git were in base (which, as was mentioned here, does not work because it comes from smelly fish-eating bird), you would not easily get a useful revision number from it.


You get both, a revision count (what's _your_ usecase here?) AND a commit hash.


PMc said:


> Because: it considers a checkout containing local patches as just fine for the purpose, and consequentially switches to "reproducibe build".


I don't know what you do to your source, but "reproducible build" is always off by default:





						kern.opts.mk « conf « sys - src - FreeBSD source tree
					






					cgit.freebsd.org
				





PMc said:


> *IF* git were installed, it might just spit out the 9f00856c417 commit-id and be done with it - which is a local one


Which is exactly the expected behavior. You're building from a local commit, so if you can't identify what you built with that, there seems to be a different problem.


PMc said:


> Do that to whomever, but not to me. It doesn't work here.


Complaining about change is all you're doing here. At least as long as you can't explain what's the actual _problem_ with things you complain about. Your wording above is also a strong indicator.


PMc said:


> And it seems You have set WITHOUT_REPRODUCIBLE_BUILD in /etc/src.conf


No. This is off by default. But good to know that's the option leading to your unusual output.


----------



## olli@ (Apr 8, 2021)

Zirias said:


> I don't know what you do to your source, but "reproducible build" is always off by default:
> 
> 
> 
> ...


Only in HEAD and 13. It is on by default in FreeBSD 12.


----------



## zirias@ (Apr 8, 2021)

olli@ said:


> Only in HEAD and 13. It is on by default in FreeBSD 12.


Huh? Then why did I never see such a shortened _uname_? I didn't have any local patches on 12.2. Were additional kernel configs enough to disable it?


----------



## PMc (Apr 8, 2021)

I


Zirias said:


> Actually, they serve the same purpose. E.g. assuming you have a hash `b7fbdb5042c` in your uname output (a commit that actually exists on releng/13.0), you could inspect exactly what changed since then with `git diff b7fbdb5042c`.


Only if you manage to still find that hash. That means, if it still exists.
I noticed some interesting things with rebase and tags... 

So, problem one: that hash is the commit-id from the tip of the branch. This might be the commit of your local patches on top of the release - then this hash is not to be found anywhere else than in your local repo clone!
That's already bad enough - we would like to also have included in uname the hash from the branch-off commit, i.e. the last official commit contained in this checkout (because that one should stay constant).

But it comes worse: as soon as you do rebase your local patches on top of the official branch (due to some security fix or whatever), the commits are rewritten anew, and this commt id hash *is gone*! (It actually is not really gone, you can find it lingering somewhere in the repo, but it's no longer referenced).

So, commit id hash is very useless for documentation. And this again is caused by the design weakness of git. 
We already agreed that git cannot provide us with monotonously increasing commit numbers, by design.
We already agreed that git cannot provide us with joint repositories where multiple similar projects could be maintained synchronously, by design.
Now we learn that git cannot easily provide us with solid tags for documentation purposes. The reason appears to be that git is not designed for such. It is NOT a database (a database would allow history-rewriting, but would do so only with strict cross-logging all previous and changed parameters). It is NOT EVEN a journal (a journal would strictly not allow to rewrite history). 
Instead, it is just some kind of tree structure, similar to a filesystem, where you can do anything with, and always have only an ad-hoc layout. Git is not _used_ as a VCS, it is _abused_ as a VCS.



Zirias said:


> *edit*: BTW, _your_ `uname -a` output looks like mine. So, still wondering what PMc is doing here


I'm doing nothing, this is the standard _*reproducible build*_ that you now get without asking.
Both of You seem to have that switched off - usually this is done by configuring `WITHOUT_REPRODUCIBLE_BUILD` to make.


----------



## zirias@ (Apr 8, 2021)

Oh, a long rant, how nice.

Fact one: You have the _option_ to have local branches/commits now, you're not obliged to use them. *If* you use them, putting your own commit hashes in `uname` output is the only sane thing to do. With subversion, you just didn't have the option.

Fact two: Whether you rebase your original branch or a copy of it is _your_ decision.



PMc said:


> And this again is caused by the design weakness of git.


The typical arrogant nonsense from a person who doesn't understand the design.


----------



## PMc (Apr 8, 2021)

Zirias said:


> So what? git-tiny is a tiny package. Complaint-mode?


Tiny package with some 89 build-prereqs, with most options turned off.



Zirias said:


> I don't know what you do to your source, but "reproducible build" is always off by default:
> 
> 
> 
> ...


No idea what you have here or from which version that is (and that's exactly what I am talking about all the time: in git you just don't find the relevant matters), but in my 12.2 Release source it is always *ON* by default (and I didn't patch that).


----------



## zirias@ (Apr 8, 2021)

PMc said:


> No idea what you have here or from which version that is (and that's exactly what I am talking about all the time: in git you just don't find the relevant matters),


It's extremely obvious, just have a look at the URL. This is getting ridiculous, only nonsensical complaints. I guess I'm out of here, the mailing lists and IRC channels are a much better place nowadays.


----------



## PMc (Apr 8, 2021)

Once again, as thats for the engineering demand:


PMc said:


> we would like to also have included in uname the hash from the branch-off commit, i.e. the last official commit contained in this checkout (because that one should stay constant).



So how do we get that into `uname`?



Zirias said:


> It's extremely obvious, just have a look at the URL. This is getting ridiculous, only nonsensical complaints. I guess I'm out of here, the mailing lists and IRC channels are a much better place nowadays.


That figures. If the rough winds of serious engineering demands don't suit, then just withdraw into some filter-bubble.


----------



## zirias@ (Apr 8, 2021)

Does it get any more moronic?

You're complaining about GIT, giving you NEW features, and then request something without showing any need for it that was never possible before because SVN didn't even offer the prerequisites to get there. All for miserably failing to understand how these things work with GIT.

Yes, I won't answer on your "content" any more.


----------



## mickey (Apr 8, 2021)

Zirias said:


> It's extremely obvious, just have a look at the URL.



Use the source, Luke!


----------



## zirias@ (Apr 8, 2021)

mickey said:


> Use the source, Luke!


Yep, that's what olli@ already mentioned. Leaves the question: What disables it, short of explicitly doing so? Is some added kernel config enough? That's strange, but wasn't the "issue" here (although, OTOH, there is no real "issue" here anyways…)


----------



## olli@ (Apr 8, 2021)

Zirias said:


> Huh? Then why did I never see such a shortened _uname_? I didn't have any local patches on 12.2. Were additional kernel configs enough to disable it?


When `REPRODUCIBLE_BUILD` is enabled (default on FreeBSD 12), it passes the option `-R` to the `newvers.sh` script during the kernel build. This option is documented thusly:

```
#     -R               Reproducible build if the tree represents an unmodified
#                      checkout from a version control system.  Metadata is
#                      included if the tree is modified.
```


----------



## zirias@ (Apr 8, 2021)

Hm, not sure this does "the right thing" then, seems to consider two added files to sys/amd64/conf not under version control a "modified tree", even when building GENERIC.

But anyways, doesn't really bother me


----------



## olli@ (Apr 8, 2021)

Zirias said:


> Hm, not sure this does "the right thing" then, seems to consider two added files to sys/amd64/conf not under version control a "modified tree", even when


Well, basically it runs `svnversion` and/or `git diff-index` in order to find out.
If you’re curious, you can read all the details in /sys/conf/newvers.sh (look for the variable `modified`).


----------



## Jose (Apr 8, 2021)

PMc said:


> Only if you manage to still find that hash. That means, if it still exists.


The commit (and therefore its hash) will always exist if it's in the history of any branch or tag. Orphan commits will linger until Git does garbage collection.


PMc said:


> So, problem one: that hash is the commit-id from the tip of the branch. This might be the commit of your local patches on top of the release - then this hash is not to be found anywhere else than in your local repo clone!


Well, if you rebase onto a remote-tracking branch, your local history will always have a common ancestor with the upstream branches.


PMc said:


> But it comes worse: as soon as you do rebase your local patches on top of the official branch (due to some security fix or whatever), the commits are rewritten anew, and this commt id hash *is gone*! (It actually is not really gone, you can find it lingering somewhere in the repo, but it's no longer referenced).


You shouldn't rely on these existing. They will be garbage collected eventually.


PMc said:


> So, commit id hash is very useless for documentation. And this again is caused by the design weakness of git.


We can agree to disagree on this, but commit hashes have a very interesting property: they are globally unique. They identify the state of the source tree at a particular point, and cannot be forged. If you like database analogies, this makes them a natural primary key for the source at a particular point. Revision numbers as used by CVS and Subversion are surrogate keys.


PMc said:


> We already agreed that git cannot provide us with monotonously increasing commit numbers, by design.
> We already agreed that git cannot provide us with joint repositories where multiple similar projects could be maintained synchronously, by design.
> Now we learn that git cannot easily provide us with solid tags for documentation purposes. The reason appears to be that git is not designed for such. It is NOT a database (a database would allow history-rewriting, but would do so only with strict cross-logging all previous and changed parameters). It is NOT EVEN a journal (a journal would strictly not allow to rewrite history).
> Instead, it is just some kind of tree structure, similar to a filesystem, where you can do anything with, and always have only an ad-hoc layout. Git is not _used_ as a VCS, it is _abused_ as a VCS.


I agree that Git is not for everyone. I personally like it and find using other RCS torture, but I can see how using Git can be a pain.

I find that it follows the Unix philosophy in that it will not try to protect you from the consequences of your actions. It will cheerfully do very destructive things if you ask it to because the assumption baked into the tool is that you know what you're doing. Yes, I've burned my fingers with it, but it's still my kind of tool.


----------

