# Why was FreeBSD 5.x so bad?



## neel (Oct 4, 2020)

I have heard from many veteran FreeBSD users that FreeBSD 5.x was the "worst" FreeBSD release after a relatively solid 4.x.

But what made 5.x so bad, when compared to say 4.x if not later releases?

For reference, I am young (only 23!) and was a child during the FreeBSD 5.x days. I started using FreeBSD daily during my high school days (at 15).


----------



## msplsh (Oct 4, 2020)

FreeBSD version history - Wikipedia
					






					en.wikipedia.org


----------



## ralphbsz (Oct 4, 2020)

Good question. One part I remember was that 5.x was the first version where the kernel became SMP-capable, meaning multiple threads could be in the kernel at once. I suspect that this caused lots of hilarity: locking is hard.

Just to make you feel too young: I just logged into a 4.x machine that is still running. The kernel was last recompiled in February 2008. Works fine.


----------



## Beastie7 (Oct 4, 2020)

I think this was around when Dillon made his tantrum fork as well. What was that all about anyway? I can’t seem to find the mailing list thread about it.


----------



## ralphbsz (Oct 4, 2020)

Hmm, Matt Dillon. The version of the story that I heard (admittedly from BSD people, who were on the other side) was that he had a big ego, big ideas about how the kernel should be structured (threads versus fibers, locking granularity, who controls the blocks of the file system, parallel access to files), and was unwilling to listen to reason, or to the opinion of other people. Fundamentally, he thought that he was much smarter than everyone else, and insisted on committing code that was done "his way". That caused a split, where he either lost or voluntarily gave up commit privileges on FreeBSD. I know that he was either talking to or working with the BtrFS people (who have their origins in the big computer companies). And I heard that the people who actually implement super-fast file systems (in the massively parallel systems community) were laughing their heads off about these rank amateurs (like the Hammer or BtrFS people). But a lot of this could be parochialism and us-vs-them.


----------



## xtouqh (Oct 4, 2020)

FreeBSD 5 was bad because it *had* to be released no matter how incomplete the changes were, otherwise waiting for everything to be fixed would mean no release ever.


----------



## a6h (Oct 5, 2020)

Good thread. I wasn't aware of all of these. FreeBSD 6.22 was my first installation. Good to know what was going on.



xtouqh said:


> it *had* to be released no matter how


This part I can't understand this part. Why they *had* to release a dubious product? Does release dates are set in stone?


----------



## Beastie7 (Oct 5, 2020)

ralphbsz said:


> Hmm, Matt Dillon. The version of the story that I heard (admittedly from BSD people, who were on the other side) was that he had a big ego, big ideas about how the kernel should be structured (threads versus fibers, locking granularity, who controls the blocks of the file system, parallel access to files), and was unwilling to listen to reason, or to the opinion of other people. Fundamentally, he thought that he was much smarter than everyone else, and insisted on committing code that was done "his way". That caused a split, where he either lost or voluntarily gave up commit privileges on FreeBSD. I know that he was either talking to or working with the BtrFS people (who have their origins in the big computer companies). And I heard that the people who actually implement super-fast file systems (in the massively parallel systems community) were laughing their heads off about these rank amateurs (like the Hammer or BtrFS people). But a lot of this could be parochialism and us-vs-them.



So much hostility and fragmentation over pride. That’s really sad. Thanks for this.


----------



## richardtoohey2 (Oct 5, 2020)

There's a bit about the developer departure here: https://lwn.net/Articles/712308/

My recollection of that time (wasn't a FreeBSD user then, but kept an eye on things) was more along the lines of the comment here:






						FreeBSD Quality
					

Richard Bejtlich's blog on digital security, strategic thought, and military history.




					taosecurity.blogspot.com
				




_Regarding the mail thread: FreeBSD 5 was a revolution in many areas comparing to 4 and as such was not as polished around the edges as people was expecting. As Scott Long stressed many times, FreeBSD 6 release management will be diferent that 5's, having less features, more testing, more refinement._

So my recollection - very well might be faulty! - is that FreeBSD and especially 4.X - had a reputation of being *rock solid*.  5.X incorporated a lot of changes and the quality (and reputation) slipped.


----------



## ralphbsz (Oct 5, 2020)

anonymous9 said:


> Nobody is laughing at Matt Dillon as a rank amateur for creating HAMMER.


At the time of its inception, Hammer was considered very amusing (in a bad sense) by some people. And those are not random people, but they write file systems for a living (get paid for it).



> And the rank amateurs who thought of and began development of btrfs were IBM (proposal) and implementation began by a SUSE engineer ...


You are correct about one part: the person who first suggested the modifiable B-tree technique is not a rank amateur at all, and I do respect him (and know him well, having worked directly with him). I did express myself rather badly above, and apologize.  The actual implementation of the file system is a different story ... still today, I refer to it as "a machine for destroying data", but at least it doesn't murder wives.

And just to be clear: There are lots of good file systems, even single-node single-volume file systems, with smart people who are good engineers working on them. That definitely includes ZFS, UFS, ext<n>, XFS. It's just that not all file systems that being written are good, and when you look inside you see too much hype, derivative work, and bad judgement.


----------



## xtouqh (Oct 5, 2020)

vigole said:


> This part I can't understand this part. Why they *had* to release a dubious product? Does release dates are set in stone?


AFAIK, it was endlessly delayed for one or another feature to be finished, and even if release dates are not set in stone, not releasing something for extended periods of time would only mean death of the project, so there was a choice, bad as it was.


----------



## SKull (Oct 5, 2020)

ralphbsz said:


> but at least it doesn't murder wives.


I'm so sorry, but that one made me laugh


----------



## vermaden (Oct 5, 2020)

neel said:


> I have heard from many veteran FreeBSD users that FreeBSD 5.x was the "worst" FreeBSD release after a relatively solid 4.x.
> 
> But what made 5.x so bad, when compared to say 4.x if not later releases?
> 
> For reference, I am young (only 23!) and was a child during the FreeBSD 5.x days. I started using FreeBSD daily during my high school days (at 15).


I started with 5.x (5.4 to be precise) when I moved from Linux so I never had a chance to rally know 4.x series. Maybe that is why I do not see anything 'tragic' in 5.x series.


----------



## a6h (Oct 5, 2020)

FreeBSD 6.22 was my first time! Reading this thread, I think I've missed a lot of epic stories.


----------



## cynwulf (Oct 5, 2020)

richardtoohey2 said:


> There's a bit about the developer departure here: https://lwn.net/Articles/712308/


The author seems biased, seems to be very much pro code of conduct (Coc) and seems to claim that a CoC would have prevented all that.


----------



## msplsh (Oct 5, 2020)

The short of it is that getting rid of GIANT_LOCK was difficult and 5.x "had to" be released.

Has nothing to do with HAMMER or CoC.


----------



## hitest (Oct 5, 2020)

I started using FreeBSD with 5.x and I didn't know what I was doing.  I really liked FreeBSD 6.x.  I remember compiling KDE 3.5x on my old IBM 300PL that had 256 MB RAM.  That was painful and it took several days. Heh.


----------



## Jose (Oct 5, 2020)

vigole said:


> This part I can't understand this part. Why they *had* to release a dubious product? Does release dates are set in stone?


Once upon a time I worked at a company that didn't release a product from the main branch in the 2.5 years I worked there. Work was done on the main branch, but then the changes that were deemed not too risky were backported into the two supported release branches and released periodically.

What happens then is you wind up with a branch that has all the major, important, and necessary changes, but is not thoroughly vetted through the release process; and one or more release branches full of old bugs and design flaws, but with the latest important medium to minor fixes.

You then have two choices: bite the bullet, release the main branch and buckle down for months or likely years of pain & suffering, or decide one of your release branches is the new main branch and abandon all the work done in the old main branch.

There's really no clearly best choice in this situation, it really depends on the particulars of the code in question. Door 1 is the correct choice given no more information about the code base under discussion, and this is the option Freebsd chose.

In case you're wondering, I heard from friends the company I used to work for also chose door 1, but they did it in the worst possible way. Thanks to input from marketing, the main branch was released as a dot release. Countless customers, some of them very large, installed it as a dot-upgrade in production systems. I wonder how much more market share the company lost thanks to that genius move.

It wasn't like there were a lot of options in that niche field, though. Our only real competitor chose door number 2. This led to some pretty hilarious bugs being perpetuated for years. I wonder who's laughing now, but don't care enough to find out.


----------



## Jose (Oct 5, 2020)

anonymous9 said:


> Thanks for a bunch of opinion based admittedly on what was heard from one side and opinion without facts on filesystems?  Okay, whatever.


So give us the other side. Curiosity is my only agenda. I was busy paying a mortgage and raising little kids at the time. I had no time to keep up with these events.


----------



## cynwulf (Oct 6, 2020)

Jose said:


> You then have two choices: bite the bullet, release the main branch and buckle down for months or likely years of pain & suffering, or decide one of your release branches is the new main branch and abandon all the work done in the old main branch.
> 
> There's really no clearly best choice in this situation, it really depends on the particulars of the code in question. Door 1 is the correct choice given no more information about the code base under discussion, and this is the option Freebsd chose.


Wasn't this, or similar, the case with MS and Windows Vista / Server 2008?  As I recall they had to release, in order to "move on" or they would have been stuck with patching Windows XP / Server 2003 for many years to come.


----------



## olli@ (Oct 6, 2020)

If I remember correctly, FreeBSD 4 was the longest branch ever, with 12 releases (4.0 to 4.11) made from a single branch. The problem with that was that the “current” branch at that time (which would eventually become 5.0) diverged farther and farther away from 4.x, making it increasingly difficult to throw the switch while maintaining upgradability. Making things worse, 5-current had serious stability problems at that time. The new SMP code (mentioned above in this thread) was part of the problem, but also several other subsystems were rewritten or got major changes (e.g. WLAN, USB, CAM, PCCard), requiring significant amounts of testing and bug-shaking. It was a difficult situation. When 5.0 was finally released, many (most?) users preferred to avoid 5.x altogether and stick with 4.11 while waiting for 6.x.

Regarding Matt Dillon and DragonFly BSD, several things came together. Most importantly, Matt and other developers disagreed on how to proceed with the SMP code, kernel threading and locking. Another point was that Matt is a “hardcore” programmer and software engineer. Working full-time on FreeBSD, he produced such massive amounts of code, that it was impossible to keep up with reviewing all of that, which was (more or less) mandatory for getting it committed. Matt felt thwarted and getting slowed down, which was frustrating. He finally took what he assumed to be the last “usable” version of 4-stable and created the DragonFly BSD project. Since then I’m running DragonFly BSD inside a VM, just out of curiosity, and I think it works surprisingly well. It has some very interesting and useful features that are lacking from FreeBSD, for example variant symlinks. I also quite like the HAMMER2 file system – while you can’t really compare it with ZFS, it also has some very interesting features, e.g. the fine-grained snapshots and history for point-in-time recovery.


----------



## a6h (Oct 6, 2020)

olli@ Thanks. You're post inspired me to install DragonFly BSD in a VM. I've never done it before.


----------



## olli@ (Oct 6, 2020)

There are some cool things you can do with HAMMER2. For example, a developer explained in a blog how to create a time machine for backups: Set up a journal stream from an active file system, buffer it for – say – one hour, and feed it into a backup file system (locally or on the other side of the planet). The result is that you have a “delayed mirror”, so to speak, a “live backup” that represents your file system as of one hour in the past, like a time machine. When you accidentally `rm` a file, use your time machine to copy it back. Just don’t wait longer than one hour, because then the file disappears from your backup, too. (Or stop the journal stream if you need more time in case of an emergency.)


----------



## chrcol (Oct 6, 2020)

Combination of two things.

1 - It was released before ready, schedule coming ahead of quality.
2 - It made major architectural changes, moving from a single CPU mode to multi CPU model, if I remember at the time it was aiming to be one of the first OS to make this change, to be a leader in the market.

I didnt especially hate it though, although I did wait until 5.3.

For me the worst release was 9.0, I remember jumping early to it, and it had a nasty UFS bug that caused data corruption.  I posted on here about it, it was was fairly quickly patched and forgotten about, but filesystem corruption is about as bad as it gets.  Not jumped to a .0 release since.


----------



## Crivens (Oct 6, 2020)

Mat Dillon locked in a room with Poettering... and I think Mat is one hell of a software engineer. But stubborn as a brick.

Is the BSD available as, sadly preferable, a git repo back to 4.x? I would like to suggest to you all the book "Your code as a crime scene" and the tools in it. Maybe one can unearth some stuff still lurking around and waiting to bite us in the you-guess-what.


----------



## olli@ (Oct 6, 2020)

chrcol said:


> 1 - It was released before ready, schedule coming ahead of quality.


Rather the opposite, there was no real schedule (at least none that was adhered to), and it was delayed far too long.
But at least they learned their lessons: stricter schedules were introduced, and the decision was made to fork new stable branches more often.


> 2 - It made major architectural changes, moving from a single CPU mode to multi CPU model, if I remember at the time it was aiming to be one of the first OS to make this change, to be a leader in the market.


No, FreeBSD already supported SMP much earlier, as did the “competitors”. I ran FreeBSD 3.x with SMP on a dual Celeron-466 in 1999. However, the old kernel threading model did not scale very well, this is why they did major changes to it in 5.x.


----------



## `Orum (Oct 6, 2020)

The first release of FreeBSD I used was 4.7, just before 5.0 came out.  And boy do I feel lucky that I didn't start a few months later, or I might have dismissed the great OS that is FreeBSD as unstable and unusable.

I did try to move to 5.0 when it came out, and all of a sudden things that had worked flawlessly before were less stable than a house of cards.  Kernel panics became a regular occurrence.  When I looked into it, I learned all about the M:N threading model they had chosen for 5, which has now all but completely vanished.  A quote from Ingo Molnar still sticks in my head to this day: _"I'm no wimp when it comes to scheduler complexity, but a coupled kernel/user scheduling concept scares the *hit out of me." (source)_ In short, I went back to the 4.x release for as long as I could.

So yes, FreeBSD has made mistakes in the past.  But, they've acknowledged and corrected them, and now it's a rock solid system that still offers great performance.


----------



## cynwulf (Oct 6, 2020)

chrcol said:


> For me the worst release was 9.0, I remember jumping early to it, and it had a nasty UFS bug that caused data corruption.  I posted on here about it, it was was fairly quickly patched and forgotten about, but filesystem corruption is about as bad as it gets.  Not jumped to a .0 release since.


I had problems with 9.0.  not that specific problem, but as a result of the switch to GPT.  My old motherboard BIOS at the time couldn't get a along with it and it would freeze on the POST screen (had to dd the disks from a different box).  I had to install FreeBSD 8.x and upgrade to 9.0, just to get it installed.  It wasn't fixed until a 9.2 RC3 as I recall:





__





						9.2-RC2 Now Available
					





					lists.freebsd.org
				





> Fix a boot issue caused by some GPT partitioning tools.



Caused me no end of headache at the time.


----------



## Mjölnir (Oct 6, 2020)

On "Dillon vs. the rest" I'd like to note that the storm has waned since.  DragonflyBSD periodically imports driver code & other (e.g. network related) from FreeBSD, and IIRC there's a project proposal to port HAMMER2 to FreeBSD.


----------



## Jose (Oct 6, 2020)

chrcol said:


> Not jumped to a .0 release since.


Wisdom is the fruit of the tree of suffering. I don't do .0 in any system that has data I care about either.

Unfortunately some folks don't listen, and still jump on the latest shiny from Cupertino despite my warnings to the contrary. The wisdom from that tree is don't try to help people who don't listen.


----------



## a6h (Oct 6, 2020)

At least it's not bad as windows minor updates.
As it happens once in a while, people are waking up at morning and discovering that Microsoft has hosed them down.









						The Windows 10 October 2018 Update is deleting some users' content
					

Microsoft released the Windows 10 October 2018 Update earlier this week to non-Insiders, and users are finding that upon upgrading to version 1809, some of their files have gone missing.




					www.neowin.net


----------



## Crivens (Oct 6, 2020)

In the old days of the Amiga, Mat did a news spooler that was very fast. Only drawback was it used crc16 to see if an item was present and then not fetch it. Hash collisions were something that could wreck your threads. I guess he is not as hot headed today as back then, but that was some fireworks.

Having HAMMER2 would be nice, real nice - as would be process checkpointing. Something we might want to port over if possible.


----------



## olli@ (Oct 6, 2020)

mjollnir said:


> On "Dillon vs. the rest" I'd like to note that the storm has waned since.  DragonflyBSD periodically imports driver code & other (e.g. network related) from FreeBSD, and IIRC there's a project proposal to port HAMMER2 to FreeBSD.


That’s true. Matt continued to be present in some of the mailing lists, even right after he started DragonFly BSD, and sometimes gave hints and tips, e.g. when he fixed a bug in DF that would also affect FreeBSD. I’m not sure if he’s still on those mailing lists today, as I look at them only very rarely myself.


----------



## chrcol (Oct 8, 2020)

olli@ said:


> Rather the opposite, there was no real schedule (at least none that was adhered to), and it was delayed far too long.
> But at least they learned their lessons: stricter schedules were introduced, and the decision was made to fork new stable branches more often.
> 
> No, FreeBSD already supported SMP much earlier, as did the “competitors”. I ran FreeBSD 3.x with SMP on a dual Celeron-466 in 1999. However, the old kernel threading model did not scale very well, this is why they did major changes to it in 5.x.



Yes the developers felt it was delayed too long so just released it, hence the problems.  For me i have the opposite attitude, if it isnt ready dont release even if it takes 10 years. Or back out the incomplete features.

Also thanks for clarifying on the multi cpu stuff, it is what I meant in terms of making the underling threading model optimised for multi core, but I was wrong in saying it introduced multi cpu capability.


----------



## neel (Oct 10, 2020)

As another example, if you look at Windows Vista, Microsoft released an slow, buggy OS incompatible with existing software. Many stayed with Windows XP.

Microsoft thought they would have a major upgrade but instead they cancelled the original Longhorn, re-done it and still got something beta-grade. The issues have largely ironed out by SP1 but by then people simply didn't want Vista, period.

Not that Vista's legacy is all dead, modern versions of Windows (7-10) are largely based on Vista (Windows 10 can run on most Vista-era hardware) bringing forward many features and design decisions, but significantly refined and the bugs ironed out. From what I read online, FreeBSD 6.x and later have largely ironed out the 5.x-era issues.

I'll admit, I was a Vista user and didn't have many issues, but never was a FreeBSD user prior to 9.x.

In short: Windows Vista is to XP what FreeBSD 5.x is to 4.x: a botched successor to a great OS.

My worst FreeBSD release was 10.1, mainly because whenever I ran `freebsd-update` on UFS with Soft Updates, the system wouldn't reboot, period. It would somewhat hang but still respond to pings. This issue once meant I couldn't access my home email server until my brother power cycled the PC. But as I mentioned earlier, I never ran 5.x.

I do get some issues from time to time with CURRENT on my desktop, but that doesn't count since it's bleeding-edge code, bugs are expected.

Disclaimer: I work at Microsoft, not on Windows (or any Linux efforts) however.


----------



## olli@ (Oct 10, 2020)

Crivens said:


> Is the BSD available as, sadly preferable, a git repo back to 4.x?


Not sure if that was actually a question, but the answer is yes, of course: https://github.com/freebsd/freebsd/tree/stable/4
It’s a mirror of the SVN repository, so it even goes back to 2.0. (1.x sources are not provided publicly because of USL license issues with the old 386BSD codebase.)


----------



## Crivens (Oct 10, 2020)

Thank you
 Now I only need to pull that complete repo, with history, and see if the tools from that book do anything. Now where did I put that book?


----------



## ronaldlees (Oct 12, 2020)

vermaden said:


> I started with 5.x (5.4 to be precise) when I moved from Linux so I never had a chance to rally know 4.x series. Maybe that is why I do not see anything 'tragic' in 5.x series.



I started with 4.x series, but used 5.3 for a long time (by 5.3 they had fixed some things).  I'd guess it depended what you were using it for, relative to having problems or not having them.  There was some turmoil involved enough so that I delayed the move to 6, thinking it would be rinse/repeat.   I used 5.3 mostly for computer programming, and that's likely not considered a very intensive use of the system ...

I used 6.x briefly, and then jumped to 7 (and thusly right into the infamous race condition bug).


----------

