# Traveling salesman and other discussions.



## astyle (Aug 8, 2021)

Braveheart_BSD said:


> My background is Transportation and Logistics, nothing to do with computer work.


You'll be surprised  at how much computing power it takes to solve the Traveling Salesman problem... in places like Atlanta, GA or Shanghai or Rio de Janeiro...


----------



## fr33bsd (Aug 17, 2021)

astyle said:


> You'll be surprised  at how much computing power it takes to solve the Traveling Salesman problem... in places like Atlanta, GA or Shanghai or Rio de Janeiro...


As a computer scientist I can ensure you to be able to solve a middle class TSP-Problem faster with an Amiga 500 by using optimized version of Kuhn-Munkres-Algorithm than by using other methods on a HPC-Cluster.

Say, 'middle class TSP-Problem' consists of a matrix of size of little less than 138 Megabytes to fit into max of Amiga RAM without using swap. 138MB of node edges is a lot ... believe me! That would be a nice 'proof of concept' research project, wouldn't it be? 

Once upon a time, there was Windows NT 4 - best Windows ever: It was stable and contained no home calling spyware. Ok, there was a lack of security. Hey, but it is Windows. But it was better than DOS 6.22, right?  Right?! 

Then  Steve "Monkeyboy" Balmer and Bill "Butthead" Gates had spamed that much crap propaganda about how good their new Windows- is: 1, 2, 3 ... (n-1)-times. Micro$oft products' quality is equivalent to the joke about standards:
"How many Micro$oft enployees do you need to change a broken bulb? - None, because darkness is declared to be new standard by Micro$oft."

Personally I think Micro$oft is a lie and greed driven company with one focus: to force dependency to earn more profit, but delivering nothing more than . Stealing ideas, selling as own. And so on .... and so on ...

Today, Micro$oft makes business with software others wrote ... hmm ... how was that ... with DOS again? Today ... Azure ... Linux ... Github...?

Once, Micro$oft advertised Linux as cancer. Today, Linux really turned into sort of cancer. At least, since amazon started to fall in love with Ubuntu. Yes, Mr. Richard Stallman, bye Linux. For me: SuSE is dead, Debian is dead (as it's initiator), Ubuntu is dead. But Micro$oft, tell me: "Today, is it love?"

Apple seems to have victim symptoms - driven by greed, of course. In the begining, Apple *was* (really) different regarding product quality. It was "sort of better" than Micro$oft.

But Apple's 'OSes'  - even it's base is BSD - turned into optically orgasmic over-engineered security-Swiss-chees- by time.

And there we are: We started with Unix (mainly thanks to those guys: Ken Thompson, Dennis Ritchie), we end-up with Unix.

Sorry, Linus. Linux was good, but since every script kiddie, who installed Linux and generated code for it to call himself hacker, turned Linux into SPAM - well, in my eyes, Linux is cancer-spam with Micro$oft-DNA.

I don't support greed and lies, I love stability and reliability. Here I am, FreeBSD!


----------



## fr33bsd (Aug 17, 2021)

Sorry, for offtopic, but ....


lloyd said:


> [..]
> 
> No one runs production workloads on FreeBSD anymore.
> [..]


There are companies, who would run vanilla FreeBSD, but there seem to be too less companies offering payed support for FreeBSD.   
Most companies demand support by vendor, another (limited) company. No support, no deal. Establish "FreeBSD Inc.", and offer support. But you will need the OK of the foundation, I guess 

In general, I think there are companies, who run FreeBSD in production.
On the other hand, i.e. Isilon is FreeBSD based and is offered by Dell who sells it to companies running production workloads on it.


----------



## SirDice (Aug 17, 2021)

fr33bsd said:


> In general, I think there are companies, who run FreeBSD in production.


lloyd was referring to the FUD surrounding FreeBSD that can be found on the internet. Of course there are companies that use FreeBSD, even a couple of really big ones (Netflix is a prime example). It's just not that visible to the "untrained" eye.


----------



## fr33bsd (Aug 17, 2021)

Yes, you can read a lot   on the internet... Yeah, I know about netflix. I read it last year. As far as I know lots of ISPs also use FreeBSD. FreeBSD rules some embedded hardware, some devices for science labs, and so on ... But as I said: Some companies do not go FreeBSD, because they are scared without professional support. I know one of that companies ...  No more comments...


----------



## astyle (Aug 17, 2021)

fr33bsd said:


> Yes, you can read a lot   on the internet... Yeah, I know about netflix. I read it last year. As far as I know lots of ISPs also use FreeBSD. FreeBSD rules some embedded hardware, some devices for science labs, and so on ... But as I said: Some companies do not go FreeBSD, because they are scared without professional support. I know one of that companies ...  No more comments...


It may come as uncomfortable news, but one area where Microsoft and Apple has Linux and BSD's completely beat is frankly upgradeability of software. Right now, I'm in the process of figuring out how to use Poudriere (ports-mgmt/poudriere) to be able to ugrade just KDE, as in download just the up-to-date tarballs, compile them with settings I like, and not break anything in the process. It takes a LOT of reading, trial-and-error, and I'm nowhere close to replicating the upgradeability and convenience provided by Windows (And I never bothered to learn how it's done on Apple). So far, I discovered that on FreeBSD, it takes having your own build environment, a LOT of fiddling around with config files, and even recovering from mistakes takes having a handle on ZFS and boot environments. For me, it's a fun process (that I do on my machines at home), but at some point you do need a way to apply a security patch without having to know how to recompile the kernel (which can be a very long process, as we all know). Sure, any sysadmin worth his/her salt should have a good handle on all that, and be able to apply industry's best practices, but they shouldn't be tying up the users' machines for hours on end just to do stuff in line with their personal philosophies. Users have their own stuff to do, too.


----------



## kpedersen (Aug 17, 2021)

astyle said:


> one area where Microsoft and Apple has Linux and BSD's completely beat is frankly upgradeability of software [...] Right now, I'm in the process of figuring out how to use Poudriere (ports-mgmt/poudriere) to be able to ugrade just KDE


Possibly but if you do a like for like comparison I can guarantee it will be even harder to upgrade Microsoft's UI shell between Windows releases. Or if you wanted to downgrade / upgrade Win32 API, MFC and other native inbuilt Windows frameworks you won't be able to (mostly because they are closed-source). The fact that on FreeBSD we can upgrade such a large component is impressive in its own right.



astyle said:


> but they shouldn't be tying up the users' machines for hours on end just to do stuff in line with their personal philosophies. Users have their own stuff to do, too.


You could pay someone else to do this. For example on Windows, you would be paying Microsoft or other proprietary vendor. Unfortunately you will need to pay more on FreeBSD because due to popularity of Windows, many people will be paying so due to "bulk buying" the vendor can do it cheaper than a single company can.

So in short; I absolutely do agree with you that it is more difficult in general to maintain a non-windows install (certainly not click and play). However once put into context, not only is it really not FreeBSD's fault but it is also a pretty decent compromise.


----------



## Argentum (Aug 17, 2021)

fr33bsd said:


> Sorry, Linus. Linux was good, but since every script kiddie, who installed Linux and generated code for it to call himself hacker, turned Linux into SPAM - well, in my eyes, Linux is cancer-spam with Micro$oft-DNA.


My personal experience was that Linux was *never* good. As I have written here before, I started using FreeBSD back in 1995. Just one week before starting using FreeBSD, I had installed Linux on this machine (server) to find out it was a *complete crap*. There is a saying here that you cannot make a bread out of s***.

Then I turned to FreeBSD to find it is a rich and complete OS. Since that, never found any reason to move away from FreeBSD.


----------



## SirDice (Aug 17, 2021)

Argentum said:


> There is a saying here that you cannot make a bread out of s***.


A good English proverb would be "you can't polish a turd". Although I do remember a Mythbusters episode where Adam Savage did manage to polish an actual turd.


----------



## astyle (Aug 17, 2021)

kpedersen said:


> Possibly but if you do a like for like comparison I can guarantee it will be even harder to upgrade Microsoft's UI shell between Windows releases. Or if you wanted to downgrade / upgrade Win32 API, MFC and other native inbuilt Windows frameworks you won't be able to (mostly because they are closed-source). The fact that on FreeBSD we can upgrade such a large component is impressive in its own right.


Windows Vista was so bad that there was truckloads of  freeware on SourceForge that offered UI enhancements for it, and yes, that stuff was upgradeable. Every component offered a way to check for updates. FWIW, Java and Firefox do that. That's impossible to to do properly on Linux/BSD.  As for downgrading Windows frameworks, all you need is an older version of .NET to be available. It's possible on Windows. 


kpedersen said:


> You could pay someone else to do this. For example on Windows, you would be paying Microsoft or other proprietary vendor. Unfortunately you will need to pay more on FreeBSD because due to popularity of Windows, many people will be paying so due to "bulk buying" the vendor can do it cheaper than a single company can.


I think that's kind of missing the point.  If you want to do stuff 'Just Right' on your personal machines, that's your call. But it's a different matter at work. At work, a user should have access to working equipment in short order. Doing professional quality work on that is a priority. If FreeBSD is the standard stuff to use, do FreeBSD, and have a handle on it. If Windows is the standard stuff to use, do Windows, and have a handle on it. Automotive industry equivalent would be this: FreeBSD is the motorcycle that you tinker with at home, and ride to work. At work, you apply the same skills to make sure the car fleet works reliably, and is quick to fix. Nobody at work wants to switch to motorcycles that you customize - they'd rather use cars, and get the job done. When cars are not enough to do the job, that's when your motorcycle modding skills (FreeBSD admin skills) are called on.


----------



## fr33bsd (Aug 17, 2021)

Argentum said:


> My personal experience was that Linux was *never* good. As I have written here before, I started using FreeBSD back in 1995. Just one week before starting using FreeBSD, I had installed Linux on this machine (server) to find out it was a *complete crap*. There is a saying here that you cannot make a bread out of s***.
> 
> Then I turned to FreeBSD to find it is a rich and complete OS. Since that, never found any reason to move away from FreeBSD.


As desktop OS Linux was fine between 1998 and approx. 2016. yast1 was ok, yast2 was big pain. apt is okay, gentoo (with merge?) was pain: download source an compile everything ... good night. yum is same as apt.
Linux as server OS ... well, maybe webserver ... CIFS ... I mean Linux+Apache/nginx is better than M$ IIS   Samba is semi broken on FreeBSD, but working well on Linux.

Linux has been changing FS very often. Default was ext{1?,2,3,4}, indeed. But redhat went btrfs for a short time. I tried btrfs, but only w/r ops, no volume or raid options, because they were instable. btrfs with softwareraid was ok.
Linux+softwareraid+data integrity is pain in the b###. But, linux software raid on SMR disks is okay (even slow). It is not that much pain as people say about zfs+smr. Maybe zfs+smr is good in mirror only 

Once upon a time, someone told me about zfs on solaris. It was the time as sun existed. Sun did a great job on solaris, zfs and nfs.

Today I would use Linux only as desktop OS, for Kubernetes (i.e. openshift), hadoop.
Databases, Fileserver/NAS, Router -> FreeBSD (data integrity, stability, security, optimized stacks)

What I miss on FreeBSD is sort of distributed zfs, like maprfs. I know you can use is gluster somehow on FreeBSD. But are there other distributed/cluster FS for FreeBSD?

I want to try nomadBSD for desktop-pc and dragonfly bsd with hammer1+2-fs for server .


----------



## kpedersen (Aug 17, 2021)

astyle said:


> Windows Vista was so bad that there was truckloads of freeware on SourceForge that offered UI enhancements for it, and yes, that stuff was upgradeable


Arguably FreeBSD has way more UI enhancements (Window Managers). They are also much easier to install. If I recall, the guys behind these Windows themes (Stardock?) had to reverse engineer and hack at (patch) the theme engine to allow third party customizations.


astyle said:


> As for downgrading Windows frameworks, all you need is an older version of .NET to be available. It's possible on Windows.


I did explicitly mention "native" frameworks (Things that the OS and Microsoft actually use internally) to discount Microsoft's Java clone. For example on FreeBSD it is very easy to install different versions of Java / Python / Mono, etc. Likewise the .NET framework isn't actually a core OS framework (It isn't even really a framework any more than Python is).

MFC, Winsock, Shell32, User32, Advapi32 are examples of things that you certainly won't be able to downgrade/upgrade at will. Where Microsoft does win is that these APIs rarely change so it gives great stability. Obviously Linux is crippled with constant mutations like systemd and Wayland. Whether it is better or not is really not important. What is important is that the API it is not stable unfortunately.


astyle said:


> If FreeBSD is the standard stuff to use, do FreeBSD, and have a handle on it. If Windows is the standard stuff to use, do Windows


Well kinda. If Microsoft owned FreeBSD, they would make sure that it would always be the "standard" stuff to use. Even if it isn't. So you can't trust that. There is no such thing as a standard thing. Usually that just translates to "the thing with highest marketing budget".


----------



## Beastie7 (Aug 17, 2021)

FreeBSD could use something like Win32 or Cocoa. it’s actually not that bad in terms of consistency at least.


----------



## kpedersen (Aug 17, 2021)

Beastie7 said:


> FreeBSD could use something like Win32 or Cocoa. it’s actually not that bad in terms of consistency at least.


I absolutely get satisfaction that knowing under all the layers of shite that Microsoft piles on-top of its OS, the good old classic Win32 API that I learned as a kid (Beep() and Sleep()!) on Windows 3.1 (Win32s) is powering it all almost unchanged.

I do agree. I really am quite jealous of the consistency of Win32 in terms of the graphics / windowing layer. I think Wayland is only going to make things worse for us in the future. We won't adopt it; software will become a mess for a couple of years and then we will all be back to X11-like emulation layers.

Whilst I have only dabbled with Cocoa, if it is indeed consistent then they will be on to a winner. Especially if Swift can run its course and Objective-C takes the reins again. From what I have heard. Apple is well maintaining their Obj-C support but keeping it very low key.

*Edit: Oops, just realized this is in the intro thread! Please ignore my rambling noise!*


----------



## astyle (Aug 17, 2021)

kpedersen said:


> For example on FreeBSD it is very easy to install different versions of Java / Python / Mono, etc. Likewise the .NET framework isn't actually a core OS framework (It isn't even really a framework any more than Python is).


Unlike under FreeBSD,  under Windows, t's very easy to ask Java / Firefox / etc. to monitor for updates, and have them properly installed without breaking anything.  Try recompiling www/firefox, it will want to update truckloads of other stuff. I tried to ask Firefox to monitor for updates under PC-BSD - and that failed. Using PC-PSD's PBI manager was OK, but the internal updater did not have the proper permissions. And maybe it shouldn't have them, after all - that's a famous security hole on Windows.


kpedersen said:


> Well kinda. If Microsoft owned FreeBSD, they would make sure that it would always be the "standard" stuff to use. Even if it isn't. So you can't trust that. There is no such thing as a standard thing. Usually that just translates to "the thing with highest marketing budget".


I'd like to clarify: By "standard stuff" I mean "whatever the workplace uses".  An auto industry equivalent (I know I'm overusing that crutch, but still) would be that Hertz rents Fords, Avis rents Chevys. Nobody cares if they both use Goodyear tires.

I love a good debate, though, so I'll add this: I personally see Wayland making better progress towards maturity and official adoption by FreeBSD.


----------



## ralphbsz (Aug 17, 2021)

astyle said:


> It may come as uncomfortable news, but one area where Microsoft and Apple has Linux and BSD's completely beat is frankly upgradeability of software.


Honestly, I don't have the problem. For me, FreeBSD has the easiest upgrade mechanism of all OSes. That's because I don't try to compile things that don't need compiling. All I do is freebsd-update {fetch,install} and pkg {update,upgrade} every few days. For a while I had a small number of things (three or four?) that were installed from ports, and then I had to go to /usr/src and issue some make command that I've already forgotten. But these days, I find that pre-compiled packages do everything I need.

One crucial difference: I only use FreeBSD for servers. Take any tool, like an OS, out of its comfort zone, and use it in an environment that it is not really intended for, and things get hard.


----------



## astyle (Aug 17, 2021)

ralphbsz said:


> Honestly, I don't have the problem. For me, FreeBSD has the easiest upgrade mechanism of all OSes. That's because I don't try to compile things that don't need compiling. All I do is freebsd-update {fetch,install} and pkg {update,upgrade} every few days. For a while I had a small number of things (three or four?) that were installed from ports, and then I had to go to /usr/src and issue some make command that I've already forgotten. But these days, I find that pre-compiled packages do everything I need.
> 
> One crucial difference: I only use FreeBSD for servers. Take any tool, like an OS, out of its comfort zone, and use it in an environment that it is not really intended for, and things get hard.


yeah, for me, I prefer ports, because pre-compiled packages are generated using defaults that don't quite work for me. I think FreeBSD does have the potential to be a usable desktop if I put in the effort it takes. I don't see that as 'being out of comfort zone' for FreeBSD.


----------



## ralphbsz (Aug 17, 2021)

fr33bsd said:


> Linux has been changing FS very often. Default was ext{1?,2,3,4}, indeed. But redhat went btrfs for a short time.


For a single-node single-disk file system without RAID and without checksums, ext4 is very good. I personally don't run SMR disks myself, but AFAIK ext4 is de-facto the only commonly available choice for SMR.

On the other hand, I think every storage system that's more than temporary files absolutely needs RAID and checksums. At that point, ZFS is the only commonly available choice. And that is the single strongest argument for FreeBSD: Of all commonly available OSes, it is the one where ZFS seems best integrated and supported.



> I tried btrfs, but only w/r ops, no volume or raid options, because they were instable. btrfs with softwareraid was ok.


Btrfs started out as a smart idea. Like many smart ideas, it died on the hill of correct and supported implementation. While Ohad's idea of a modifiable B-tree with snapshots was great, the actual implementation was de-facto a machine to create data loss. And then, due to some corporate interest and people's ego conflicts, Btrfs got shipped to the masses as the default file system WAY too early, which ruined its reputation. So today, I think 99% of all Linux installations are back to running ext4, usually on top of various interesting storage layers.



> Today I would use Linux only as desktop OS, ...


But note that Linux has a gigantic market share among servers, in particular in the cloud.



> What I miss on FreeBSD is sort of distributed zfs, like maprfs. I know you can use is gluster somehow on FreeBSD. But are there other distributed/cluster FS for FreeBSD?


Cluster file systems are typically used on supercomputers. If you look at the Top500 list of supercomputers, you will find that Linux has 100% market share there: No single supercomputer among the 500 largest in the world runs anything other than Linux. Now, some of the largest compute clusters in the world are not on that list, because they are hidden inside companies from the FAANG list. But for those, nearly all also run Linux, with two smallish exceptions: A small fraction of Microsoft's cloud machines run Windows, and a small fraction of Netflix' machines run FreeBSD (but those are not clustered AFAIK).

The reason it's really hard to find cluster file systems on FreeBSD is simple: there is no demand for them.


----------



## ralphbsz (Aug 17, 2021)

astyle said:


> I think FreeBSD does *have the potential to be** a usable* desktop *if I put in the effort it takes*.


(I added the bold and underline). You may be correct on that. But FreeBSD *is** a great* server OS, and it requires *very little* effort. (I put the bold underline in the corresponding places.)


----------



## Beastie7 (Aug 17, 2021)

kpedersen said:


> Whilst I have only dabbled with Cocoa, if it is indeed consistent then they will be on to a winner. Especially if Swift can run its course and Objective-C takes the reins again. From what I have heard. Apple is well maintaining their Obj-C support but keeping it very low key.



i think Swift was a pre mature move IMO. Cocoa is a huge objective-c API, and expecting people to rewrite their stuff just to maintain relevance with google hipsters doesn’t bode well.


----------



## fr33bsd (Aug 17, 2021)

ralphbsz said:


> Cluster file systems are typically used on supercomputers. If you look at the Top500 list of supercomputers, you will find that Linux has 100% market share there: No single supercomputer among the 500 largest in the world runs anything other than Linux. Now, some of the largest compute clusters in the world are not on that list, because they are hidden inside companies from the FAANG list. But for those, nearly all also run Linux, with two smallish exceptions: A small fraction of Microsoft's cloud machines run Windows, and a small fraction of Netflix' machines run FreeBSD (but those are not clustered AFAIK).
> 
> The reason it's really hard to find cluster file systems on FreeBSD is simple: there is no demand for them.


What about Isilon?


----------



## astyle (Aug 17, 2021)

Can't believe I'm participating in hijacking a thread like this, but still: What about ZFS? with its ability to 'add disks to a pool', ZFS is nicely suited for clusters (or, and the total capacity of (2^128 bytes being just the max size of *one* zpool, and a given system supporting up to 2^64 zpools) far exceeds what Isilon's OneFS has to offer! and it's usable on plain, vanilla FreeBSD of the highest quality.


----------



## ralphbsz (Aug 17, 2021)

Isilon is not accessible as a cluster file system: Instead, a group of Isilon servers create a file server, and clients connect to the Isilon servers using the protocol du jour (NFS, CIFS, ...). So as seen from the file system client, Isilon is just like buying a NetApp, except that the Isilon "machine" uses many nodes internally. But the Isilon server uses a cluster internally, and interconnects their servers and disks with an internal cluster network. And the way the Isilon system is architected, the workload within their server cluster is spread very nicely.

Just using ZFS is even less a cluster file system: In ZFS, every disk can only be used by one server at a time. You can build active/passive systems if you have multi-ported disks (for example external SAS enclosures that can connect to multiple servers, or using iSCSI disks as the back-end), but there is no cluster at run time. And there is no internal load balancing: If you have for example three ZFS servers, one serving /a, the second one /b and the third one /c (perhaps with failover, the disks that form the pool that handles /a is physically connected to servers 1 and 2, but currently being served from server one), and then there is an unbalanced workload (for example all IO requests right now are on the /a file system), then the 2nd and 3rd server will be idle, while the 1st is overloaded.

In an actual cluster file system, the file system runs on the clients, and caching and file system decisions is done on the client. The clients then get their data and metadata using some technique; sometimes directly from network-attached disk devices (works good with iSCSI), sometimes from a second SAN (this used to be done in the old days of FiberChannel), sometimes from data-/metadata- servers using other protocols over the network.


----------



## fr33bsd (Aug 18, 2021)

I see, ralphbsz.

What about this hybrid mixture? I am missing FreeBSD there.


----------



## SirDice (Aug 18, 2021)

Split a bunch of posts off from the introduction thread.


----------



## Zvoni (Aug 18, 2021)

Chipping in to the Cluster-Filesystem-Topic (in this case GlusterFS):
In my skydiving-club i've a setup exactly like that:
Two old commodity-PC (DELL's with an i5 and 8GB RAM each, 3 HD's each --> ada0=OS, ada1 and ada2 as Bricks) running FreeBSD as OS and GlusterFS.
The Gluster-Volume is setup as a 2x2 Distributed Replica (Brick1 on S1 replicated with Brick1 on S2 a.s.o.).
Meaning: If a Server goes down, all files are still accessible on the other Server (and i actually really got into that situation! Worked like promoted!)

On a Performance-Note: I didn't notice any grave differences in Performance/Speed for read and write (and i would have noticed that, since the Gluster-Volume is mainly used for Video-Files which on average are about 1.5-2.0GB in Size). At a guess i would say maybe up to 5% for a loss in Perf./Speed.

Lessons learned: 
1) Use UFS for the Bricks, since Gluster doesn't play well with ZFS on FreeBSD (even mentioned on the official Gluster-Documenation) --> A sound backup-strategy is still needed (why would we have to discuss something essential like a backup-strategy?)
2) Setting up Samba to export the Gluster-Volume was a PITA, since i initally used the binary packages (pkg install samba), and missed, that the support for GlusterFS is per Default OFF. So i had to use ports for Samba.


----------



## ralphbsz (Aug 19, 2021)

Jose asked (in a different place): "What is your opinion of HAST"

Answer:

I've never used it, but know how it works. For a small system with a fast network and moderate performance needs, it might work.

It has a few massive costs. First, it doubles the amount of disk space you need, since it forces RAID-1 on everything. In theory, you can get some of that space usage back, if you are using RAID too (like ZFS). But that assumes that HAST has as good read error handling as ZFS does, including being tied in with the CRC system ... which it doesn't (if ZFS detects a CRC error on copy #1, it needs to read copy #2; HAST doesn't do that). So to protect against silent data corruption you still need some form of RAID on top of it, so you end up with something like RAID51, and at that point your space overhead is annoyingly high.

Second, it has a nasty performance cost, because you need to wait for every write to be done twice. But because of its extent-based metadata, that's not on a per-sector basis, but per extent. And if you are doing truly random IO above HAST, that will lead to massive false write sharing (a 4K sector write turns into a 2M extent write on recovery). In default mode, it is also not completely safe, since it doesn't harden the extent bitmaps to disk immediately. I suspect it might even have "unstable reads" if you switch over between primary and backup server frequently enough (because the extent bitmaps might get lost); if not careful, that could wreak havoc with the file system on top.

For a small system, where the factor of 2 space overhead is not that important, and where write speeds are low, it might work great. But there another problem comes in: HAST needs to be integrated with a control mechanism above that switches primary and backup nodes. For a small system user (typically a hobbyist), getting that correct in the presence of all possible failure situations is SUPER hard. Distributed consistency is a nasty problem, in particular in two-node clusters (where you can't use quorum algorithms to force consistency, since with just two voters there is never any clear majority quorum). So for an actually secure system, one would probably have to implement something like Disk Paxos on top of HAST ... and implementing Paxos or Chandra-Toueg is exceedingly hard, and finding pre-cooked packages that implement it is nasty too.

Deep underneath this criticism is two observations. One is still true, the other only partially. HAST is fundamentally a RAID layer (it gives redundancy). The first one: To work best, RAID needs to be tightly integrated with the file system. RAID needs to know which blocks are live (so it doesn't waste effort on tracking / resilvering unallocated space), the RAID implementation needs to be tuned to the IO pattern of the file system (log structured vs. extent based vs. traditional allocating, and matching block sizes), and checksums need to be end-to-end. Second: RAID can't be done over the network, it's too slow. That statement has become partially wrong today, because (a) fast networks (Infiniband and 10G+ Ethernet) have latency comparable to disk latency, so going over the network "only" doubles the latency, and (b) some of the performance hits of RAIDing over the network can be fixed by tightly integrating the RAID layer with both the network and the file system layer, so the cost of RAID updates is amortized, and RAID data structures match the file system data structures. For example, you can redesign your file system so metadata is stored mirrored (even more than 2-way mirrored, I've seen 5-way replication be a good thing), while file data is stored very efficiently parity encoded. Like that the cost of degraded operation and resilvering hits areas that are less latency sensitive, and the throughput-vs-latency tradeoff can be made.

But: HAST is not integrated in the file system, and it runs over the network without any remediation for the shortcomings. So it's not going to be great.


----------



## Jose (Aug 19, 2021)

ralphbsz said:


> The reason it's really hard to find cluster file systems on FreeBSD is simple: there is no demand for them.


What about Ceph?






						Intro to Ceph — Ceph Documentation
					






					docs.ceph.com
				








						Manual Deployment on FreeBSD — Ceph Documentation
					






					docs.ceph.com


----------



## ralphbsz (Aug 19, 2021)

Jose said:


> What about Ceph?


I'm so sorry! I completely forgot that Ceph exists for FreeBSD. It is an excellent system for large cluster storage. I particularly like that you can use it to get both file storage and object storage on the same system; that's really important if you have existing file-based workflow that still need to be supported, while also trying to migrate to more efficient object storage. Don't know how easy it is to set up Ceph on a small system, and how well it works with the minimal solution (a 2-node cluster).

And just to be clear: HAST and other such solutions are a block-based RAID layer (just a networked one). Ceph and friends are file systems (just networked ones).


----------



## fr33bsd (Aug 20, 2021)

What about glusterfs?


----------



## ralphbsz (Aug 20, 2021)

All I know about glusterfs is that it exists. I don't know any of its internal details.

And just to be clear: When I said above "really hard to find cluster file systems on FreeBSD" what I mean is the following: Cluster file systems are hard to write, typically done by large commercial companies (Oracle, IBM, EMC, Ceph -> RedHat, in earlier days Sun and Digital), and are typically used in large commercial or academic institutions. They are not typically found in hobbyists or small commercial settings.


----------



## grahamperrin@ (Aug 22, 2021)

fr33bsd said:


> … other distributed/cluster FS for FreeBSD? …



I thought of Andrew File System, only because I toyed with it many years ago on a Mac. 

<https://www.freshports.org/net/openafs/> deleted a couple of years ago | <https://wiki.freebsd.org/AndrewFileSystem> | <https://www.bsdcan.org/2006/papers/afs_bsd_slides.pdf>


----------



## fr33bsd (Aug 22, 2021)

ralphbsz said:


> I'm so sorry! I completely forgot that Ceph exists for FreeBSD. It is an excellent system for large cluster storage. I particularly like that you can use it to get both file storage and object storage on the same system; that's really important if you have existing file-based workflow that still need to be supported, while also trying to migrate to more efficient object storage. Don't know how easy it is to set up Ceph on a small system, and how well it works with the minimal solution (a 2-node cluster).
> 
> And just to be clear: HAST and other such solutions are a block-based RAID layer (just a networked one). Ceph and friends are file systems (just networked ones).


I know ceph. In theory it is cool concept. In real world it sucks due to high latency and terrible read/write performance. I did a ceph PoC on a cluster, but even with tuning and special advises of experts and some custom deployment scripts ceph acted as fast as a slimy Octopodidae in fresh concrete. Ceph is   (crap). Even bigger deployments of ceph suck! In a company I worked for we had ceph. The company wasted $500,000 on the hardware and then another 200,000 on "best" NVMe SSDs on the market at that time to improve performance by upgrading ssd-hardware, but ceph kept being slow as ... 
Due to some missing but important information I even once suffered data loss, because they forgot to mention something in their documentation about some parameter that makes ceph stop working without any message (not even error in logs).
I actually pissed at sage's leg because of that, but he did not care  He sold this company to redhat for 20 mio. Not even a lawsuit could make him fear, because redhat, now IBM, is responsible for ceph aka crap (how I call it).


----------



## free-and-bsd (Aug 23, 2021)

Yea... But you know, this upgradability is not always such a good thing!
Windows 10, for example, is SO UPGRADABLE that my friend at work can't find a way to BLOCK those imposed updates. The machine runs some very-very $$important software. Win 10 is pushing the updatable updates... right!
Only the update install ends up with black screen on the next reboot and no way out but triple Reset. 'Known problem', yesss, 100 fixes published but none works for that particular machine...
So much for easy upgradability. So... you better stay where you are and say thank you


----------



## fr33bsd (Aug 23, 2021)

free-and-bsd said:


> Yea... But you know, this upgradability is not always such a good thing!
> Windows 10, for example, is SO UPGRADABLE that my friend at work can't find a way to BLOCK those imposed updates. The machine runs some very-very $$important software. Win 10 is pushing the updatable updates... right!
> Only the update install ends up with black screen on the next reboot and no way out but triple Reset. 'Known problem', yesss, 100 fixes published but none works for that particular machine...
> So much for easy upgradability. So... you better stay where you are and say thank you


I experienced this upgradability fetishism on hardware, too. sometimes it brings you only few IOPs more, but costs the price as all installed server mainboards together. Then rather wait and buy complete new hardware (i.e. upgrade pcie3->pcie4). But sometimes even bombarding the problem with hardware won't fit it, just because the software you use is crap (examples: windows*, ceph, linux (in some cases like systemd)). then rather switch to freebsd. back to the roots   or get other software (regarding apps) open src or write it by yourself/your developers.

But there is a problem: if your boss is a stupid greedy one, then change the job! Over 14 years ago, I had a meeting with the company's boss. He seriously asked me the purpose of buying few expensive IBM x86 servers (dual socket, ECC RAM, SAS ...) , because he was sure he can conquer all the workload by using ten cheap noname consumer lowend PCs (no ecc ram) from walmart. Why buy 900G 15k SAS12 HDD with proper broadcom 9400 HBA for $500 each if you can buy 18T (where you can store more data) 7.2k SATA HDD for $450 that you can connect to the mainboard of your PC? Why use raid if you can go tape backup if your drive fails - so, why buy things twice, right? 

This guy was so smart, he would accept running windows 95 (32-bit) on epyc7xx3 CPUs with 64 cores, 1T RAM, 12 NVMes, 12 SAS 15k drives, 10MBit NIC with BNC connector and a 14.4 dail-in modem by motorola to act as new company high end windows file share server.

What have I done? I left that company.


----------



## Deleted member 30996 (Aug 28, 2021)

astyle said:


> I think FreeBSD does have the potential to be a usable desktop if I put in the effort it takes. I don't see that as 'being out of comfort zone' for FreeBSD.


 But is it out of _your_ comfort zone?

This is when I like to show a shot of my X61 at 306 days uptime:





The W520 that took it's place is at 127 days and 306 looks likely... Sans electricity outage.


----------

