# Does each rack mounted server have an OS?



## na5m (Jun 27, 2017)

Does every server in a rack system have an OS installed locally? Lije the rack systems you see in businesses like data centers and such. 






I guess I'm trying to ask if you can have a computer (a box) without on OS on it that is being controlled by a remote computer. 

Sorry if this is a dumb question. 
Thanks!


----------



## SirDice (Jun 27, 2017)

na5m said:


> Does every server in a rack system have an OS installed locally? Lije the rack systems you see in businesses like data centers and such.


Usually, yes.



na5m said:


> I guess I'm trying to ask if you can have a computer (a box) without on OS on it that is being controlled by a remote computer.


If it doesn't have an OS on it it's just a piece of dead metal. Even if it's being controlled remotely there still needs to be something on the receiving end to understand what to do with those commands. How this OS is started is another matter, it can be on a locally attached disk (harddisk, SSD, or even an USB stick), it can also be 'diskless', meaning it boots off the network using PXE and NFS for example.


----------



## na5m (Jun 27, 2017)

Thanks for the answer, SirDice!


----------



## balanga (Jun 27, 2017)

I can't help but wonder how Google's computer systems work... It amazes me that I can get a reply to a Google search in a fraction of the time it takes me to find a file on my local system. I wonder if Google's computer ever comes up with *Disk Full* error msg on the operator's console...


----------



## na5m (Jun 27, 2017)

balanga said:


> I can't help but wonder how Google's computer systems work... It amazes me that I can get a reply to a Google search in a fraction of the time it takes me to find a file on my local system. I wonder if Google's computer ever comes up with *Disk Full* error msg on the operator's console...


----------



## SirDice (Jun 27, 2017)

balanga said:


> I can't help but wonder how Google's computer systems work... It amazes me that I can get a reply to a Google search in a fraction of the time it takes me to find a file on my local system. I wonder if Google's computer ever comes up with *Disk Full* error msg on the operator's console...


Unfortunately I've never looked "behind the scenes" (would love to have a look) but I imaging they use clusters of machines. In which case a single machine can be broken without interfering with the operations.


----------



## Beastie (Jun 27, 2017)

balanga said:


> I can't help but wonder how Google's computer systems work... It amazes me that I can get a reply to a Google search in a fraction of the time it takes me to find a file on my local system.


Perhaps because they have a series of building complexes housing thousands of machines all around the world


----------



## chrcol (Jun 27, 2017)

You can have a server class bios installed which allows remote control for purpose of accessing the console (KVM) and other bios functions.


----------



## SirDice (Jun 27, 2017)

chrcol said:


> You can have a server class bios installed which allows remote control for purpose of accessing the console (KVM) and other bios functions.


You mean IPMI, DRAC, iLO and a few more? Those really only allow you to remote control a machine, but if that machine doesn't have an OS you can't do anything besides changing some UEFI/BIOS parameters. The machine still requires an OS to be functional.


----------



## na5m (Jun 27, 2017)

Why not have 1 computer with 1 OS, but that has hundreds of CPUs & disks, instead of the other way around? Seems funny to have hundreds of OS's running all in the same data center.


----------



## ekingston (Jun 27, 2017)

na5m said:


> Why not have 1 computer with 1 OS, but that has hundreds of CPUs & disks, instead of the other way around? Seems funny to have hundreds of OS's running all in the same data center.



We call that a "mainframe".

If  you start as a small company, you don't need a mainframe. So, you start with a Linux/FreeBSD/UNIX server and start clustering them. The high-end clustering software is sufficiently advanced that you end up with about the same capabilities for about the same cost as a mainframe but you were able to grow to get there instead of starting out with a $10,000,000.00 computer.


----------



## SirDice (Jun 27, 2017)

na5m said:


> Why not have 1 computer with 1 OS, but that has hundreds of CPUs & disks, instead of the other way around?


Because one computer can break down, taking everything with it. With hundreds of computers you can have dozens that are down while the service itself keeps working. So it's usually better to create, say, 4 machines load-balanced instead of one single machine that handles everything. With the 4 machines I can take one or more machines offline for updates for example, or it can have a hardware failure. While the other machines take over the load.


----------



## ralphbsz (Jun 28, 2017)

na5m said:


> Does every server in a rack system have an OS installed locally?


Usually, one or more: Today, many physical boxes (a sheet metal box with a motherboard and a few daughter cards, and the motherboard has CPUs and memory) run a hypervisor (such as VMWare or KVM), and run multiple client operating systems in there.  A few years ago, the record was about 30,000 guest operating systems on a single machine (but it was an awfully big machine, a multi-million $ monster, and not powered by Intel CPUs).



> I guess I'm trying to ask if you can have a computer (a box) without on OS on it that is being controlled by a remote computer.


In nearly all cases, every box (sheetmetal/motherboard/cards/...) will have at least one OS.

In some rare instances (I only know about it in supercomputers), many of the boxes will have an ultra-lightweight OS, which does not have normal facilities (no IO, no login, no user/system split), and are used as pure compute servers.  Actually, I don't know for sure that any such machines are still available for sale, but I think some are still running.

There are rare exceptions; in high end supercomputers you can have one "OS image" (single node, since kernel running) that is split over multiple boxes (each with CPU chips, memory, and networking gear).  This kind of thing is only available in multi-million-$ supercomputers, but I've seen 4 physical boxes, each with 8 motherboards, each with 4 CPU chips (that works out to 128 chips) that form a single computer.

But again, the very rare exceptions I'm quoting are mostly there to prove the rule: Nearly always, a box (meaning CPU + memory + sheetmetal + ...) will have an OS.



> Sorry if this is a dumb question.


There are no dumb question, unless you ask the same question twice (then the second one is dumb).


----------



## na5m (Jun 28, 2017)

Thanks for all the helpful responses! This FreeBSD forum is the greatest!


----------



## tingo (Jun 28, 2017)

SirDice said:


> You mean IPMI, DRAC, iLO and a few more? Those really only allow you to remote control a machine, but if that machine doesn't have an OS you can't do anything besides changing some UEFI/BIOS parameters. The machine still requires an OS to be functional.


Some of them also allows to insert media into virtual devices, which the machine can boot from. And then there is network boot. So many possibilities!


----------



## SirDice (Jun 28, 2017)

tingo said:


> Some of them also allows to insert media into virtual devices, which the machine can boot from. And then there is network boot. So many possibilities!


Absolutely. But the point was, the machine needs an OS to be functional. How you start that OS or where you start it from is irrelevant.


----------



## tingo (Jun 28, 2017)

SirDice said:


> Absolutely. But the point was, the machine needs an OS to be functional. How you start that OS or where you start it from is irrelevant.


Absolutely true as well.


----------



## phoenix (Jun 28, 2017)

SirDice said:


> Usually, yes.



Except when the "server" in the rack is just a chassis full of disks.    In which case, that box doesn't have an OS installed on it, there's just cables going from that box to another one that actually has an OS on it.  Or cables going to a storage switch, which then goes off to the servers.

We have several servers where there's a 2U "head" unit with the OS installed, and a bunch of HBAs plugged in.  Those HBAs are then attached via external cables to either 2x or 4x JBOD  chassis each with 45 harddrives in them.

There are datacentres where entire racks are nothing but disk chassis, without any local OS installed.

But, yes, usually you need to have an OS installed on a server in order for it to be useful.


----------



## leebrown66 (Jun 29, 2017)

The latest server I configured allowed for an iSCSI mount during the BIOS phase to boot off.  No need for local disk if your application is appropriate.


----------



## daBee (Jul 5, 2017)

Great thread.  I've been thinking about this as I've been installing several flavours of FreeBSD onto several types of machines.  This type of optimization could be very key to the adoption rates of an OS.  

The vision I had was that "servers" (from rack mounts to workstations to items like Raspberry Pi) could somehow be integrated with minimal instructions.  Much like how drives are hot-swappable, servers might be able to do the same, considering that all of them are essentially on a LAN.  A $300 older server can saturate a LAN channel, so these older units can easily be part of the fore-mentioned cluster for local OS management and distribution.  

It is significant in that there are man-hours and troubleshooting.  Much like how ports are managed with default configurations, directives could be recorded for OS_version-machine_model matchups.  Scripting that with some kind of installation error reporting, would be quite the tool.  If anything, it could be automated with automatic erasure of tested installations, and reports generated for maintainers.  

MacOS has/had (?) something called NetBoot.  Granted, every Mac shipped has an OS installed, but I'm thinking that the BIOS could/should have some kind of protocol where it looks to 10.x.x.x for a negotiated OS installer system of some sort.  If an installation requires static IP assignments, it could be databased and servers are completely databased for DNS tables, maintenance, maybe SNMP applications, etc.  But that would take collaboration with hardware providers...I'm assuming the motherboard manufacturers are responsible for the BIOS design.  I'm surprised that this hasn't already been accomplished, a server taking instructions from its network environment as to how it should be flashed with an OS, and who it becomes on the LAN.  

So a server becomes more of a robot than anything else.  Fresh, right out of the box.  

Ya, great thread.


----------



## uchman (Jul 5, 2017)

na5m said:


> Why not have 1 computer with 1 OS, but that has hundreds of CPUs & disks, instead of the other way around? Seems funny to have hundreds of OS's running all in the same data center.



Well, Of course there is pros with a true multiprocessor system but when go you to scale (say millions of CPUs or more) its not practical or even feasible to build such a single system. Even if it was redundant and modular it would be impossible to build. 

Another problem is the amount of memory that is needed. Today a regular server can have about 4TB of RAM but what if you need 5PB of RAM?

The solution to this is like people have said before, to interconnect many servers into big clusters with thousands of machines. And then adding a very fast interconnect like Intel Omnipath or infiniband for RDMA (Remote Direct Memory Access) over the network. It really boils down to what is practical and economically favourable.


----------



## uchman (Jul 5, 2017)

daBee said:


> Great thread.  I've been thinking about this as I've been installing several flavours of FreeBSD onto several types of machines.  This type of optimization could be very key to the adoption rates of an OS.
> 
> The vision I had was that "servers" (from rack mounts to workstations to items like Raspberry Pi) could somehow be integrated with minimal instructions.  Much like how drives are hot-swappable, servers might be able to do the same, considering that all of them are essentially on a LAN.



I manage clusters at work and I think you should look at some automated installation procedure (maybe cobbler) and then some configuration managament tool or orchestration tool. There are many out there, Chef, Puppet and the new star right now Ansible. Ansible is more like "scripts on speed", but with the ability to orchestrate so its very popular among "DevOps"


----------



## na5m (Jul 5, 2017)

uchman said:


> Well, Of course there is pros with a true multiprocessor system but when go you to scale (say millions of CPUs or more) its not practical or even feasible to build such a single system. Even if it was redundant and modular it would be impossible to build.
> 
> Another problem is the amount of memory that is needed. Today a regular server can have about 4TB of RAM but what if you need 5PB of RAM?
> 
> The solution to this is like people have said before, to interconnect many servers into big clusters with thousands of machines. And then adding a very fast interconnect like Intel Omnipath or infiniband for RDMA (Remote Direct Memory Access) over the network. It really boils down to what is practical and economically favourable.


OK. I think I now kind of can wrap my head around what people are saying. I'm imagining it being like having 1 guy with an epic brain that can control thousands of dudes that have no brain. But if the guy gets hurt or whatever, then the thousands of dudes stop in their tracks? Is this a fair analogy?

[edited to correct spelling]


----------



## SirDice (Jul 5, 2017)

na5m said:


> I'm imagining it being like having 1 guy with an epic brain that can control thousands of dudes that have no brain. But if the guy gets hurt or whatever, then the thousands of dudes stop in their tracks? Is this a fair analogy?


Yes, that's close enough. The other side would be a couple of hundred guys with decent intelligence. If one of those guys get sick the others can easily take over.


----------



## daBee (Jul 5, 2017)

uchman said:


> I manage clusters at work and I think you should look at some automated installation procedure (maybe cobbler) and then some configuration managament tool or orchestration tool. There are many out there, Chef, Puppet and the new star right now Ansible. Ansible is more like "scripts on speed", but with the ability to orchestrate so its very popular among "DevOps"



OK, so there is an approach already.  Do the servers need to be flashed initially with something before these can take control?  The challenge I see in any application is the very introduction of a "black server" (fresh, right out of the box) to the point of control over the LAN.


----------



## SirDice (Jul 5, 2017)

daBee said:


> Do the servers need to be flashed initially with something before these can take control?


That depends. I've worked for companies that installed a basic OS on the server by hand (or remotely via IPMI) and let Ansible or Puppet take care of the rest. And I've worked for companies that had a whole automatic infrastructure where the machine would initially PXE boot to an automated installer. It all depends on how much time you can invest in setting this up. For a handful servers the automatic approach may be a bit too much work to set up with little to gain in the long run (how often would you need to re-install, how long does it take, etc), but for hundreds of servers it may be worthwhile to spend the time and effort to automate everything.

Note that enterprise-grade remote control like IPMI, DRAC and iLO always work, even when the server itself is switched off. You would need to physically unplug the machine for IPMI/DRAC/iLO to stop working.


----------



## daBee (Jul 5, 2017)

OK, this is exactly what I'm talking about.  Reliance upon BIOS and semi-intelligent NICs.  

`Power on –> BIOS –> Network Card’s PXE stack –> Network Boot Program (NBP) downloaded using TFTP from server to Client’s RAM –> NBP’s responsibility to perform the next step (a.k.a. 2nd stage boot).`

I don't see why there can't be a ruby gem that works in tandem with bash scripts and a master controller database that can direct this type of stuff.  

Ideally, a proper solid database and web stack with CDN, something along the lines of a half to full rack, would be the target market.  OS images can be public domain for generic DHCP, then fine tuning through SSH.  I'm guessing that on a full rack, something like this could be performed over a long liquid lunch.


----------



## SirDice (Jul 5, 2017)

daBee said:


> I don't see why there can't be a ruby gem that works in tandem with bash scripts and a master controller database that can direct this type of stuff.


Because you mentioned Ruby, you may want to look into Puppet (Ruby based) and Puppet's ENC (External Node Classifier). That pretty much does all that for you. 

The PXE installer could automatically install/configure a basic OS plus Puppet, then Puppet can take care of installing/configuring everything else based on the information from an ENC. That's not something you can have up and running in 10 minutes, it's going to take a lot of effort building it. But once it's done....


----------



## daBee (Jul 5, 2017)

OK so over 4800 modules working with it.  Sounds like it's the deal.  So for a common hearty web stack of db-driven schtuff, what kind of time investment do you think would be needed for something like this?  

I don't know why, but designing systems with a 'hot-swap' mentality, is just super sexy.  This really starts at $1200/year.


----------



## SirDice (Jul 5, 2017)

Typical webservers we use at the moment are all dual 6-core Xeons (plus Hyper-Threading gives you a total of 24 logical cores), 64 GB RAM and 2 x 2TB HDs. The DB servers usually have 300+ GB RAM and 2-3 TB of SSD in RAID10. Definitely check SuperMicro, FreeBSD usually runs well on most of their models (with some exceptions) and they're reasonably priced. Never had any major issues with their servers. 

The setup I built for a client currently consist of 2 servers running HAProxy (load-balancer), 4-6 webservers and 2 or 3 DB servers. For management we have 2 log servers (slow CPUs but a lot of storage) and 2 control servers (Puppet masters, Zabbix, Poudriere package building, etc). Almost everything is set up with Puppet (took me a couple of weeks to script everything). Thanks to HAProxy I can easily take one or more webservers out of the pool (to update for example) without interfering with the running sites. Rebuilding a webserver is done initially by hand for the OS, then Puppet takes care of the rest. It takes about 30 minutes to rebuild it from scratch.


----------



## daBee (Jul 5, 2017)

Yeah I've been looking at the Supermicro for just that setup.  Interesting you went with SSDs for DB.  Speed, but the longevity is my concern there.  

Is there a use for flashing SSDs locally on USB (MacOS) from a FreeBSD image that's generic?  So ports are installed (bash, ntp, file sharing, pico, etc.), then scripting various changes once online?  I guess using stick SSDs would need some kind of interface for that.  Ah, maybe manual is the answer using RAID for the OS volume.


----------



## SirDice (Jul 5, 2017)

daBee said:


> Interesting you went with SSDs for DB. Speed, but the longevity is my concern there.


It wasn't my choice. I just use whatever they think they need. But the rational was that they needed a lot of IOPS or else the DB simply can't keep up (it was already tuned as fast as possible). That's also the reason why the DB servers have a ridiculous amount of memory.  But these are enterprise grade SSDs, they typically last a lot longer than their consumer counter-parts. And everything is either mirrored (OS) or RAID10 (data). So it's not a big problem if one of the SSDs decides to quit.



> Is there a use for flashing SSDs locally on USB (MacOS) from a FreeBSD image that's generic? So ports are installed (bash, ntp, file sharing, pico, etc.), then scripting various changes once online?


I never bothered with it. It takes less than 10 minutes to install FreeBSD from normal installation media. I then configure pkg(8) to use our own repository, install Puppet and let it take care of the rest. So it's only ~15 minutes of actual work, the rest is installed and configured automatically with Puppet.

Because they've only got a handful of servers and I rarely need to rebuild it wasn't worth the effort to automate things further.


----------



## ralphbsz (Jul 5, 2017)

daBee said:


> Yeah I've been looking at the Supermicro for just that setup.


Supermicro is also my favorite hardware vendor (in those cases where I have the freedom to choose hardware).



> Interesting you went with SSDs for DB.  Speed, but the longevity is my concern there.


It depends.  To begin with, SSDs can live for a very long time, if a lot of the workload is reads, and only a small fraction is writes.  Such workloads do exist, in particular in database applications.  Another thing that modern applications do is to optimize their write traffic to prevent "false write sharing" on SSDs, by doing interesting logging techniques.

And remember that hard disks don't live forever either.  Matter-of-fact, modern hard disks are specified to have a maximum data traffic per year; for most models I've seen recently, that specification is 550 TB per year (so a 10TB disk can only be read 55 times per year, or roughly once a week).  Note that for SSDs the endurance limit is determined by writes, for spinning rust by total traffic, which makes a huge difference.

And ultimately, if an application simply needs the write speed of SSDs, then endurance becomes secondary.  There are customer environments where SSDs need to be replaced every few years.  Obviously, this is expensive (service personnel needs to be scheduled, spare parts need to be stockpiled and shipped, support contracts need to be written and paid for), but having a broken computer may be more expensive.  Obviously, this only makes sense with some sort of RAID mechanism, and in most cases only if the replacement can be done "hot", on a live system.

Think of it this way: If you build a high-end storage server, the computer itself (the CPU box, with PCIe disk controller and network cards) may cost roughly $10K, depending on how much memory you put in there.  The external disk enclosures (say 5 extra rack-mount JBODs with room for 80 disks each) can easily add another $50K.  The disks themselves start at $100K (if you use reasonably inexpensive enterprise-grade near line drives), all the way up to quarter million or half a million (if you use flash storage like SSDs).  Add to that some software, vendor overhead, and support contracts, and you have a very expensive single system, with most of the cost in the storage disks themselves.  But why do people buy these very expensive systems?  Because they need them to make money.  Usually that means that the data on the server is more valuable than the cost of the server, by a large margin.  In the overall scheme of a typical corporation, the cost of IT infrastructure is still a minor factor, even though a high-end server cluster (like the one SirDice described above, or like the storage server I'm talking about) can cost as much as a private residence.


----------



## uchman (Jul 5, 2017)

daBee said:


> OK, so there is an approach already.  Do the servers need to be flashed initially with something before these can take control?  The challenge I see in any application is the very introduction of a "black server" (fresh, right out of the box) to the point of control over the LAN.



No, the only thing you need is to set it to boot from the network (PXE boot) all server hardware supports this. The procedure is basically that the network card does a DHCP request to the network and a properly configured DHCP server responds with a IP and also a location on the network where it can find a bootable media. Most oftenly its a tftp server. From the server it downloads the installation program which is run and the OS gets installed. This is a over simplification but you get the point.


----------



## daBee (Jul 6, 2017)

ralphbsz said:


> Supermicro is also my favorite hardware vendor (in those cases where I have the freedom to choose hardware).<snip>



Longevity was more the issue, as it wasn't long ago that SSDs were much more expensive than they are now.  I have never had the opportunity to use a stick SSD, but I would prefer the hot swappable drive-format SSD for OS as well.


----------



## daBee (Jul 6, 2017)

uchman said:


> No, the only thing you need is to set it to boot from the network (PXE boot) all server hardware supports this. The procedure is basically that the network card does a DHCP request to the network and a properly configured DHCP server responds with a IP and also a location on the network where it can find a bootable media. Most oftenly its a tftp server. From the server it downloads the installation program which is run and the OS gets installed. This is a over simplification but you get the point.



Yes, sounds like Netboot.  This and Puppet sounds like it's all it needs for an accelerated introduction.


----------



## chrcol (Jul 15, 2017)

SirDice said:


> You mean IPMI, DRAC, iLO and a few more? Those really only allow you to remote control a machine, but if that machine doesn't have an OS you can't do anything besides changing some UEFI/BIOS parameters. The machine still requires an OS to be functional.


yes but you can also install an OS with the bios tools.


----------

