This is just a basic primer on server storage intended to get you up to speed on what technologies exist, how and when they’re seen in servers, and for you to make not only an informed decision on the hardware that you want/need but also so that you know how to quickly resolve any issues that may occur.
It’s an important topic to cover, as often people who are first moving into enterprise hardware, or often people who are configuring enterprise hardware for the first time (having come from an enterprise environment where someone else had done it for them) may not be aware of some of the “gotchas” in storage that can delay or waste hours of your time finding out.
Servers by default, almost always have hardware RAID cards
New Terms covered:
- RAID
- JBOD
- Hardware RAID
If you’re reading this article, you probably know what RAID is (if you don’t, a quick summary is that RAID is a method of combining multiple disks for redundancy/performance, like writing everything to two disks so that if one fails your data is still there, or splitting data across two disks so you can read/write twice as fast).
Enterprise servers, by default, almost always comes with an integrated hardware RAID card which will not allow JBOD operation. JBOD (literally “Just a Bunch Of Disks”) is the standard operating mode for personal computers where you plug a disk in, and the operating system can see it.
These RAID cards are almost always Broadcom/LSI based MegaRAID controllers and built into the motherboard itself, and are designated “hardware RAID” because the work is done by the chip and not offloaded to the processor, then depending on the manufacturer certain models will come with optional, or preconfigured upgrades to these cards for more common use-cases. As an example, the HP Gen8 series will often have the B120i on the board, then a P420i mezzanine card mounted.
It is safe to assume, that if you’re buying a server from the primary OEM manufacturers (HP/Lenovo/IBM/Dell/Cisco etc) they’ll have an onboard RAID controller that will not allow JBOD passthrough to the OS.
RAID vs JBOD configuration
New Terms covered:
- HBA
- IT Mode
- Software RAID
It’s worth noting, that neither RAID, nor JBOD are bad or worse than the other. They’re just different technologies for different needs. And, in many modern scenarios, are both used simultaneously.
RAID can improve performance of disks and/or offer redundancy to disks for the first level of data protection. I won’t go deeper, but RAID isn’t a backup, it’s redundancy.
Meanwhile, JBOD mode is a direct passthrough of the drives to the operating system. This is the usual method deployed by personal computers, and is offered on servers by HBA cards, RAID cards operating in “HBA mode” or RAID cards operating in Initiator Target (IT) Mode. JBOD mode is convenient when you do not want/need the benefits supplied by a RAID array, or wish to use a software RAID solution such as ZFS.
Between Hardware and Software RAID, Hardware RAID is usually more performant and reliable than software RAID, but is limited to the capabilities of the physical chip which often means it misses out on newer technologies, additionally it requires that additional chip. While software RAID is usually slower as processing is done by the generalist CPU (which also eats at available CPU performance) but is able to stay up to date with the latest technologies and work on any device.
Finally, if you do wish to use ZFS, it is recommended that you purchase an LSI branded card in “IT mode” as opposed to running “HBA mode” on a server brand card as the latter sometimes (but not always) emulates JBOD via software without passing through SMART capability and sometimes performance.
Other compatibility considerations
New Terms covered:
- SATA
- SAS
- Sector/Block Sizing (512/520/4K)
Finally, when deciding on your needs it’s important to consider the other compatibility characteristics of the drives you wish to run (or, alternatively, choose drives based on your controllers).
Almost all servers will have a SAS compatible backplane, but may not have a SAS controller, all SAS controllers are also SATA capable. Most enterprise servers will, though but it is an important thing to check when purchasing. Back on the HP B120i vs the P420i, the B120i is SATA only, while the P420i is SAS/SATA.
Knowing how the technologies differ isn’t really important, but a quick summary is that SAS directly connects drives to the controller via parallel data streams, while SATA is a serial link between devices and the controller. More important to know is that the industry has standardised on SAS as the format of choice for enterprise drives, and SATA for personal compute. Which means you can fundamentally assume that if it’s a SAS device, it’s built for server workloads, but you’ll need to research whether a SATA device will be suitable for a server workload. Similar to the RAID vs JBOD discussion, SATA isn’t bad, it’s just different. Additionally there are now other standards such as NVMe but these will be ignored in this document.
You’ll also find there’s 6Gb and 12Gb SAS drives available on the market. As they describe, this is the available bandwidth for a SAS channel and both sides need to support the speed to use it. (SATA maxes out at 6Gb). However they are fully backwards compatible, so a 6Gb drive will work with a 12Gb controller, it will just be limited to 6Gb and vice versa. For the most part, it honestly doesn’t matter on Hard Disk Drives (as opposed to Solid State Drives) and you can just use the measure as a means to determine age. The reason for this is that 6Gb/s data speeds are more than double the maximum throughput of even the latest 10K SAS drives. So a 10K SFF 2TB SAS drive that maxes out at 275MB/s transfer speeds won’t go any faster if you increase the bus cap to 1.5GB/s. However if you’re intending to buy SAS SSDs, then it will matter as they can easily hit the 6Gb cap.
Finally when talking about SAS drives in server workloads, you must pay special attention to the Sector/Block sizes. I mentioned earlier that hardware RAID may miss out on newer technology and this is most apparent with the new Advanced Format/4Kn drives. The old standard was 512 bytes and this has been universally supported for many years. A 512 byte sector size means that each addressable space can hold 512 bytes, so a 6KB file will consume 12 blocks, while a 256 byte file will consume 1 block (not half a block).
With storage devices growing in size significantly, and normal data also being significantly larger, a new standard has emerged of 4KB sector sizes. As it only needs to parse 1/8 of the address space to read/write data, it is potentially 8x faster. However older controllers do not support 4K (Advanced Format) sectors and so a newer 4Kn (4K native) hard drive will not be usable on a controller without support (such as the P420i). 4Kn support is basically universal on storage controllers released around Intel Haswell onwards (2014 and later, Intel Xeon E5 v3, Dell x30, HP Gen9, IBM M5 etc).
Thus, if you have an older controller you must buy either a 512 drive (older), or a 512e (512 emulated) drive. The latter is often found on newer drives that are compatible with older controllers and they are in actuality 4K drives that allow a controller to assign a 512 byte block to a 4K sector.
Finally, be wary of some drives that come from specialised storage controllers with names like 3PARC and NetApp. They’re controller manufacturers that have gone out of their way to use a wild and wacky block size to prevent you buying drives from third parties. They’ll use block sizes like 520, 534 or others and can sometimes, but not always, be restored to 512 and used in other devices.
Last, but not least, some manufacturer’s are just greedy. And can prevent you using drives not flashed with their firmware (usually they’re branded with their brand, such as an HP drive that’s just a normal Seagate drive with HP firmware). The HP P440 is a good example of this, where if it’s in RAID mode it won’t allow configuration of a non-HP SAS drive but will run any old SATA drive, but can be switched to HBA Mode to passthrough the SAS drive.
My operating system can’t see my drives
Which brings us to the all important part of installing an operating system. It’s all well and good having drives, but the OS actually uses them and may not see them.
If you have a RAID controller, as mentioned earlier, it will not allow JBOD operation. In some cases it may have an HBA mode, but results can vary, if I get the opportunity I will go through all of the servers and controllers we sell and compile a list of what supports what, you can also sometimes reflash the firmware to an HBA with IT mode, but I won’t be covering that process here as it’s not supported operation.
So, through standard operation if you’re running a RAID controller in default (RAID) mode, you MUST set up a RAID volume and pass it through. However there’s nothing to stop you setting up a RAID-0 on just one drive if you only have the one. Once the virtual volume exists, it will become available to the OS*
*Most of the time, HP controllers can sometimes require that you set it as a bootable volume. Others might as well.
If you’re running an HBA or a RAID card in IT mode, the drives should already be available to the OS.
But, you discover, “Windows still can’t see the drives!”. Windows doesn’t usually have RAID drivers preloaded, even Windows Server. Linux usually has them natively available, but not Windows.
So if you’re looking to install Windows to a server, you’ll need to seek out the drivers for the RAID controller you have, put them on a separate USB and select them for use during the Windows install.
Not all RAID controllers are supported on all versions of Windows but that’s not to say they won’t work. Drivers are signed for certain versions of Windows, and outside of that they won’t pop up as automatically found drivers during the Load Process, but often you’ll find drivers for older operating systems will work perfectly well on later versions, so you can select the older version and just force them to use it.
However, if you’re not comfortable doing so, you can also just purchase another RAID card, like an LSI 3008, which not only supports 4Kn, but 12Gb SAS and has both RAID and Initiator Target capability, and easy to find drivers for the latest versions of Windows making it an excellent all rounder for all scenarios.
Enterprise SSDs are expensive, can I use consumer SATA SSDs?
The short answer is YES, absolutely. There are limitations and drawbacks, and I’m not going to delve into DRAM considerations but you certainly can use them, just DO NOT expect Enterprise Reliability in them.
As long as you accept there will be drawbacks, modern consumer SSDs are very fast, cheap means of storage and if you’re running them in RAID arrays, you can at least get some warning before they start experiencing problems.
Problems that can occur with these kinds of drives revolve around the fact that NAND flash has a limit to the number of writes it can take, and that reliability exponentially decreases as manufacturers run more bits per cell. So often manufacturers who make enterprise SSDs tend to account for these drawbacks and run two (and these days three) bits per cell max and overprovision the available NAND and add plenty of DRAM to reduce writes. While consumer SSDs are made to maximise capacity and performance at the expense of everything else so they can hit a minimal price point. Using as many as four bits per cell (Samsung QVO), no RAM and no overprovisioning.
So by all means, if your choices are between 10K HDDs and consumer SSDs at the same capacities, and data integrity is second to performance, they’re a perfectly good option. Just make sure you run them in RAID1 at least.
What about ZFS, that’s popular these days?
Nothing wrong with ZFS, much like RAID vs JBOD, it’s just another technology. For the best chance of success, you’ll want an LSI branded card and probably 12Gb for maximum compatibility. Something like the LSI 3008. ZFS is simply a software RAID solution that’s getting continuous development, so you will offload a lot of processing to the CPU and RAM but with modern 20/40/60 core machines and hundreds of GB or terabytes of RAM that shouldn’t be a big problem. One great thing about ZFS if that you can pull drives out, shove them in another machine and the array will be recognised without having to have the same hardware RAID controller in the new one.
Anything else?
Cables, cables matter, have a look at our short primer here on the most common ones you’ll see on enterprise servers.
Conclusions
Hopefully this acts as a good primer for understanding storage on servers as it can differ significantly from storage on personal computers, something often overlooked by people who are moving into enterprise hardware for the first time.
Fundamentally, it’s better, but there’s plenty of considerations and planning that has to go into it. You may not get it perfect the first time so try to plan for something to go wrong and if in doubt, ask.
But key takeaways:
- HBAs pass through drives to the OS, RAID configuration must have a RAID array for the OS to see it
- Not all RAID cards support HBA modes
- Most servers have RAID cards only, but you can add an HBA via a PCI-E slot
- There’s a few compatibility considerations, especially around block and sector sizes like 4K and weird sizes
- Windows doesn’t natively support most RAID controllers, so you need to add drivers manually
- Consumer SSDs are okay as long as you understand their limitations
- And get an LSI card for ZFS
But at the end of the day, feel free to ask questions. We’re going to try and write up a few blog posts like this dealing with subject matter that’s relevant for home labs and small to medium business to move into enterprise servers because they’re simply just better.
0 Comments