If you are getting 15 drives, you should start determining what an optimal configuration of the drives is in terms of performance and data protection. There are many options, and the more experienced people here can guide you better. If you don’t have your plan laid out ahead of time, you might make a bad choice, and once a NAS starts filling up with data, you may not want to redo everything. With 15 drives, how many will you use for parity, and how many vdevs?
2 Likes
I don’t truly need SAS drives. I have only ever used SATA but my understanding was that the anticipated working life for SAS drives was longer, so that was my reasoning for going that route. The 250-300TB is “raw” space and does not account for RAID overhead. I would like to have maybe 15TB usable space for device backups (laptops, tablets, and phones) and shared family photo backups on Immich. Beyond that, I am looking for 100+ TB usable space for TV shows and movies that would be shared with friends and family. I am definitely open to buying refurbished drives but they would ideally come with at least a 2 year warranty to cover the possibility of early failure. As for RAID / parity, I’m still reading about the different options but tentatively plan for 30-40% of the raw storage space to be devoted towards that task. Any specific recommended readings or guidance otherwise is welcomed!
I’m not sure what the best ZFS explainer is, Lawrence Systems probably has one. The trouble is detailed primers could take you off into the weeds for your relatively simple use case.
First, RAID is not a backup. For anything that is dear and irreplaceable, you need a 3-2-1 backup strategy or such, You need another copy on a different physical box and/or preferably in a different physical location. RAID is a scheme for high availability, but people abuse it assuming that they won’t corrupt the array or multiple drives can’t fail when they rebuild the array.
Others can chime in, but I think two typical HL15 configurations people are running who don’t need high write IOPS are either;
- RAID Z2 with two VDEVs of 7x drives each, and one hot/cold spare
- RAID Z2 with two VDEVs of 8x drives each, making use of a PCIe slot HDD bracket to add a 16th drive.
In the first configuration, four of the 14 drives’ space are for parity (28.6%) and then there’s a bit more for ZFS metadata. In the second configuration, four of the 16 drives’ space are for parity (25%) and then there’s a bit more for ZFS metadata.
Backblaze publishes failure rate data for the tens of thousands of drives in their data centers. Although not exhaustive, it can be a guide to choosing which used models might be more reliable;
New or used, you should try to get drives with different manufacture dates, perhaps buying from different vendors, or staggered purchases. Since all the drives in the system are going to be on for approximately the same amount of time, it is more likely that they would tend to fail around the same time if manufactured close together.
3 Likes
This has been super helpful for me. I plan to go with RAID Z2 with two VDEVs of seven drives each with one cold spare.
My last questions for the moment are regarding flash storage…
I’ve done some reading and encountered differing opinions regarding potential benefit from cache drives for metadata and SLOG/write cache. I’d like the ability for quick write speeds to offload large files to the HL-15 device and my understanding is that SSDs could help by “holding” those files while they’re written to the slower rust HDDs.
As of now, I don’t plan to be editing video or hosting games on this device. Also worth mentioning that I’m tentatively planning on using unRAID for my OS, which apparently boots from a USB drive.
-
The HL-15 2.0 as ordered comes with a single 1TB NVMe boot drive. Do I need an additional NVMe boot drive for redundancy of the Houston / Rocky Linux system? Or any other compelling reasons I should get more NVMe drives?
-
Should I get some SSDs for caching purposes? Is this the best place to store and run virtual machines for docker containers? Recommended quantity and array for the SSDs? Any adapter cables or PCIe expansions needed on the new ASRock mobo to support these SSD in addition to the 15 HDD and NVMe drives?
-
If I’m running unRAID from a USB thumb-drive, recommendations on which one? Best method for parity of this data in the event of a USB drive failure?
I’m sure it’s painfully obvious that I’m treading water in foreign seas here, so again, thank you and everyone else on the 45homelab forum for sharing your insights and expertise!
1 Like
These are sort-of mutually exclusive statements. Youre either going to run Houston/Rocky as your OS or UnRAID. I’m no UnRAID expert, but my understanding is the ZFS features they added recently are more for the cache than the data array. UnRAID at its core manages the data disks in a way that is different from ZFS. UnRAID allows one to mix drives of different sizes but still retain some redundancy, whereas ZFS effectively requires all the disks in a VDEV to be the same size. UnRAID comes with a benefit of being able to grow the pool slowly a disk at a time rather than having a big budget outlay for 8+ drives, but at the cost of relative read/write performance–files aren’t split into chunks across disks they way they are with ZFS for parallel reads and writes across disks.
How you handle boot drive redundancy and whether you need caches and how to implement them really depends on which OS you use. It will also depend on how fast the client connections are to the network (1G, 2.5G, 10G, Wifi, …) and how much RAM you get.
It depends on what the containers do. Are they heavily reading and writing to their own application data, or just doing most of their work in memory? A single SSD will be faster than a single HDD, but may not be faster than a RAIDZ2 8 disk VDEV of HDDs. Most container applications are not using the data pool at all or just saving and loading a minimal amount of application configuration info with most of its disk access coming from the main data pool (Immich, Plex, Jellyfin, Nextcloud, etc,). I’m not sure about UnRAID, but I’m imagining for it a container might start a few seconds quicker from SSD, but once started you wouldn’t notice much if any difference. Now, if you were running Windows VMs or local LLMs that would be different, but I don’t think that’s part of your use case.
3 Likes
Bought mine … now to find some hard drives
2 Likes
Hi Homelabers,
Big news: HL15 2.0 production is rolling! Shipping starts soon, so if you haven’t placed your order yet, NOW is the time.
PLUS… we’re giving away a FREE HL15 2.0!
Follow us on Instagram
Like + comment what you’d do with it
Share the post
Follow & share our posts on Instagram, YouTube, X & LinkedIn to join the hype. Don’t sleep on this.
#HL1520 #HomeLab #Giveaway
Stay connected on Instagram - https://www.instagram.com/45homelab/ , X - https://x.com/45Homelab, LinkedIn - LinkedIn Login, Sign in | LinkedIn, and YouTube for updates and more cool content!
Is there a way to enter without social media? I’m afraid I avoid social media these days.
If not, I get it, but wanted to ask.
Thanks.
Kevin
I run unRAID in the “traditional” way, with individual drives rather than using ZFS. You can find more detailed explanations on the unRAID forums of the difference, so what I’m going to say applies to the traditional way unRAID works.
In the traditional unRAID setup, files are written to individual disks. This has the advantage of allow only discs in use to be spun up (saving power) and if you have a catastrophe, you can do some data recovery from the individual HDDs as they are all readable directly by other computers.
The tradeoff, is that as read and write operations are on to a single HDD at a time and you are limited by the speed of that drive. (You also lose bitrot projection offered by ZFS, among others).
In this configuration, I think having a cache drive or pool made of SDDs is worthwhile for temporary storage of files as well as metadata files for Dockers.
(As an aside you’ll want to keep metadata backed up, for which there is a plugin called “Appdata Backup” you may wish to check out.)
In unRAID, a process called the “mover” moves files from cache to the array over night as a default. You can also configure different shares to never or only use the cache drive if you want to.
I find a 1 TB sized cache pool more than adequate. If you regularly add file(s) in >1TB quantities, you may want more space. I use a mirrored pool so if I have a drive failure, I can swap out the faulty drive and replace it without much downtime. (Understanding, of course, that redundancy is not the same as backups.)
I believe you can use the ZFS pools not just for cache but also for the main data pools, though I have not explored that myself. If you’re using a ZFS pool, reading and writing should not be limited by a single drive’s speed, so a cache of SSDs may not make sense there. As I’ve never done that, I cannot really speak to that.
Only if you want to. You can put NVMe drives in to an unRAID cache pool. unRAID boots from USB, but if you want to boot to other OSes running on bare metal then you may want to use the NVMe slot for that purpose.
If you want multiple NVMe drives, you can look into a PCIe riser card for NVMe drives. Some of these cards require PCIe lane splitting to be configured in the BIOS, so you’d need to explore how many lanes you have available and how they can be configured before going down that route, though.
I’m not familiar with that motherboard. You can put regular SSDs into the slots of the HL15, though. There are 3D printable files for drive holders for that, or you can buy them pre-printed from 45drives.
SpaceInvaderOne does great videos on unRAID topics on YouTube https://www.youtube.com/c/SpaceinvaderOne, and he tested a number of USB drives to failure a few years ago. The one that did best in that test hasn’t been available anywhere I can find it for quite a while.
Some general observations/recommendations for which I am aware of no formal testing to back them up:
- USB 2 drives are more reliable than 3. As unRAID loads in to RAM when it starts, the speed difference really isn’t an issue.
- Drives with metal cases are thought to disperse heat better than ones in plastic cases.
Really, though, the USB drive doesn’t get much wear and tear. Data is read off during boot, but very few writes occur.
AFAIK there isn’t any way to mirror or parity the USB drive itself. The license for unRAID is tied to a specific USB drive, but in the event of a failure you can change to a new drive with the online license management. I did that once years ago, but it’s been so long I dont remember the details, though I do not recall it seeming particularly difficult.
If you are interest in unRAID I imagine you’ve already found their forums https://forums.unraid.net/. There are people there who (like here) are very helpful, and know far, far more than I do.
Have fun!
Kevin
2 Likes