Hey folks. I just ordered my HL15 chassis + backplane a day ago and now I’m figuring out what I’m going to put in it.
My planned usage:
Primarily storage - I plan to store hundreds of thousands of hours audio - ingesting about 24GB / day. I also plan to read all of it remotely from another machine on the same network multiple times a day.
I think I’m going to be optimizing for storage space + transfer speed.
I think I need a bit of a sanity check on my list of parts. I’m used to working with consumer parts but this is my first time building something enterprisey from scratch.
How much is “all of it”? The daily 24GB or the hundreds of thousands of hours? How much data do you need to move across the network each day and how fast does that need to happen?
What are your plans/needs for RAID and backup of the data?
You might want more smaller drives than fewer larger drives, or SAS drives, if you want to get that data over the network at 40Gb/s.
Every job will process all of it (the hundreds of thousands of hours). At some point, running that job will take more than a day and that’s fine. I don’t need overlapping runs.
I just want it to be as fast as possible without spending too much over $3k (an artificial limit, I can go higher if I need to).
Backup + Raid: No plan for backups. I’m ok with data loss here. Possibly multiple raid0 arrays?
If you’re reading the same files multiple times a day it will come from ARC most probably. Those 4 SATA drives won’t give you much speed directly so the first read might be slow.
You won’t need a CPU that powerful just to serve files, even at 40gbps. The fastest access to files will always be locally. I’ll ask the dumb question, just to be sure; the other machine has some proprietary software or hardware, or for some other reason needs to be physically separate from the HL15 and can’t be run in/on it either directly in the OS or a VM solution?
Do you know what’s the minimum I can get away with for saturating 40gbps (assuming everything else is in place)? I thought a 7402p was being conservative.
The reason I’m keeping them separate is because the box doing all the processing is going to have multiple GPUs and I don’t think they’d fit in my HL15 after I add networking, storage add-in cards and the HBA.
My main point was just about CPU cores. You’ll probably find the CPU running at only a few percent of load. The CPU doesn’t have to do a lot when reading the disks or sending data over the network. A bit like some gaming builds that don’t really need a powerful CPU because the GPUs are doing the majority of the workload. It seems like the HBA and NIC you listed are both PCIe 3 x8 cards, so in a sense the minimum would be any CPU that supports that, since you haven’t defined much of any additional workload (transcoding, encryption, VMs, …). I would think even the HL15 “Full Build”, even though the CPU used has 1/15th the compute of your Epyc, can still push data around about the same. Having more compute available is certainly not a bad thing. I didn’t go with the full build. But, if someone wants to come in and correct me that’s fine.
That’s what I thought. Just felt like I needed to ask.
As Fossil said, the bottleneck isn’t going to be with the HBA or the NIC or the CPU. It’s going to be with the “spinning rust”. A single Seagate Exos X22 lists a maximum sustained throughput of 285MB/s. That’s very much ‘best-case’. A typical sustained transfer rate for a 7200 RPM drive is more like half of that. You can have some data cached in RAM or SSD, and the drives have a small cache, but a 4-disk RAID 0 is only going to return data at say a typical sustained (4 disks * 125MB/s * 8 bits/byte =) 4 Gb/s. I think you’d be better off with fifteen 6TB or 8TB drives than four 20TB drives. Even then that might only put you in the 15-20 gbps range. 15K rpm drives might get you to 25 gbps. To get to 40, I think you’d need dual actuar disks, SSDs, or a chassis like the 45Drives Q30 or something from a different company like Supermicro that has 24 or 36 3.5" drive bays.