I’m looking for some guidance on how to setup my NAS build - as I have no experience with most of these things to know which will deliver the best performance for my use case. Long post ahead, your input is appreciated.
Use Cases
Desired uses for this NAS.
ML share to host various models (~6gb files) & datasets that can saturate a 10g nic
Database share to serve [[PostgreSQL]] database(s) for [[Services|Containers]] that can saturate a 2.5g nic
Can one DB server other containers? Would I need multiple [[PostgreSQL]] instances or can I use one?
Data shares to serve…
Files for [[NextCloud]] container
Media for [[Jeffylin|Jellyfin]] container
[[Backups]]
NVR Datastore, I would like to process my security footage with AI using Frigate - which I believe is a different use scenario from a share which is strictly for NVR storage. Would using a dedicated NVR share and “video being processed by AI” share be a better config for any reason?
All of the above simultaneously, ideally bottlenecked by nic speeds - because of course I want everything.
I’d like to know how to organize my devices / pools to achieve as many of the above use cases as possible, and am looking for recommendations.
Are there any TrueNAS features that I would miss with Houston?
I’d like to use Ubuntu 20 + Houston to manage this device since I do not use the additional features on my current TrueNAS setup and the access permissions there are an enigma to me.
The purpose of this machine is purely to be a NAS, so I do not need most of the additional features in TrueNAS.
How many of the available NVMe drives do I need to saturate a 10gbe nic for the Model / Dataset share?
Idea: Stripe the 2x 2tb NVMe 4.0 drives for speed and backup the data in case of “boom”. (by pool, snapshot, rsync, carrier pidgeon, etc.)
Which of my use cases / shares would benefit most from adding a SLOG and/or L2ARC device or are they unnecessary?
I’d like to avoid purchasing more drives, but do need ssd’s for the database share?
Hi @Specter, Here are my recommendations for your questions below
Without knowing the full requirements of each use case looking at the Storage drives you have I would recommend making a RAIDZ1 ZFS pool with the 3 HDDs. this would be used for your file server needs and backups.
You could then also make another ZFS pool with a mirror of the 1. 2x Silicon Power 2TB UD90 NVMe 4.0 Gen4 this would be for the database and high IO use cases
as for the, you could add this to the other mirror pool as a second VDEV for additional storage or depending on how much RAM and metadata you could use these as a ZIL or SLOG or special vdev for the HDD Z1 pool to give it a bit more performance
If you would like you could reach out to info@45drives.com and purchase time to have a discussion call with our team on your use case and they sit down and better scope out this for you
We strongly recommend Houston over TrueNAS, Most of our enterprise servers we setup and manage using Houston and we do not have a lot of experience when it comes to TrueNAS( We stopped testing it after they went from FreeNAS to TrueNAS). With that we can do everything TrueNAS can do with Houston, I would argue there are more features with Houston with our disks tab and aliasing we use to be able to alias the exact slots on the HL15 into the UI in a nice graphic to easily be able to identify a drive.
As far as I know, there are no major features TrueNAS would do over the opensource Houston we use
Our Houston UI supports both Rocky8 and Ubuntu20 so you will have no issues with using Ubuntu 20 as the OS if you want to use Houston to manage your server.
You should be more than OK with using Houston if that is the case
Looking at the Specs on the NVMe drives you have they are rated for 4,125 MB/s which is 4X the speed a 10Gb/s connection can handle (10Gb=1GB) So you are already going to be bottlenecked in terms of throughput with 1 NVMe drive but it’s the IOPs that may cause issues for some of these use cases.
That is what I was thinking also. Have the 2 X 2TB drives in a mirror to allow the full IOPs and bandwidth of this NVMe to be used. If you wish to strip the NVMe drives together you would essentially be getting both drives or performance but you would have no redundancy so if a drive were to fail you would lose all your data on that strip. If you strip the 2 2T and then add another vdev with the 512G this will give you redundancy as each drive would be mirrored to its partner but also allow you to get 2 drives of performance out of the pool. but I think 2 NVMe drives of performance might be overkill for your 10G connection
This would come down to how much RAM and how many IOPs you are trying to get out of the system. In all cases, we would first recommend maxing out the RAm in the server before adding an L2ARC as RMA is always going to be faster and more beneficial over a flash SATA/SAS drive.
As for a SLOG, this would depend on how much data you are planning on writing to the server at a time and how big these files are. Again this may not be needed if the storage pool is set up.
This would depend on how many IOPs your database workload needs. without having exact requirements it is hard to say
All this being said again if you want to have an exact recommendation from an architect on our team who can design and spec out all your use cases please reach out to info@45homelab.com.
Thanks a million for the detailed response! Incredibly helpful and informative. Wish I’d asked these questions sooner / as they arose.
Pool Setup - Thank you!
Pool 1: RAIDZ1 ZFS w/ 3 HDDs for Media / Backup / NFS / SMB Shares
Pool 2: mirror of 2x 2TB NVMe for Database Share
2x 512GB Drives
SLOG for Pool 1
ML Share Use Case:
1. Models: read a 7 GB file at line speed
2. Datasets: read / write many small files >10 MB and (ideally) support rapid archive compression/decompression
My HL15 is sitting on the coffee table in my office waiting to be built… and for time & energy to restack my rack first.
Houston
Sure, was planning on it.
Curious about a small detail - Will the backplane visual elements in Houston work with my mobo/hba (AsRock X570 Creator / LSI SAS 9300-16I?)
Houston Host OS
Are there major advantages to using Rocky8 over Ubuntu 20? I’d prefer to stick with where I have my limited Linux experience.
While I’d prefer Ubuntu 22 (just to use the standard image as my other devices & vm’s), I’m aware it isn’t supported yet
Happy to hear it.
Great. I wonder how to calculate the speed loss to overhead and what that % will be…
Pool 2 Config
Great. If 2 drives are overkill for 1x 10Gb perhaps I can utilize my nVME devices more efficiently/creatively while considering IOPS over throughput. What about Pool2 = a mirror pool of 2x 2tb nvme devices, and use the 512GB devices as SLOG/ZIL devices.
Use Case📝 PostgreSQL DB Share & AI Model Share: Would this 2x2tb mirror pool be performant enough to serve…
7GB file at 10Gb from Pool2_Share2_ML
…without impacting the performance of a database serving containers on my cluster that prefer…
synchronous writes to the DB on Pool2_Share1_PSQL (assuming DB traffic over dedicated NIC)
Sweet. A SLOG device for each spinning pool seems like a decent use for the 512GB drives.
I’m a bit out of my depth to even know how to measure exact requirements - or what they’d be measured in.
With 6x4TB HDDs (plus a less-trusted 7th), what differences between the following configs. I’m not familiar enough with ZFS to remember how the pool setups aggregate disk speeds…
Single pool rz1
Single pool rz2
2x rz1 vdevs of 3 HDDs
Edit: Kudos to @DigitalGarden for helping set me straight on this one.
I don’t think that is what the manual says at the support link you provided.
PCIE1 is an electrical X16 slot
PCIE4 is an electrical X8 slot
PCIE6 is an electrical X4 slot
Each of those slots will never use more lanes than that.
With a Ryzen 4600G (Renoir) APU, which I assume has the same behavior as Matisse, you can either have
PCIE1 = running at X16
PCIE4 = empty
PCIE6 = empty
PCIE1 = running at X8
PCIE4 = running at X8
PCIE6 = empty
PCIE1 = running at X8
PCIE4 = running at X8
PCIE6 = running at X4
What other documentation do you have besides the User Manual? I don’t see anything in that manual talking about disabling the M.2 slots based on the presence of PCIe cards, but it is a thing on some boards, so maybe I’m missing it.
I also don’t see anything in the manual talking about being able to bifurcate the X16 slot into X4/X4/X4/X4 for the NVMe card. But it is possible that is a feature in the BIOS not documented in the User Manual.
I’m not sure there is any consumer motherboard that is going to let you run both an X16 and X8 card and not knock the X16 down to X8. That CPU only has 20 PCI lanes. I think you need Epyc with enough PCI lanes to do that. If you want full speed out of the HBA, I think you’re going to be limited to two NVMe drives on a riser card, assuming the X570 Creator has bifurcation options supporting that.
@Specter, Without going into too much detail. I don’t believe the visual graphics will work for that HBA. i believe for the aliasing to work you would need to have either a full build or a 9305 HBA card for our aliasing tool to work correctly.
As for you use case this is where an architect would be more suited for answering these kinds of questions so its a bit out of my depth.
for random IO each vdev you have is 1 HDD or performance so the more vdevs you have the more IOPs you get.
For sequential you can benefit from having 1 vdev but using all the drives. for example, a single vdev of 4 drives in a raid z1 would have 3 drives or performance and 1 drive for parity
If you would like a more in-depth answer and scope you would need to contact info@45drives.com and get something scheduled with an architect to spec out your exact use case
X570 Creator certainly supports PCIE lane bifurcation…
But I’m fresh out of pcie lanes. looks I can get 2 nvme drives in the “top slot” on x8 split to 2 x4, my HBA in the next slot, and 1x nvme slot on the mobo.
So… I’m going build 2 pools with my current setup with the idea of migrating my pools to the upgraded platform… which is probably going to involve some SuperMicro board that’s compatible with Houston.
Oh well, should have asked first - now I get to enjoy these self imposed limitations.
You’ve answered all my questions with some great info. Many thanks - I have Houston up and running and I’m off to monkey around. I’m sure I’ll be back soon.
Backups]]
Desired Outcomes
Pool Setup - Thank you!
I’m a bit out of my depth to even know how to measure exact requirements - or what they’d be measured in.