Edits made.
Some corrections were done to the post, primarily a miscalculation on my part for SAS lanes, a single SFF-8643 port has 4 lanes, it’s NOT a single 6Gbps/12Gbps connection as originally stated.
Added bits and bytes table for easy reference when sourcing your HBA or SAS expander.
For a real life (theoretical bandwidth) calculation example:
I’ll be using ASUS Pro WS W680-ACE motherboard.
It has 2 x PCIe 5.0 x16 (x16/x0 or x8/x8) giving me a 64GB/s connection straight to the CPU.
It has 8 x DMI 4.0 giving me a 16GB/s throughput through the W680 chipset.
It has 2 x PCIe 3.0 x16 (x4 mode) giving me a 4GB/s connection on each trough the W680 chipset.
I’ll be using a 9300-16i which has a PCIe 3.0 x8 in a PCIe 3.0 x4 slot giving me a 4GB/s connection to the backplane trough the W680 chipset.
I’ll be using a 9300-16i which has a PCIe 3.0 x8 in a PCIe 5.0 x8 slot giving me a 8GB/s connection to the SSD cache (x8 870 EVOs) straight to the CPU.
4GB/s ÷ 15 drives ≈ 250MB/s per drive
8GB/s ÷ 8 drives ≈ 1GB/s per drive
If in the future I were to install faster drives I would need to move my backplane HBA to another slot like the PCIe 5.0 x8 to achieve PCIe 3.0 x8 speeds of 8GB/s or ≈500MB/s per drive. Even faster drives would need a different HBA altogether to achieve their max speeds.
Being as the backplane is 15 x SAS3 ports for 1.5GB/s each, we would need a ~24GB/s HBA with a PCIe 4.0 x16 or PCIe 5.0 x8 in a matching PCIe slot to completely saturate each drive.
After a lot of research I tried to make the basic concepts of the math involved easy to understand. Since there are so many possible configurations you need to understand your specific case and use the math in your own formula to calculate your own numbers.
Feel free to mention your own hardware and I’ll do my best to run the numbers for you. (hopefully after a couple of examples it will become even easier to understand).