Inserting x16 PCIe card disables second slot

Hardware
1

Has anyone added an x16 PCIe card to their HL15 and also used the slot below on the SuperMicro X11SPH-nCTF (prebuilt system)? I am seeing that when I use an x16 card (GPU) the slot below is disabled. This should not happen according to the manual, it should auto switch the x16 to x8 and the slot below should still work at x8.

Note: The X11SPH-NCTPF/-NCTF auto configures SLOT5 and SLOT6. If the user
populates SLOT5 and SLOT6, both slots will be configured as PCI-E 3.0 x8. If SLOT6
is populated with a PCI-E 3.0 x16 card with no population on SLOT5, SLOT6 will be
configured as PCI-E 3.0 x16. However, if SLOT6 is populated with a PCI-E 3.0 x16
card and SLOT5 is also populated, the PCI-E 3.0 x16 card will run on a PCI-E 3.0 x8
slot. This may decrease performance for SLOT6.

image
image1317×1108 288 KB

2

Please look at the PCIe bifurcation table. If the manual says that the lane would be shared, try changing the settings in the BIOS.

2 Likes
3

Thanks, I tinkered with the settings for hours last night. There is no mention of how things map in the manual:

There are 3 settings you can change in the BIOS (pg 87)

IOU0 (II0 PCIe Br1)
This item configures the PCI-E port Bifuraction setting for a PCI-E port specified by the
user. The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto.
IOU1 (II0 PCIe Br2)
This item configures the PCI-E port Bifuraction setting for a PCI-E port specified by the
user. The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto.
IOU2 (II0 PCIe Br3)
This item configures the PCI-E port Bifuraction setting for a PCI-E port specified by the
user. The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto.

I am unsure which settings maps to which slot. Each bridge is set to auto currently.

I will tinker some more and I am sure I did try this combination however SLOT5 was never trained when the SLOT6 x16 was populated.

4

Just circling back to say that no matter which combination of settings I use fro all 3 bridges, SLOT5 is never trained when SLOT6 is populated.

5

After hours upon hours of testing every combination, it seems that once the 16x is split between slot 5 and 6 (to x8x8) it cannot be split again to x4x4.

I also wanted to confirm that auto splitting works fine when using an x8 and x16 card however not when using an AOC-SLG3-2M2 in slot5 while slot6 is populated. This card works great in slot 3.

https://www.supermicro.com/en/products/accessories/addon/AOC-SLG3-2M2.php

7 Likes
6

Adding this visual in case you’re wondering what the heck I was trying to accomplish. I did try an x8 card (not video) in slot6 and same results. I’m convinced it’s not possible, would love to be wrong.

image
image1258×1257 38.5 KB

2 Likes
7

HI @technotim, I think you are correct. I don’t believe it can be split 2 times.

Have you tried putting an x8 card in slot 5 to see if that works? (a card that is not trying to split the connection again)

So have your GPU in slot 6 and then put a NIC or something else into slot 5 to see if both are picked up.

8

Thanks! Yeah, that does work if I use a native x8 card but I was hoping to take advantage of 2 of AOC-SLG3-2M2 cards :smile:. It seems like the only slot one of these will work in is SLOT3 since SLOT5/6 are shared.

1 Like
9

Sorry to resurrect a stale post, but I’m wondering if you were able to get it working with one NVMe in the x8 slot?

Right now I’m running two SLG3-2M2 cards, but the one in slot 2 only connects one SSD, since its a x4 in an x8 slot.

I’m curious if you have slot 6 running at x8, and slot 5 running at x8, if you can use one SSD in a carrier in slot 5, since it wont be bifurcated, it will just run at x4.

10

Just posted here how I use a 4 x NVME Card in Slot 6 and leave Slot 5 unpopulated.
Works great sofar, yet still loadtesting to do for Error checking.