You could try updating the motherboard BIOS if you hadn’t tried that already, but my hopes would be low.
Also, QNAP makes a similar card. It sounds like you can’t return the Synology card, or don’t want to sell it used, but a similar QNAP card (2.5G instead of 10G) did seem to work with the HL8:
B&H is the only retailer for The QNAP network/MVMe expansion cards I’ve seen that doesn’t price gouge. Here’s a link to the 10G version;
https://www.bhphotovideo.com/c/product/1688430-REG/qnap_qm2_2p10g1tb_qm2_series_2_x.html