HL15 Full Build - IPMI Not Working; Nics Not working

ipmi_denied · May 17, 2024, 10:25pm

First HL15 full build came in and yesterday I had a chance to power it on; about 5 min in, I disabled ipv6 in the Supermicro IPMI and lost all connectivity to IPMI.

Packet capture shows it does respond to arp’s but not pings. Connected VGA, usb keyboard / mouse & booted into Rocky linux, set static IP’s on the 2 ports (per HL15 documentation) however, both ports are not reachable (can’t ping out from Rocky or ping from my desktop to either nic IP.) Setting the ports to dhcp, port 1 gets an IP (port 2 shows NO-Carrier) but issues persist. Obviously haven’t tried getting to Houston UI but I did the “systemctl start cockpit” & “systemctl enable cockpit.socket” but no change.

Multiple times I’ve changed bios (setup) IPMI to use both static and dhcp (it does get the correct IP based on my dhcp reservation) but still can’t access IPMI. Left light on IPMI nic is amber and right is yellow (blinking). I’ve tried connecting it to different switches, changed cables, and even directly attached a laptop in the same subnet to the IPMI port but still no luck. It’s like the lights are on but no one is home.

The only thing that gives some hope is in Rocky Linux running “ipmitool raw 0x3c 0x40” (resetting IPMI); after about 20 seconds I get between 11 and 15 ping replies from the IPMI IP and then it just stops (not enough time to access the gui).

Not sure if this is related but I’ve rebooted this about 50 times trying different things and notice that “sometimes” pressing F11 to access the boot menu actually takes me to the bios (setup). After exiting, it will go to one of two boot option menus (sometimes disk, sometimes Rocky Linux versions). I couldn’t get ipmicfg working in Rocky (using the Supermicro download). Also, Rocky isn’t keeping the root password change (goes back to the 45drives password after reboot).

In the Bios Event Logs, I see error code EFI 03051002 “DXE BS driver Unrecognized” – I haven’t installed any drives at this point so I’ll open it up and see if all cables are seated properly.

I’ve spent about 10 hours on this hitting brick walls; I’m a bit disappointed – took today off for 3-day weekend to provision this.

Do I have bad Nics? Bad board? Seems like something is taking control of the ports, blocking all traffic. I did disable firewalld in Rocky but no change. I also restored all defaults to the bios – no joy.

Any advise or suggestions would be greatly appreciated.

Bios V. 3.5
Build Date 06/01/2021
CPLD V. 02.B1.0B
Not sure what other info would be relevant

Thanks!

rymandle05 · May 17, 2024, 11:49pm

Since ipmitool appears to be working, maybe try sudo ipmitool lan print and sudo ipmitool lan6 print commands from rocky? Might provide some hints as to what’s going on.

ipmi_denied · May 18, 2024, 12:21am

Thank you for the assistance; I’ve never used ipmitool until today so I’m not really familiar with it. The only interesting thing that I see is “IPv6/IPv4 Addressing Enables: ipv6”; in my earlier t-shooting I did disable ipv6 in the bios but issues persist. Not sure if pasting the output is going to cause a mess (or if I need to put it in code block – not sure how to do that) but here it is:

[root@hl15 45drives]# ipmitool lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD 
Auth Type Enable        : Callback : MD2 MD5 PASSWORD 
                        : User     : MD2 MD5 PASSWORD 
                        : Operator : MD2 MD5 PASSWORD 
                        : Admin    : MD2 MD5 PASSWORD 
                        : OEM      : MD2 MD5 PASSWORD 
IP Address Source       : DHCP Address
IP Address              : 172.16.3.17
Subnet Mask             : 255.255.255.0
MAC Address             : xx:xx:xx:xx:xx:xx
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : 172.16.3.2
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max   : XaaaXXaaaXXaaXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : 3
Invalid password disable: yes
Attempt Count Reset Int.: 300
User Lockout Interval   : 300
[root@hl15 45drives]# ipmitool lan6 print
Getting parameter(s)...
IPv6/IPv4 Support:
    IPv6 only: no
    IPv4 and IPv6: yes
    IPv6 Destination Addresses for LAN alerting: yes
IPv6/IPv4 Addressing Enables: ipv6
IPv6 Header Traffic Class: 0
IPv6 Header Static Hop Limit: 64
IPv6 Status:
    Static address max:  5
    Dynamic address max: 4
    DHCPv6 support:      yes
    SLAAC support:       yes
IPv6 Static Address 0:
    Enabled:        no
    Address:        ::/0
    Status:         disabled
IPv6 Static Address 1:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 Static Address 2:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 Static Address 3:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 Static Address 4:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 DHCPv6 Static DUID Storage Length: 1
IPv6 DHCPv6 Static DUID 0:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 1:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 2:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 3:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 4:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 Dynamic Address 0:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Dynamic Address 1:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Dynamic Address 2:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Dynamic Address 3:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Router Address Configuration Control:
    Enable static router address:  yes
    Enable dynamic router address: no
IPv6 Static Router 1:
    Address: ::
    MAC:     00:00:00:00:00:00
    Prefix:  ::/255
IPv6 Static Router 2:
    Address: ::
    MAC:     00:00:00:00:00:00
    Prefix:  ::/255
IPv6 Number of Dynamic Router Info Sets: 0
IPv6 ND/SLAAC Timing Configuration Support: global
IPv6 ND/SLAAC Timing Configuration 0:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 1:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 2:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 3:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 4:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0

And when IPMI stopped working, the only change I made on the HL15 was disabling ipv6 in IPMI.

Thanks!

DigitalGarden · May 18, 2024, 12:36am

Have you tried resetting the IPMI and/or BIOS to factory default?

https://serverhub.com/kb/how-to-reset-ipmi-bmc-to-factory-default-using-ipmicfg/

Seems like it’s not a corrupt BIOS if it was working, but you could also try re-flashing the BIOS.

rymandle05 · May 18, 2024, 12:39am

I think the IPv6/IPv4 Addressing Enables: ipv6 is the problem. I also disabled IPV6 on my HL15 full build so it would only use IPV4. My output included below.

I’m also not too familar with ipmitool but I think you could try this: sudo ipmitool lan6 set 1 enables ipv4

Truenas-HL15:~$ sudo ipmitool lan6 print
IANA PEN registry open failed: No such file or directory
Getting parameter(s)...
IPv6/IPv4 Support:
    IPv6 only: no
    IPv4 and IPv6: yes
    IPv6 Destination Addresses for LAN alerting: yes
IPv6/IPv4 Addressing Enables: ipv4
IPv6 Header Traffic Class: 0
IPv6 Header Static Hop Limit: 64

rymandle05 · May 18, 2024, 12:43am

One more question, is this correct for your default gateway? Usually it’s the first IP address in the subnet which for /24 is X.X.X.1. It’s entirely possibly that is correct but just thought I’d check as X.X.X.2 is a little unusual.

ipmi_denied · May 18, 2024, 4:31pm

Thank you for the help! I was able to resolve the IPMI issue with your help. The two SFP ports are still not pingable but I’ll open another topic if I can’t figure that out. Like @rymandle05 pointed out,

IPv6/IPv4 Addressing Enables: ipv6

Looked suspect.

sudo ipmitool lan6 set 1 enables ipv4

Led me down a different path; when I ran this command logged in as 45Drives user
and sudo, I was getting:

Failed to set Set In Progress: Set already in progress

I then logged as root user, reran

ipmitool raw 0x3c 0x40

And voilà, pings to IPMI persisted and I was able to regain access.

When this issue started, I must have disabled IPv6 in Bios under IPMI first then went into the GUI > Configuration > Network > “IP Protocol Status”, and changed this drop-down to ipv6 since I noticed the IPv6 Settings were greyed out and IPv6 Setting didn’t have the radio button set to “Disabled”… and that’s when I lost connectivity. Suspect this forced IPMI over to IPv6 so… layer 8 issue (not understanding Supermicro IPMI). I am very familiar with HP iLO but not SM’s IPMI. The “IP Protocol Status” seems a little misleading… just a little.

I’m not going to try to reproduce this so if anyone else wants too, please post your findings.

@DigitalGarden, thank you for the link – that would have been my next step.
@rymandle05, thank you for the suggestions and assistance, it got my brain moving again. And yeah, dfgw is .2… that would have been an epic Layer 8 (which I’m known for).

Thanks!

rymandle05 · May 18, 2024, 4:48pm

Awesome! Glad you are back up and running again and the suggestions were helpful.

Nate_Diggity_Dog · February 5, 2025, 8:01am

Did you ever resolve the issue where you couldn’t see the two 10G SPF+ ports. I’ve had my new HL15 for three weeks now and have the same issue. I even replaced the motherboard with a new one (Supermicro X11SPH-NCTPF) and I am still having an issue. I sould have ordered the board with the RJ45 ports. My excitement has turned into me regretting my purchase. Hopefully someone here has some idea how to resolve this. Thanks.

DigitalGarden · February 5, 2025, 9:29am

There’s another thread here where the resolution turned out to be bad hardware (bad NICs). Networking down / SFP+ DAC Cable - #24 by matthammond .

But you say you’ve tried a second motherboard. Did that come from 45HL?

What cabling are you using (DAC, optical, transceiver) (SFP+ on both ends or mix SFP/SFP+ port or transceivers), what are you trying to connect to (switch make/model? another PC direct), and what specific errors or symptoms do you have (NIC doesn’t show up, NIC shows up but no IP)? You’re using the Rocky Linux/Houston OS? What does ethtool output look like?

Nate_Diggity_Dog · February 5, 2025, 4:50pm

Thanks for the reply. Yes, I read that post and I did replace the board which came from 45Drives. Hard to believe that I would get two bad boards. I feel like I’m either missing something or my other though would be a driver issue.

This was a full build from 45Drives however I added my own:
**CPU ( Intel Xeon Silver 4214 12 Cores 2.2GHz 16.5MB 85W LGA 3647 CPU SRFB9)

**8 sticks RAM ( Samsung 64GB DDR4 2400MHz PC4-19200 ECC LRDIMM 4Rx4 Quad Rank 1.2V Load Reduced DIMM 288-Pin Server RAM Memory M386A8K40BM1-CRC).

I purchased these two SPF+ adapters which seem to work as I have connected an ethernet cable (one at a time) to both the SFP+ ports including the IPMI port and I can access the IPMI from all three on my office PC.

**Amazon.com: 10Gtek 1.25/2.5/5/10G-T SFP+ to RJ45 CAT.6a Copper Transceiver, Auto-Negotiation SFP+ Ethernet Module, up to 30-Meter, for Cisco SFP-10G-T-X, Ubiquiti UniFi UF-RJ45-10G, Netgear AXM765 and More : Electronics

**Amazon.com: 10Gtek 1.25G SFP-T, 1000BASE-T Copper SFP, SFP to RJ45 SFP Module Transceiver for Cisco SFP-GE-T, Meraki, Fortinet, Ubiquiti UniFi UF-RJ45-1G, D-Link, Supermicro, Netgear, TP-Link and More : Electronics

When I log into the MB via IPMI, under System, here is what I see:

When compaired to the manual from 45Drives, this is what I think I should see:

If you notice, the example from the manual shows System LAN1 and LAN2 MAC Address.

Here is what I see in terminal “ip -c a”. I believe I should see the “eno1” and “eno2” for the two SFP+ ports.

In Network Manager in terminal, I see eno1 however it has no MAC Address and when I go to ACTIVATE the connection, it is not there.

These are all screen captures from my desktop via the IPMI interface via one of the SPF+ ports so I know at least something is working.

I have been banging my head on the desk. I have been on the phone with tech support at 45Drives and they though it was bad hardware, New board is installed and same thing. Any help would be appreaciated.

Thank in advance.

DigitalGarden · February 5, 2025, 11:47pm

I don’t have a full build, but my expectations would parallel yours. If the NICs aren’t present in the BIOS or IPMI there is a fundamental problem with the board and not your transceivers or network setup. Based on posts here, I think there may be a more recent version of the BIOS, and perhaps the firmware for the Intel X722 (?) NIC, but I wouldn’t expect that to fix anything.

Hopefully someone else has better ideas, but I think it’s pretty clear what the BIOS/IPMI should or should not look like bare without any transceivers installed for a support ticket. The MAC address should be part of the NIC so should be detectable by the IPMI.

If the 10GBaseT version is acceptable to you, maybe see if they can send you the NCTF as a replacement instead. Alternately, you could offer them the option to send you one of the 10G PCIe NICs in the store if shipping mobos back and forth is becoming a hassle, and you’re not ready to just return the whole unit;
https://store.45homelab.com/products/16
https://store.45homelab.com/products/17
A Network Attached Storage device isn’t much use if you can’t Attach it to the Network.

As an aside, here’s another internet post about an X11SPH-NCTPF that ends in a mobo with faulty NIC(s); Second SFP+ never reports LINK UP - TrueNAS General - TrueNAS Community Forums
Seems like QC may be slipping over at SuperMicro.

rymandle05 · February 6, 2025, 5:03am

I know it’s tough to fathom but the bad luck lottery does strike like this from time to time. @Nate_Diggity_Dog what do you see in the BIOS menus (delete key at startup)? There are options to disable the onboard LAN as well as loading option ROM’s for the NIC’s. I’m guessing you’ve checked all this but I didn’t see any BIOS screenshots in your post above so thought I’d make sure to ask.

There are some BIOS and firmware updates available on SuperMicro’s website. It’s a hail mary but I don’t think you’re out anything at this point by trying.

Do you still have the Intel Bronze CPU that came with the full build? Have you tried swapping back to that? This is a long shot but I’m wondering if it’s actually a problem with CPU? I’m pretty sure the NIC’s go through the Intel Chipset so you’d think you’d have more issues if that was the problem.

Nate_Diggity_Dog · February 6, 2025, 7:08pm

I suppose lightning can strike the same place twice. I just wish it would have happened with something like a winning lottery ticket. I’ll post some bios pics below. As for a BIOS update, I have thought of this but afraid to screw something up even worse. That said, I’m out of options so…

OLD BIOS Version: 3.5

NEW BIOS Version: 4.4
Notice, there still isn’t any MAC’s showing for onboard NICs.

Here is a few BIOS screenshots:

*Advanced
**PCIe/PCI/PnP Configuration

*IPMI
**BMC Network Configuration

jrhenderson · February 6, 2025, 9:15pm

My 2 cents. Return it and ask them to test the replacement before shipping.

ipmi_denied · February 6, 2025, 9:29pm

My issue was with vlan tagging on the uplinked 10gb switch; both are working now with DAC in one and fiber in the other.

I installed Truenas scale and both nics show as eno1 & eno2; SM ipmi shows them as
System LAN1 MAC Address
System LAN2 MAC Address:

Which is missing from your screen shot… also, Firmware Revision is older on yours. Here is mine:

Nate_Diggity_Dog · February 6, 2025, 11:00pm

Not sure why it didn’t show previously, but the update is showing now. I’m also showing an extra IPv6 address now as well. Still no LAN1 or LAN2 MAC Address.

Nate_Diggity_Dog · February 6, 2025, 11:08pm

@jrhenderson, Thank you for your input. 45Drives is working on the issue to help resolve this.

rymandle05 · February 7, 2025, 2:30am

I see your IPMI network is setup as “Share LAN”. Have you tried setting this up in dedicated mode? I have another system where TrueNAS would lose its IP after a few days. I had to change the IPMI network to only use the dedicated port. I haven’t had the issue since. Not quite the same as your problem here but does show that the IPMI settings can impact the OS.

nickw · February 7, 2025, 10:33am

Hi Nate_Diggity_Dog, just so I understand correctly, you are able to access the HL15 via both SFP+ ports (using your rj45 transceivers) in addition to the management port?

If you plug all 3 of the connections into your switch, does your router assign each connection an IP address? Can you see any more info (such as mac address) on your router for each connection?