HL15 Full Build - IPMI Not Working; Nics Not working

Networking
,
1

First HL15 full build came in and yesterday I had a chance to power it on; about 5 min in, I disabled ipv6 in the Supermicro IPMI and lost all connectivity to IPMI.

Packet capture shows it does respond to arp’s but not pings. Connected VGA, usb keyboard / mouse & booted into Rocky linux, set static IP’s on the 2 ports (per HL15 documentation) however, both ports are not reachable (can’t ping out from Rocky or ping from my desktop to either nic IP.) Setting the ports to dhcp, port 1 gets an IP (port 2 shows NO-Carrier) but issues persist. Obviously haven’t tried getting to Houston UI but I did the “systemctl start cockpit” & “systemctl enable cockpit.socket” but no change.

Multiple times I’ve changed bios (setup) IPMI to use both static and dhcp (it does get the correct IP based on my dhcp reservation) but still can’t access IPMI. Left light on IPMI nic is amber and right is yellow (blinking). I’ve tried connecting it to different switches, changed cables, and even directly attached a laptop in the same subnet to the IPMI port but still no luck. It’s like the lights are on but no one is home.

The only thing that gives some hope is in Rocky Linux running “ipmitool raw 0x3c 0x40” (resetting IPMI); after about 20 seconds I get between 11 and 15 ping replies from the IPMI IP and then it just stops (not enough time to access the gui).

Not sure if this is related but I’ve rebooted this about 50 times trying different things and notice that “sometimes” pressing F11 to access the boot menu actually takes me to the bios (setup). After exiting, it will go to one of two boot option menus (sometimes disk, sometimes Rocky Linux versions). I couldn’t get ipmicfg working in Rocky (using the Supermicro download). Also, Rocky isn’t keeping the root password change (goes back to the 45drives password after reboot).

In the Bios Event Logs, I see error code EFI 03051002 “DXE BS driver Unrecognized” – I haven’t installed any drives at this point so I’ll open it up and see if all cables are seated properly.

I’ve spent about 10 hours on this hitting brick walls; I’m a bit disappointed – took today off for 3-day weekend to provision this.

Do I have bad Nics? Bad board? Seems like something is taking control of the ports, blocking all traffic. I did disable firewalld in Rocky but no change. I also restored all defaults to the bios – no joy.

Any advise or suggestions would be greatly appreciated.

Bios V. 3.5
Build Date 06/01/2021
CPLD V. 02.B1.0B
Not sure what other info would be relevant

Thanks!

2

Since ipmitool appears to be working, maybe try sudo ipmitool lan print and sudo ipmitool lan6 print commands from rocky? Might provide some hints as to what’s going on.

3

Thank you for the assistance; I’ve never used ipmitool until today so I’m not really familiar with it. The only interesting thing that I see is “IPv6/IPv4 Addressing Enables: ipv6”; in my earlier t-shooting I did disable ipv6 in the bios but issues persist. Not sure if pasting the output is going to cause a mess (or if I need to put it in code block – not sure how to do that) but here it is:

[root@hl15 45drives]# ipmitool lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD 
Auth Type Enable        : Callback : MD2 MD5 PASSWORD 
                        : User     : MD2 MD5 PASSWORD 
                        : Operator : MD2 MD5 PASSWORD 
                        : Admin    : MD2 MD5 PASSWORD 
                        : OEM      : MD2 MD5 PASSWORD 
IP Address Source       : DHCP Address
IP Address              : 172.16.3.17
Subnet Mask             : 255.255.255.0
MAC Address             : xx:xx:xx:xx:xx:xx
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : 172.16.3.2
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max   : XaaaXXaaaXXaaXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : 3
Invalid password disable: yes
Attempt Count Reset Int.: 300
User Lockout Interval   : 300
[root@hl15 45drives]# ipmitool lan6 print
Getting parameter(s)...
IPv6/IPv4 Support:
    IPv6 only: no
    IPv4 and IPv6: yes
    IPv6 Destination Addresses for LAN alerting: yes
IPv6/IPv4 Addressing Enables: ipv6
IPv6 Header Traffic Class: 0
IPv6 Header Static Hop Limit: 64
IPv6 Status:
    Static address max:  5
    Dynamic address max: 4
    DHCPv6 support:      yes
    SLAAC support:       yes
IPv6 Static Address 0:
    Enabled:        no
    Address:        ::/0
    Status:         disabled
IPv6 Static Address 1:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 Static Address 2:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 Static Address 3:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 Static Address 4:
    Enabled:        no
    Address:        ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/0
    Status:         disabled
IPv6 DHCPv6 Static DUID Storage Length: 1
IPv6 DHCPv6 Static DUID 0:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 1:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 2:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 3:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 DHCPv6 Static DUID 4:
    Length:   14
    Type:     unknown
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IPv6 Dynamic Address 0:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Dynamic Address 1:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Dynamic Address 2:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Dynamic Address 3:
    Source/Type:    static
    Address:        ::/0
    Status:         disabled
IPv6 Router Address Configuration Control:
    Enable static router address:  yes
    Enable dynamic router address: no
IPv6 Static Router 1:
    Address: ::
    MAC:     00:00:00:00:00:00
    Prefix:  ::/255
IPv6 Static Router 2:
    Address: ::
    MAC:     00:00:00:00:00:00
    Prefix:  ::/255
IPv6 Number of Dynamic Router Info Sets: 0
IPv6 ND/SLAAC Timing Configuration Support: global
IPv6 ND/SLAAC Timing Configuration 0:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 1:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 2:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 3:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0
IPv6 ND/SLAAC Timing Configuration 4:
    MAX_RTR_SOLICITATION_DELAY: 1
    RTR_SOLICITATION_INTERVAL:  4
    MAX_RTR_SOLICITATIONS:      3
    DupAddrDetectTransmits:     1
    MAX_MULTICAST_SOLICIT:      3
    MAX_UNICAST_SOLICIT:        3
    MAX_ANYCAST_DELAY_TIME:     1
    MAX_NEIGHBOR_ADVERTISEMENT: 3
    REACHABLE_TIME:             30
    RETRANS_TIMER:              1
    DELAY_FIRST_PROBE_TIME:     5
    MAX_RANDOM_FACTOR:          0
    MIN_RANDOM_FACTOR:          0

And when IPMI stopped working, the only change I made on the HL15 was disabling ipv6 in IPMI.

Thanks!

4

Have you tried resetting the IPMI and/or BIOS to factory default?

https://serverhub.com/kb/how-to-reset-ipmi-bmc-to-factory-default-using-ipmicfg/

Seems like it’s not a corrupt BIOS if it was working, but you could also try re-flashing the BIOS.

5

I think the IPv6/IPv4 Addressing Enables: ipv6 is the problem. I also disabled IPV6 on my HL15 full build so it would only use IPV4. My output included below.

I’m also not too familar with ipmitool but I think you could try this: sudo ipmitool lan6 set 1 enables ipv4

Truenas-HL15:~$ sudo ipmitool lan6 print
IANA PEN registry open failed: No such file or directory
Getting parameter(s)...
IPv6/IPv4 Support:
    IPv6 only: no
    IPv4 and IPv6: yes
    IPv6 Destination Addresses for LAN alerting: yes
IPv6/IPv4 Addressing Enables: ipv4
IPv6 Header Traffic Class: 0
IPv6 Header Static Hop Limit: 64
6

One more question, is this correct for your default gateway? Usually it’s the first IP address in the subnet which for /24 is X.X.X.1. It’s entirely possibly that is correct but just thought I’d check as X.X.X.2 is a little unusual.

7

Thank you for the help! I was able to resolve the IPMI issue with your help. The two SFP ports are still not pingable but I’ll open another topic if I can’t figure that out. Like @rymandle05 pointed out,

IPv6/IPv4 Addressing Enables: ipv6

Looked suspect.

sudo ipmitool lan6 set 1 enables ipv4

Led me down a different path; when I ran this command logged in as 45Drives user
and sudo, I was getting:

Failed to set Set In Progress: Set already in progress

I then logged as root user, reran

ipmitool raw 0x3c 0x40

And voilà, pings to IPMI persisted and I was able to regain access.

When this issue started, I must have disabled IPv6 in Bios under IPMI first then went into the GUI > Configuration > Network > “IP Protocol Status”, and changed this drop-down to ipv6 since I noticed the IPv6 Settings were greyed out and IPv6 Setting didn’t have the radio button set to “Disabled”… and that’s when I lost connectivity. Suspect this forced IPMI over to IPv6 so… layer 8 issue (not understanding Supermicro IPMI). I am very familiar with HP iLO but not SM’s IPMI. The “IP Protocol Status” seems a little misleading… just a little.

I’m not going to try to reproduce this so if anyone else wants too, please post your findings.

@DigitalGarden, thank you for the link – that would have been my next step.
@rymandle05, thank you for the suggestions and assistance, it got my brain moving again. And yeah, dfgw is .2… that would have been an epic Layer 8 (which I’m known for).

Thanks!

1 Like
8

Awesome! Glad you are back up and running again and the suggestions were helpful. :raised_hands:

1 Like