RESOLVED:ZFS Write Errors with HL15 Full Build and SAS Drives

Using LSIUTIL to set SAS3008 Controller to SAS-2 6Gbps Max Speed

  1. Download a copy of lsiutil. I used lsiutil version 1.7 archived on github thanks to Thomas Lovell. I’m running Ubuntu 22.04 so wget to download and chmod to make it executable.
wget https://raw.githubusercontent.com/thomaslovell/LSIUtil/master/Binaries/LSIutil_1.70_release_binaries/linux/lsiutil.x86_64 && chmod +x lsiutil.x86_64
  1. Execute lsiutil with elevated privileges or as root.
sudo ./lsiutil.x86_64
  1. Now in lsiutil, select the controller by entering its number. There’s probably only one so enter 1.
Step 3 Results
LSI Logic MPT Configuration Utility, Version 1.71, Sep 18, 2013

1 MPT Port found

     Port Name         Chip Vendor/Type/Rev    MPT Rev  Firmware Rev  IOC
 1.  ioc0              LSI Logic SAS3008 C0      205      10000a00     0

Select a device:  [1-1 or 0 to quit] 1
  1. Verify your the SAS controller settings using option 68. You should see SAS3008's links are 12.0 G, ....
Step 4 Results
 Main menu, select an option:  [1-99 or e/p/w or 0 to quit] 68 

Current Port State
------------------
SAS3008's links are 12.0 G, 12.0 G, 12.0 G, 12.0 G, 12.0 G, 12.0 G, 12.0 G, down

Software Version Information
----------------------------
Current active firmware version is 10000a00 (16.00.10)
Firmware image's version is MPTFW-16.00.10.00-IT
  LSI Logic
  Not Packaged Yet
x86 BIOS image's version is MPT3BIOS-8.37.00.00 (2018.04.04)
EFI BIOS image's version is 18.00.00.00

Firmware Settings
-----------------
SAS WWID:                       5003048xxxxxxxxx
Multi-pathing:                  Disabled
SATA Native Command Queuing:    Enabled
SATA Write Caching:             Enabled
SATA Maximum Queue Depth:       128
SAS Max Queue Depth, Narrow:    256
SAS Max Queue Depth, Wide:      256
Device Missing Report Delay:    0 seconds
Device Missing I/O Delay:       0 seconds
Phy Parameters for Phynum:      0    1    2    3    4    5    6    7    
  Link Enabled:                 Yes  Yes  Yes  Yes  Yes  Yes  Yes  Yes  
  Link Min Rate:                3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  
  Link Max Rate:                12.0 12.0 12.0 12.0 12.0 12.0 12.0 12.0 
  SSP Initiator Enabled:        Yes  Yes  Yes  Yes  Yes  Yes  Yes  Yes  
  SSP Target Enabled:           No   No   No   No   No   No   No   No   
  Port Configuration:           Auto Auto Auto Auto Auto Auto Auto Auto 
Interrupt Coalescing:           Enabled, timeout is 10 us, depth is 4
  1. OPTIONAL: Reset to Defaults for good measure by using option 61. You can also check if anything is other than defaults using 60.
Step 5 Results
Main menu, select an option:  [1-99 or e/p/w or 0 to quit] 61

Restoring default persistent settings...

IO Unit Page 1 is persistent

IO Unit Page 8 is persistent

IOC Page 1 is persistent
Defaults restored

IOC Page 8 is persistent

BIOS Page 1 is persistent

BIOS Page 3 is persistent
Defaults restored

BIOS Page 4 is persistent

SAS IO Unit Page 1 is persistent

SAS IO Unit Page 4 is persistent

SAS IO Unit Page 5 is persistent
Defaults restored

SAS IO Unit Page 7 is persistent
Defaults restored

SAS Phy Page 3 is persistent

Type 17h Page 0 is persistent

Type 19h Page 1 is persistent
  1. Change maximum sas link speed rate to 6Gbps by using option 13 on the main menu. Press enter to accept defaults for the first five prompts.
Step 6 Results
Main menu, select an option:  [1-99 or e/p/w or 0 to quit] 13
SATA Maximum Queue Depth:  [0 to 255, default is 128] 
SAS Max Queue Depth, Narrow:  [0 to 65535, default is 256] 
SAS Max Queue Depth, Wide:  [0 to 65535, default is 256] 
Device Missing Report Delay:  [0 to 2047, default is 0] 
Device Missing I/O Delay:  [0 to 255, default is 0] 

PhyNum  Link      MinRate  MaxRate  Initiator  Target    Port
   0    Enabled     3.0      12.0    Enabled    Disabled  Auto
   1    Enabled     3.0      12.0    Enabled    Disabled  Auto
   2    Enabled     3.0      12.0    Enabled    Disabled  Auto
   3    Enabled     3.0      12.0    Enabled    Disabled  Auto
   4    Enabled     3.0      12.0    Enabled    Disabled  Auto
   5    Enabled     3.0      12.0    Enabled    Disabled  Auto
   6    Enabled     3.0      12.0    Enabled    Disabled  Auto
   7    Enabled     3.0      12.0    Enabled    Disabled  Auto
  1. Enter 8 when prompted for Select a Phy: [0-7, 8=AllPhys, RETURN to quit] to change settings on all the SAS links.
Step 7 Results
PhyNum  Link      MinRate  MaxRate  Initiator  Target    Port
   0    Enabled     3.0      12.0    Enabled    Disabled  Auto
   1    Enabled     3.0      12.0    Enabled    Disabled  Auto
   2    Enabled     3.0      12.0    Enabled    Disabled  Auto
   3    Enabled     3.0      12.0    Enabled    Disabled  Auto
   4    Enabled     3.0      12.0    Enabled    Disabled  Auto
   5    Enabled     3.0      12.0    Enabled    Disabled  Auto
   6    Enabled     3.0      12.0    Enabled    Disabled  Auto
   7    Enabled     3.0      12.0    Enabled    Disabled  Auto

Select a Phy:  [0-7, 8=AllPhys, RETURN to quit] 8
  1. Press enter for the next two prompts. When asked MaxRate: [0=1.5 Gbps, 1=3.0 Gbps, 2=6.0 Gbps, 3=12.0 Gpbs, or RETURN to not change] enter a value of 2 to set the max speed to 6.0Gbps.
Step 8 Results
Link:  [0=Disabled, 1=Enabled, or RETURN to not change] 
MinRate:  [0=1.5 Gbps, 1=3.0 Gbps, 2=6.0 Gbps, 3=12.0 Gbps, or RETURN to not change]
MaxRate:  [0=1.5 Gbps, 1=3.0 Gbps, 2=6.0 Gbps, 3=12.0 Gpbs, or RETURN to not change] 2
Initiator:  [0=Disabled, 1=Enabled, or RETURN to not change]  
Target:  [0=Disabled, 1=Enabled, or RETURN to not change] 
Port configuration:  [1=Auto, 2=Narrow, 3=Wide, or RETURN to not change] 

PhyNum  Link      MinRate  MaxRate  Initiator  Target    Port
   0    Enabled     3.0      6.0    Enabled    Disabled  Auto
   1    Enabled     3.0      6.0    Enabled    Disabled  Auto
   2    Enabled     3.0      6.0    Enabled    Disabled  Auto
   3    Enabled     3.0      6.0    Enabled    Disabled  Auto
   4    Enabled     3.0      6.0    Enabled    Disabled  Auto
   5    Enabled     3.0      6.0    Enabled    Disabled  Auto
   6    Enabled     3.0      6.0    Enabled    Disabled  Auto
   7    Enabled     3.0      6.0    Enabled    Disabled  Auto

Select a Phy:  [0-7, 8=AllPhys, RETURN to quit] 
  1. Press enter to return to the main menu.

  2. OPTIONAL: Try option 99 to reset the links to pick up the changes. NOTE: This will cause drives to disconnect momentarily.

  3. Enter 0 and 0 again at the next prompt to exit lsiutil.

  4. Poweroff or reboot the computer to pick up changes (assuming step 10 wasn’t executed or didn’t work).

  5. Use sudo smartctl -x /dev/sgX to see the drive link speed reported as 6 Gbps on the Protocol Specific port log page for SAS SSP section.

Step 13 Results
Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 0
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c50084dfd3b1
    attached SAS address = 0x500304801d01d40e
    attached phy identifier = 2
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 2
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 2
     Phy reset problem count: 0
relative target port id = 2
  generation code = 0
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c50084dfd3b2
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

Check out this LSI User Guide for more information:

3 Likes