So I have spent the better part of 2 weeks (in my free time) trying to sort out what exactly is going on with this thing. Let me air out this story for some context.
I had purchased this kit off of eBay almost as soon as I had seen the posting. It was a solid price ($551USD) and already had the upgraded Sunrich S-331 backplane so I was pretty stoked that there really wouldn’t be anything for me to do beside have fun…
Specs -
1 x SuperMicro X9SRH-7TF
1 x Intel Xeon E5-1620V2
4 x Hynix HMT31GR7CFR4C-PB 8GB DDR3
3 x PI49230-2X2B
2 x PSM-5760V
Fast forward to today and its honestly been a huge headache. Initially after the first boot, I was having issues with the board either not detecting my keyboard or locking up entirely. So I had spent some time reading the manual and had sorted out howe to reset the BIO to factory defaults.
So I did what anyone else does and I pulled the motherboard and found that there were a 2 standoffs that didn’t correlate to any holes so I had assumed the board maybe had died. Reached out to the seller to request a new board ands the sent one over. Out of curiosity I put the board on a test bench to tinker around and behold it boots? I though ok, maybe that wasn’t the issue, after a quick bios check, and adding one PCIe card it would bios lock, so I had assumed ok maybe it is dead.
New board arrived, and I had assembled it in the chassis(after removing the problematic standoffs) I verified a few times that everything was in order before booting and on first boot it boot locks…again… seemingly on bios code B4. I have since spent 4 hours along with a few friends troubleshooting various things and still no luck. Ubuntu doesn’t recognize any drives (attached to the board directly or to the backplanes) and it still will lock up during post for some unknown reason. The board is on BIOS version 3.0 and the new version don’t have any relevant updates so here I am, asking if anyone else has had this issue.
Not sure what else to try, the backplanes don’t work with my desktop (No Post) and directly adding the drive bypassing the backplanes also yields no success.
So, you have two motherboards, or did you return the first one to get the replacement?
For #1 sort-of boots, but locks up on boot when you have a particular PCIe card inserted? You BIOS reset this board. Which PCIe card locks it up? If one of the SATA controllers, does having drives attached or not make a difference?
For #2 you are saying the board locks up with BIOS code B4 on POST, but also that you can get into Ubuntu?
Which board are we debugging?
Were you using the same CPU in both boards? Maybe a CPU problem? Did you go through combinations of the RAM sticks and slots? Do you happen to have spare RAM around? Maybe a RAM issue?
From the info you provided it seems like maybe an issue with one of the PCIe cards or slots, but it’s a little bit unclear. Try to build the simplest system and then add in items until it fails.
I see beep codes mentioned in the manual for the X9SRH-7TF but not 2 character POST codes. Is the POST code coming from the mobo or a PCIe card? Where are you seeing that (monitor or 7-segment LED)? Do you have a speaker (or maybe get one) to see if there is a beep code when it locks up on boot?
Is the keyboard/mouse issue resolved? If not, what type of connection is it using (wired USB, USB wireless dongle, PS/2)? If USB, be sure it is plugged into a USB2 port and preferably wired. You don’t mention any other USB devices, but if you have any other thumb drives or whatever attached, remove those.
Ok so let me try to clarify it a bit. I have both boards, both came with their own Ram and CPUs. I have tried various combinations of each on both boards and just last night I made some progress and was able to get windows to boot on with a PCI card in the bottom slot (Slot 3 as labeled on the board diagram) without issue. If I move the PCI card to the second slot (Slot 5), it pasts bios post, but doesn’t display any output using the onboard graphics and will eventually lead to a DRIVER_POWER_STATE FAILURE BSOD in windows once, but now has yet to repeat itself which is weird. Finally when trying it in the last slot (Slot 6), it also does the same as slot 5, it BSODs on the first boot and then is fine every subsequent boot after.
The bios hang seems to happen if there are any USB devices plugged into the board, which happens on both with either CPU. The K&M are known good and don’t cause issues on any of the other servers or devices that I have and the PCIe card was pulled from one of my servers in working condition and when put back and tested (R510) it works without issue. For reference its an LSI 9211-8i in IT Mode.
The B4 error code is coming from the Motherboard I suppose as it shows in the bottom right hand corner during post.(Without cards) As for beep codes, this board hasn’t made a sound which I also thought was odd as it has a speaker.
I have yet to test with the backplanes, I seem to be making some progress. I don’t have the Molex cables for this PSU so planning on picking one up for BB on the way home from work. Ill post another update tonight letting you know how that goes, hopefully that all makes sense?
How are you connecting the OS boot drive? Via the HBA or a mobo SATA port? If the mobo, which port … SATA 0/1 (white)?
It seems like the DRIVER_POWER_STATE FAILURE could be firmware or driver related, or Windows power down options. Where did the Windows install come from; was it done on that motherboard, or was it pulled from some other system? Does it have the chipset drivers, etc for that motherboard installed? What version of Windows … 10?
When it doesn’t lock up have you looked around in the BIOS?
Do you have your heart set on using that motherboard? I mean, it seems like it was great 10 years ago, but might not be the best choice now depending on your use case. Alternately you might (I’m not sure) need to use a period appropriate OS (Ubuntu 12.04, Windows Server 2012) to use the PI49230-2X2B/S-331 SATA port multiplier solution connecting the 45 drives; I’m not sure that will be supported in a modern OS (?). I mean if it does work for your use case, great, less ewaste.
It sounds like you can at least reliably boot the motherboard now without hanging and with the 9211-8i in slot 3, so that’s progress.
So the OS drive is connected to the White ports (2 above the grey ones) and its Win10 installed fresh on this system. I don’t plan on using windows, but was in tally having issues with linux booting so I tried something else. It has absolutely no drivers installed and I’m not really attached to this Mobos, but I really want to validate these backplanes as they don’t seem to work as I’d expect and I think the OS could be an issue? Like I have a plan if I HAD to direct wire all of these back to some new HBA cards, but I’d really like to avoid that if I have to since it will end up being close to $800.
I was able to test with a different power supply and it seems like (Yet to determine reliability) I can see a drive that is on the backplane without any issues. Now, I do have 5 of them plugged in, so not sure why only one is showing up, unless this is something that might need a driver?
I do think the USB issue might be something in BIOS I can fix, I just need to look around a bit more, but really what would the point be if I end up swapping it for something else.
If they made an HL30 that would be like the ideal solution tbh, but here I am experimenting with older hardware.
(EDIT) I think we can consider this matter closed. I finally found a forum thread for someone using these cards and they had issues and had to revert back to Ubuntu 16.04. (Good call on your part, always something to learn)
I have a Q30 that is about 10 years old. So not quite as old as your pod. It doesn’t have the SATA expander backplanes (is direct wired), but used a Highpoint Rocket 750 HBA to attach the drives. That HBA, although a SAS HBA, also uses some on-noard SAS expander functionality to support all the drives. Support for it was dropped from Linux at some point, although I think it is still supported in Windows 10.
The S-331 backplanes aren’t like the HL-15 backplanes, the S-331 multiplexes the five SATA drives over one SATA cable, whereas the HL15 backplane is just the wiring to support direct connection of all the SAS channels and power to the drives. The port multiplier backplane solution can be kind of jenky and is the reason 45Drives moved away from it. The SFF-8087/8643 connectors are designed to have four separate drives running over them. The 7-pin SATA cables and connectors aren’t.
Based on this, you might be able to get the PI49230-2X2Bs and backplanes working in Windows 10, but it doesn’t surprise me you might have to fall back to an older version for Linux… If you search for the Rosewill RC-230 that seems to be a rebrand of the PI49230-2X2B and apparently came with a driver CD. I don’t know that whatever drivers were on that are still available for download.
SAS expanders are pretty common and I think any HBA will handle them, but I think you need special hardware (that the PI49230-2X2B has) to handle the multiplexing of more than one drive over a SATA cable, and is probably why they needed to use the PI49230-2X2B cards and not the blue SATA ports on the motherboard.
Yeah, really the plan was to use TrueNas Scale, but with the age of and limitation of these parts I really don’t think it would be worth the time and effort. I was already planning on swapping out the PI49230-2X2B with a LSI SAS 9305-24i any ways since I wasn’t really a fan of the cheaper cards.
So I guess now I should decide if I want to keep this pod and have a fun summer project or send it back and keep waiting for an HL30 to appear someday or find something more modern to appear on eBay.
You can contact 45Drives and they will give you a quote on a bare Q30.
It seems like most of the design work has been done and it’s really more a matter of acquiring the boards the drive connectors mount to and doing all the cable management to wire those in. Tedious, but probably less expensive than a bare Q30. I think the HL folks have been pretty adamant that a HL30 is not on the radar any time soon. I shouldn’t think ‘direct wired’ with actual wires would be less reliable than the direct wired backplanes.
Yeah those sata expander cards are very picky about what HBAs will work. Are you sure the HBA you have has the correct Marvell chipset? At least with the older sata2 sata multiplexers you were just asking for trouble if you used anything besides the same chipset. I did a decent bit of testing with other HBAs to try to get more bandwidth before I started my direct attach project.
As for the chassis you have I would still keep the pod regardless of what you want to do at present. Even if you don’t want to go through the effort now it may make sense in the future. I find these chassis very well made, including most things 45drives makes. I can’t think of any other server gear I have had from the same era that is as upgradable or makes as much sense to run. There is also someone working on new sas backplane PCB for the 2.0 chassis so that might make the process a lot easier.
But yeah, as far as the process for direct attach converting your pod it is tedious. You could probably have everything fully assembled in a weekend or less if you have everything you need. For reliability so far I have had no issues but another person whom also upgraded previously, with a slightly different design, did have some trouble after a year or so.
Thanks for the Info! I read through your post and I will say there was a ton of great information in there. I have the means to make the conversion, but don’t really have the time to commit to it. I have decided to keep the chassis, I’ve realized just how flexible it can be at least for me. I have already started mocking up some 3D printed additions for SSDs, Maybe a micro ITX. board and a few other things. Right now I think my focus is to really get rid of the dual power supply architecture and move to a single PSU design.
It all seems easy enough, I just really need the time to sit down and work through it. I was able to get everything working with Unraid and TrueNas so thats always a plus. I saw the notes on the individual working on the backplane and I might just wait to see how that project turns out, I think for me it seems like a clean path forward imo.
