lsi 9260-8i MegaRaid Card, Crap performance on a good card?

[androbourne]

So...

I'm running a supermicro server, in two raid 5s. One for OS and one for Data\VMs. The board is a X8DA3. http://www.supermicro.com/products/motherboard/QPI/5500/X8DA3.cfm

I was using the Intel ICH10R SATA 3.0Gbps on board raid controller. However that model was old back in 2008 and has very bad performance rating. I ran a CrystalDisk report on it before I swapped it out with the new one. I was getting around 600mb/s read and 400mb/s write. Using the two Raid5 with 3x 7200 rpm drives in each Raid5.

I found this LSI Mega Raid 9260-8i and researched the card before buying it. Apparently it is a decent card. Has double the on board cache then the Intel ICH10R had and also included a BBU so I can enable write back caching for increased performance. Also able to perform at 6.0GB speeds for SataIII. This card performance wise, is way better then this old ICH10R on board raid controller, by far.

I installed the new LSI 9260, recreated the arrays exactly the same way as they were with the old controller and reinstalled fresh copy of Windows Server 2012 std then restored my data files to the data partition etc... everything is working fine.

I allowed the array to fully complete building etc.... and began drive tests again with Crystal Disk.

I'm getting complete crap results. 149mb/s on the read and 101.1mb/s on the write...

I followed this document https://www.supermicro.com/CaseStudies/6Gbs_LSI_b.pdf

Which suggested the raid/ caching settings. Made the changes and tested again. Same results. I even turned on no write buffering on the drives. No change again.

I went as far as updating the disk drivers and LSI Raid Controller firmware to the newest versions, I still get the same results...

The weird thing is not only do I get the same results, but making those changes SHOULD provide some change in results. Either negative or positive, but it doesn't. Every run is identical...

I can tell the changes are taking place as I'm using 3 different programs to monitor the Raid and the changes reflect each software properly.

I've googled around and see tons of people having performance issues with LSI but also tons of videos about it actually working pretty decently. (even Linus from TechTips did a video on this exact card and got pretty good performance).

I don't expect to get SSD like speeds, however I do expect a better card to get better performance then the on board raid controller...

Any ideas?

 

[marko55]

Yeah something's not right. I've used that card for RAID-5 arrays with spinning disks and it performed very well.

In the config of your RAID-5, ensure disk caching is enabled. This alone can destroy performance.

Do you have MSM installed? Check the temp of the RAID chip. These tend to run REALLY hot. I install 9266 and 9271 cards regularly and I always put a noctua 40mm fan directly on the ROC's heatsink to keep it cool.

Also, can you post every setting you've configured on your RAID-5?

 

[androbourne]

Disk caching could be possible. I noticed that under my settings it says "disk caching unchanged". I don't see an option to enable it via the software so I'm guessing it has to be done on the BIOS side? Can't you just configure it from device manager? Because it is enabled in there.

Here are my settings.

I've tried it with write caching enable and disabled. I've tried it with both options enabled at the same time, one other the one enabled at different times and both options turned off. No change.

I noticed here that LSI suggests turning off "read ahead" but if I turn it off. It disables write caching on the hard drive in device manager. So I left it on. However. I have it off for my OS partition and disk speed tests are the same between array raid groups. Also you can see there "Disk Cache Policy: Unchanged". But if it is enabled in the above under device manager, shouldn't that be reflected here? Or is that two totally different settings altogether?

As for heat. I actually have pretty decent cooling in my case as is. The heatsink even when on is almost bearable to the touch! However, I've read that they do heat up quite abit during full load operation, so I may take your suggestion and add a fan. Can you provide the model fan you used?

As for the heat. Where is MSM is it located? I'm more familiar with HP Arrays. I've only done a few LSI installations on older Lenvo servers.

And below is just the status of the Card overall.

I use SuperDoctor for Supermico, LSI MegaRaid Manager and also the web version of LSI MediaRaid Manager called LSI Storage Authority. All settings appear to reflect on them properly.

 

[marko55]

Install megaraid storage manager. You want to configure your raid there. Enable disk caching there

 

[androbourne]

I have MSM installed. I just don't see where that option is in the menus. Instead, I went into the Raid BIOS and enabled it from there. I still getting the same results... WHY!?!!?

Before enabled Drive Cache

After enabling Drive Cache

Disk Cache Enabled

Any other ideas?

 

[marko55]

Try setting your IO policy to "Cached IO" and see what happens. It not utilizing the cache right now. Outside of that everything looks good.

I'd change back anything you've changed on the virtual drive in Windows. You don't need to touch any of that. The RAID controller controls all the important settings right here.

 

[androbourne]

Those disk settings are controlled by the Raid software anyways. I can't manually change them.

None of this makes sense what so ever....

I'm now getting worse performance then before!!

Drive Cache and Cached IO enabled

I'm really considering at this point the card is defective and I'm either going back to the default on board Raid or buy a new card... This makes no sense!

 

[marko55]

Ok, so all those settings are ideal. In your shoes at this point, given all the changes that have been made, I'd delete the data array in Megaraid storage manager and recreate it. Set all the settings you have there, and use a 64K strip. When you format the disk in Windows, use 64K there as well. Do a fast initialization at the end of the creation of the logical volume in MSM so it presents it to Windows right away. Even during background initialization it should perform better than what you're seeing.

Do not make any changes at all to the disk in windows.

If this doesn't help, I've gotta think bad firmware. Most performance cards with LSI cards all dated back to old firmware that was resolved in later firmwares. Unfortunately you can't get firmware for that card anymore from broadcom's website. Not really sure where you can. I'm not even sure what the latest version for that card was to be able to compare to the version you've already flashed to.

What I can tell you is I had one of those cards a couple years ago with at least 6 x HDDs and tested several different RAID configurations and had ideal/expected/normal performance. The only place that card lacked in performance for me was when putting SSDs in RAID. It just didn't have the juice to keep up with the SSDs. The 9266's and 9271's I use nowadays all rock.

 

[androbourne]

Yeah I've use LSI in the past and never had an issue. It is weird for sure.

As for the firmware. I believe this was a Lenovo card. I was able to flash the firmware from Sept 2016, I think that was the last released and its fairly current. So I don't think it is a firmware problem. Plus it was acting this way before I flashed the firmware, it was one a version from 2008 I believe. Same performance.

I'll try recreating my D: array as you suggested and see what happens. I'll post results soon.

 

[androbourne]

Well. Did as you suggested. No change really.

Current firmware installed.

http://support.lenovo.com/us/en/downloads/DS040844

I dont know whats going on... I'm going to try on board again and see what gives. May just need to get another card : /

I'll let it run over night and try it again. I'll also try again with cached IO but I doubt anything will change.

Between the 9266's and 9271's, which one would you recommend? It's only 100$ more I may just upgrade to that instead.

Right now I'm just using HHD 7200rpm drives but in the future I may upgrade to SSDs.

 

[androbourne]

Yep. Changed to use Always Write Back instead of Write Back with BBU (just incase BBU was failing and just not reporting it properly), also enabled Cached IO.

I'm getting 166 mb/s Read and 96 mb/s Writes. No real change.

Something is broken here!

 

[androbourne]

So I went ahead and purchased that fan you recommended. I also bought two additional drives to make both my raid 5s, into Raid 10s instead.

Just to be safe I also purchased a new BBU. This one was used with the device when I purchased it and while it is not displaying any errors. I want to test it with a spare just to be safe.

I'll post back in a week or so when I have the new parts installed. I should atleast see a decent performance boost with the Raid 10 in place. If I don't see expected results (4x read and 2x write is expected). I'm at the point of thinking the card is defective.

 

[marko55]

Well. Did as you suggested. No change really.

Current firmware installed.

http://support.lenovo.com/us/en/downloads/DS040844

I dont know whats going on... I'm going to try on board again and see what gives. May just need to get another card : /

I'll let it run over night and try it again. I'll also try again with cached IO but I doubt anything will change.

Between the 9266's and 9271's, which one would you recommend? It's only 100$ more I may just upgrade to that instead.

Right now I'm just using HHD 7200rpm drives but in the future I may upgrade to SSDs.

Both cards are basically identical. Same ROC and same RAM. The 9271 just has a PCIe 3.0 connection to the mobo so more bandwidth there. But if you're feeding 8 PCIe lanes to the 9266 you're talking about roughly 4GB/s of bandwidth to the motherboard. If you fully populated that card with SSDs you may be able to saturate that, again, theoretically. Where the additional bandwidth to the board is needed is if you're got a SAS expander (or multiple) chained off the RAID card and a whole bunch of drives feeding up through that connection to the system. If you're running 8 or less drives I'd go with which ever you want. I've seen no performance difference between the two. I've got a 9266-8i with a SAS expander in my home ESXi host with a SAS expander feeding a RAID-5 of 4 x SSDs and a RAID-5 of 5 x 4TB Seagate Enterprise drives. It handles it all with ease.

I'll be straight with you man, grab one of these off Ebay for cheap. Even the ones listed as new and from Hong Kong. I've bought several of those and had absolutely zero issues. Even IF they're "counterfit" (some people claim they are but I'm not so sure) you'll be damn hard pressed to find a difference between them and a real one (I can't). My guess is they fall off trucks out there and get sold on Ebay. Regardless, we get em dirt cheap, you can buy two for the price of one if warranty is your concern, and they run 100% the same as a brand new retail card from newegg or somewhere. They 100% support the firmware you pull from broadcom.com, work with BBUs and work with all Megaraid utilities I've used including MSM and MegaCLI. I honestly think they're the real deal.

Ones like this are sold from US stock so they get to you quicker: http://www.ebay.com/itm/New-Sealed-LSI-MegaRAID-9271-8i-8-port-PCI-E-6Gbps-RAID-Controller-Card-US-Sell-/301025144053?epid=1525114010&hash=item46167f2cf5:g:nZwAAMXQUmFSkOxl

 

[marko55]

I had another thought too. I didn't realize you had one of the rebranded IBM cards. I wonder if you can hunt down a firmware from LSI, vs IBM/Lenovo, and manage to get it flashed to the adapter (may require using MegaCLI) if that will make a difference.

 

[androbourne]

Well I tried LSI. Couldn't find anything on the broadcom website. (LSI site redirects to broadcom). Found Lenovo ones, currently what is installed.

I'll look on Dells site and see if I can find any firmware for it instead. You'd think it accepted the firmware from Lenovo and its fairly recent. That wouldn't be an issue. (s/n sv03403078)

But let me look on Dells site and see if I can find anything.

Could the breakout cable be an issue? That is the only thing different. Before I was using 6x USB cables straight to the motherboard and backplane. When I put in the Raid controller I also added breakout cable to replace the 6x USB cables. (required anyways for the Raid card to go to 6x drives).

This is the breakout cable I'm using.

https://www.amazon.com/gp/product/B012BPLYJC/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

To the same backplane and same hard drives as used with the on board raid.

 

[marko55]

I couldn't find firmware for the 9260 on broadcom.com for the life of me.

 

[androbourne]

This is the only two of relevance I could find.

http://www.dell.com/support/home/us/en/4/Drivers/DriversDetails?driverId=V7KFR

http://support.lenovo.com/us/en/downloads/ds040844

I went ahead and flashed the Dell version even through its from 2013 and a few versions behind what I already have installed.

Rebooted, double checked settings and reran the test.

I'm getting basically the same results on disk speed tests.

 

[marko55]

Can't be something crazy like a PCIe connectivity issue can it? Maybe take a shot at a different PCIe slot on the board. Beyond that, may be time to abort brother. Grab another card.

 

[androbourne]

I'll try that this weekend when I get the rest of my parts in. I'm pretty sure it is in a PCIe8x slot that I've used for a video card earlier so i know it works.

But I'll give it a shot.

 

[androbourne]

So the card was originally in a 6x slot which is full size for the card. I moved it to a working 8x slot to test. No changes in speeds either.

 

[androbourne]

So I think there is an issue with the BBU.

I've bought two used BBU's (granted, used) however stated they have been tested for 30 days. I installed it properly and it appears to charge fine but when I run a relearn life cycle. They all failed.

That is 3 BBU's in a row that fail every single time I run a relearn or let it do an automatic relearn.

I highly doubt I have 3 bad BBU's... I'm starting to think something is faulty with the card.

I've also noticed something very strange. I was looking through my hard drive speeds and noticed 2 of the 8 drives are reporting speeds of 1.5 instead of 3.0... They are the same makes and models and serials even match the other drives as well...

Something isn't right here.

Same serial numbers on the drive... in fact all of them in the span has the same serials. 2 of the 4 in this span are reporting 1.5gpbs speeds.

I have another 4 in a span for my OS partition, also in a Raid 10. They all report 3.0gpbs speeds....

 

[marko55]

Ok that HDD connection speed is definitely an issue. Either that RAID card is bad or your SAS-to-SATA break out cables are bad. I'd really look to get in to a 9266 or 9271.

 

[androbourne]

It maybe the break out cable. I doubt the raid card because it only has two SAS ports and some of the drives on that second SAS port is functional at 3.0 speeds while 2 others are not.

I'll check it this weekend.

 

[marko55]

What kind of HDDs are you using? Are they only SATA II drives?

 

[androbourne]

I'm using 4 of these in a Raid10. SATA II I believe.

https://www.newegg.com/Product/Product.aspx?Item=9SIAAEE3ZZ7556&cm_re=st3500630as-_-9SIAAEE3ZZ7556-_-Product

Same serials. Same drives. Different speeds : /

All 4 are plugged into the same SAS port on the card. I can only think of is a backplane issue or like you suggested. maybe the breakout cable but that thing is brand new!

I'm going to power down the system and sit the drives and while I have them out verify the serials match what the LSI states.