RAID Guru Help Required....

[blackmesa]

Hi,

I just registered to this forum to confirm exactly the same problem with a ServeRAID 7k on a xSeries 346.

The ServerRAID 7k also is only an "addon" RAID Controller which is added to the motherboard and uses the onboard scsi controller. It brings 256 MB cache and uses the Adaptec AIC 7902B as back end as well as your ServeRAID 6m does.

I have 4 300 Gig 10k RPM U320 drives connected at channel 1 with a RAID5 and 8 kb stripe size.

None of the benchmarks ever reaches above 60 MB/s sequentially reads.

Unfortunately my IBM hardware supplier also doesnt consider the poor transfer rate as a problem.

I benched with HDtach, HDtune, h2benchw and will try IOmeter.

What I already did to find the cause:
- Upgraded controller firmware from 7.12.07 to 7.12.12 -> nothing
- Upgraded controller driver from 7.10.08 to 7.12.11 -> nothing
- Used the hard disk firmware update pack -> nothing
- Verified cache settings in controller bios -> didn't find any cache related settings

We use this system as "backup collector" before staging about 800 gigs to a single lto3 drive. The array is unable to feed the lto3 drive with the needed 80 mb/s!

Things to do:
- Remove the logical drives from the array and do "raw" benchmarks
- Delete the array and create a new one with 256k stripe size.

I found an interesting article from an Adaptec guy about stripe sizes and RAID5, read here: http://storageadvisors.adaptec.com/2006/06/05/picking-the-right-stripe-size/

hdtach result:


hdtune result:

 

[Whizzard9992]

Thanks for your input; I appreciate you taking the time to register and post.

I tend to think that people in enterprise environment just don't pay enough attention to the performance of the disk array. I've received a lot of resistance here to the notion that our array is slow, but I'm putting a lot of pressure on the help desk.

Ultimately, I would expect any RAID configuration to perform sequential reads faster than the single-drive equivalent.

One thing that you can try is to enable "advanced performace". It's one option that, if not enabled, can really hurt RAID array performance in windows 2000/2003. Unfortunately, the option requires drvier support which the 6M doesn't appear to have. Maybe the 7M does?

If you right-click the drive -> properties -> hardware -> Properties -> Policies, you'll get the option. Let me know if you can select it, and if it helps.

Can you use robocopy to copy to/from different volumes?

What really pisses me off is that I can copy a file from my home computer to my external E-IDE drive (on a USB bridge) in excess of 50MB/s, but when copying a file on my RAID array from one channel to another, I get < 30MB/s.

 

[blackmesa]

If you right-click the drive -> properties -> hardware -> Properties -> Policies, you'll get the option. Let me know if you can select it, and if it helps.

Can you use robocopy to copy to/from different volumes?

What really pisses me off is that I can copy a file from my home computer to my external E-IDE drive (on a USB bridge) in excess of 50MB/s, but when copying a file on my RAID array from one channel to another, I get < 30MB/s.

The option is greyed out by the driver.

Did not do any robocopy tests because I do not expect a better result.

 

[Whizzard9992]

I've scheduled a series of outages for the box.

I'm going to try changing (increasing) the stripe size, updating the drivers, and changing the drive distribution across the channels. I'm also going to move to 5EE (We're running 5. I thought we were running 5EE previously).

It's going to take some time (a few weeks, probably ), but I'll keep posting updates here.

 

[blackmesa]

My benchmarks from IOmeter:


Again poor results.

I will increase the stripe size on tuesday to 256 kb, although I don't believe that will show a negligible improvement.
After that I will have no ideas left for tuning the RAID and will contact my dealer again and ask him to put pressure on IBM. At least IBM should give us a statement regarding the expected or estimated performance of their RAID controllers.

BTW: This guy here seems to suffer under the same problem: http://forums.storagereview.net/index.php?showtopic=24017&

 

[Whizzard9992]

BTW: This guy here seems to suffer under the same problem: http://forums.storagereview.net/index.php?showtopic=24017&

Um, yeah, that's me

 

[sandmanwn]

LOL

Heres my final answer. Take 6M out, clutch with both hands, bang repeatedly over help desk guys head until either the card is shattered or until the help desk guy is laying helplessly on the floor.

Then get better Raid card. Benchmark new results, print out copy, send them inside the get well soon card to the help desk guy in the hospital.

:lol:

 

[blackmesa]

Slight improvement with increased stripe size. Yesterday I increased the stripe size from 8k (default) to 64k (max) and formatted the logical drive with ntfs 64k block size.

Needless to say that I'm still not satisfied with this 25-30% improvement.

HDTach (32 MB Zones) - 8k stripe size:

HDTach (32 MB Zones) - 64k stripe size:

HDTune - 8k stripe size:

HDTune - 64k stripe size:


Reference - just to make clear what is possible with only 2 drives reading sequentially:

HDTach on Intel ICH7R RAID0 over 2x Seagate SATA 7200.8:

 

[Whizzard9992]

UPDATE

For anyone still interested, here's the deal:

We've convinced the help desk that there's a problem. They opened up a ticket with IBM, and IBM inspected the logs and said, "Everything appears to be fine. It's a software issue."

*cough* BS *cough*

So now we have to pay for a software support call to convince IBM that our server is configured properly, and that it's an issue with the hardware. Unfortunately, we're entering into a situation where different IBM support divisions will be pointing the finger at one-another while our production server limps around.

IBM sucks.

I'll post an update once we make some progress with IBM. All and all, it's probably just an underperforming card, and it surprises me that after being on the market for X years, there are still serious performance issues.

If anyone has a 6M that performs WELL, please let me know.

 

[blackmesa]

Exactly the same IBM reaction here.

And we DO have a service contract. IBM declares the problem to be no problem.

Next week I will remove the RAID controller and benchmark the performance with soft-raid. I feel confident that it will outperform the hw-raid. :x

 

[Whizzard9992]

Yeah. If we put a 3rd-party card in there, it'll void our warranty on the server (for which we paid a premium).

We spoke to the pre-sales engineer, and they've forwarded it to the performance testing labs @ IBM. We're also setting up a test box here so we can run tests on a machine that's NOT in production.

It's going to take some time, but I think they're on it. It's rediculous how much BS we have to go through, despite the fact that we have a support contract.

Again, I'll keep everyone up to date when we make more progress. Things are moving, albeit slowly.

 

[tsanders]

Hello all,

I am also having the exact same issue as blackmesa w/ a Xseries 346 Server w/ 7K RAID card.

My raid arrays are:
2 U320 SCSI 10K 74 GB drives in RAID1
4 U320 SCSI 15K 36 BG drives in RAID5 8K stripe
Spoke w/ IBM regarding channel configuration...apparently all drives must remain on channel 2. Channel 1 for external use only....

HDTACH
RAID1 = 55 MB/s sequential read.
RAID5 = 60 MB/s sequential read.

I was planning on implementing this server for SQL 2005.
After reviewing these numbers, I am very skeptical.
My ThinkPad T60p has better I/O.
It is apparent to me that this server has one hell of a bottleneck.

IBM support contract is worthless because they only support problems.
Slow performance somehow does not fall into this category.

I am about a day or two away from selling it or throwing it out of window.
Dell sounding pretty nice rigtht now.

Has anyone had IBM confess to this issue?
Has anyone resolved the performance issue? 20% improvement w/ 64K stripe is not much w/ #'s this low.

Also curious what U320 SCSI RAID benchmarks others have had w/ previous Xseries servers.

Thank you

tsanders

 

[sandmanwn]

IBM did confess to an issue with the first series 5i but no word sense way back then. It clearly hasnt gone away however.

More than likely you can get a generic LSI or something comparable and it will beat a 5i, 6i, 7i in its own system as we have discovered.

Dell, well what can I say positive about dell, hmmph. I would suggest HP as an alternative. Theyve been in the game a lot longer than Dell and their are lots of benchmarks using the DL585 which seems to be a favorite model.

 

[blackmesa]

IBM sucks. And so does the wannabe-RAID-controller ServeRAID7k. It's nothing but a software-RAID-controller with hardware-Cache.

IBM does not support any other RAID-Controller within xSeries 346 servers beside their ServeRAID7k.

I bought myself an ICP Vortex (German company which belongs to Adaptec) RO9014 PCI-X RAID-Controller (~450 Euros, 1 Channel, 256 MB) and connected it with a hd68-female-to-female-adapter and extension cable to the backplane. The damn backplane has an IBM proprietary connector so you won't find any suitable cable to connect it directly with the third party PCI-X Controller.

Here my new benchmarks, remember the old ones. Best result was 77 MB/s average read with HDTach 3.1 about 4x 300 GB disks in RAID5.

Now I am on 182 MB/s average read!



And to proof how much performance is swallowed by the wannabe-serverraid I benchmarked the performance of a single U320 300 GB 10.000 rpm disk, 73 MB/s average read:

 

[Whizzard9992]

Wow. Those are great benches. Thanks for sticking around. It's nice to see someone can actually pull good benches from a SCSI RAID controller (They're hard to find online).

I haven't posted in a while, but they set up a test box with another 6M they had "laying around" so that they could test out new settings and what-not.

Now we're just waiting on IBM. Unfortunately, there are so many people between me and the people who can actually fix this problem, I'm pretty much in the dark, forced to beg the 1 contact I have for info on a daily basis. Needless to say, it's 3 months from the discovery of the problem, and we're about 5% closer to a fix than when we started.

I'll post a fix if and when we get one.

I honestly think that IBM controllers just suck, but there are so many incompetent techs that few people notice. It's sad, really.

 

[sandmanwn]

yeah its a shame that you can pick up just about any controller, run it through adapters which is probably eating away some performance, and it still outperforms the IBM controller by a wide margin even though the IBM solution was designed for the server.

 

[tsanders]

IBM sucks. And so does the wannabe-RAID-controller ServeRAID7k. It's nothing but a software-RAID-controller with hardware-Cache.

IBM does not support any other RAID-Controller within xSeries 346 servers beside their ServeRAID7k.

I bought myself an ICP Vortex (German company which belongs to Adaptec) RO9014 PCI-X RAID-Controller (~450 Euros, 1 Channel, 256 MB) and connected it with a hd68-female-to-female-adapter and extension cable to the backplane. The damn backplane has an IBM proprietary connector so you won't find any suitable cable to connect it directly with the third party PCI-X Controller.

Here my new benchmarks, remember the old ones. Best result was 77 MB/s average read with HDTach 3.1 about 4x 300 GB disks in RAID5.

Now I am on 182 MB/s average read!



And to proof how much performance is swallowed by the wannabe-serverraid I benchmarked the performance of a single U320 300 GB 10.000 rpm disk, 73 MB/s average read:

Could you be more specific as to what you used to connect the PCI-x RAID controller to the backplane and where you got them at? We are in the same situation and would like a solution. It seems you have one.

Thanks,

 

[blackmesa]

Could you be more specific as to what you used to connect the PCI-x RAID controller to the backplane and where you got them at? We are in the same situation and would like a solution. It seems you have one.

Thanks,

Within the x346 server you will find a 10 inch cable which connects the backplane with the onboard scsi connector. You have to extend this one. You will need a HD68 female-to-female Adapter. Something like this: http://www.extend.de/cgi-local/katalog?1DB011~detail
Take the screws out, throw the plate away and use it as internal female-to-female adapter. On the one side you plug the internal IBM backplane-to-motherboard cable and on the other side you can connect a usual HD68 scsi cable with the appropriate length to reach the PCI-X controller.

Hope that helps!