RAID cap?

[dgingeri]

I've got 4 WD Caviar Black 750GB drives in a RAID 0 on a 3Ware 9650 controller for my OS and storage. (Yes, they're backed up daily to an external.) The system is a Core i7 920 overclocked to 4.2Ghz on an Asus P6T. I have 2 GTX 470 video cards in the 2 x16 slots, and the 3Ware controller in the bottom x4 slot.

I've had an odd issue where the transfer rate stays at 300MB/s for 90% of the capacity, then tapers off to 250MB/s. I'm fairly certain the drives aren't holding it back, since they'll transfer at 110MB/s each on their own. I know the RAID won't scale perfectly, but the way it caps at 300MB/s leads me to believe something else is interfering. The PCIe slot is x4, so the cap for that transfer rate should be right around 1GB/s. I'm thinking this thing should be able to reach around 380-400MB/s for the first 1/4 of the capacity. What do you guys think is causing the cap at 300MB/s?

 

[baddad]

If there SATA II drives that would be correct and really good.

 

[dgingeri]

for one drive, sure, but not for 4. The limitation on the bandwidth across 4 drives is 12Gb/s or about 1.2GB/s. The realistic limit for a single SATA lane would be about 280-285MB/s, so 4 lanes should cap out at 1.1GB/s.

 

[baddad]

http://www.xbitlabs.com/articles/storage/display/3ware-9650se_10.html#sect0

 

[sminlal]

I've had an odd issue where the transfer rate stays at 300MB/s for 90% of the capacity, then tapers off to 250MB/s. I'm fairly certain the drives aren't holding it back, since they'll transfer at 110MB/s each on their own.

The transfer rate of every hard disk drive is slower at the innermost cylinders (tracks) than the outermost. The vast majority of drives only have a transfer rate that's about half as fast at the last sectors as for the first ones. Your drives may have a transfer rate of 110MByte/sec near the first part of the first partition, but they sure won't be anywhere near that fast at the last part of the last partition.

The transfer rate difference of your RAID volume is undoubtedly due to the slower transfer rates of the drives themselves at their innermost cylinders. A transfer rate of 250MByte/sec for the four drives would work out to about 80MByte/sec or so per drive, which sounds about right if the average transfer rate of the drives is 110MByte/sec.

 

[dgingeri]

250MB/s would work out to about 62.5MB/s per drive, which is about right near the end of the drive. My transfer rate curve, through either HD Tach or HD Tune, show a flat 300MB/s from 0 to about 95%, then curving down to about 250 at the end. Oddly, once in a while, I get a 350MB/s for the first 25%, then a sharp drop to 300, flat at 300 from 25% to 95%, then a slight curve down to 250, but this only happens once in about 10-15 tries. I wish I could show you, but I can't. It should curve from about 380-400 at the beginning down about 250 near the end, if things are working correctly. I'm not sure what is capping it, though.

 

[sminlal]

It should curve from about 380-400 at the beginning down about 250 near the end, if things are working correctly. I'm not sure what is capping it, though.

It's because the innermost tracks on a disk platter are shorter than the outermost tacks. Basic math: track length (circumference) = track diameter * pi (3.14159).

Disks can only store so many bits per inch, therefore they store fewer bits on the innermost tracks.

Since most drives have inner tracks that are about half the diameter of the outer tracks, that means they only store about half as much data on the innermost tracks as the outermost ones.

Let's say a drive spins at 7200rpm and has a maximum transfer rate of 120MByte/sec. 7200rpm means 120 rotations per second (7200 rotations/minute divided by 60 secs/min = 120 rotations per second).

If the drive transfers 120MBytes at 120 rotations per second, it would mean that each outermost track holds 1MB (it's reading 120 complete tracks of 1MB each every second = 120MByte/sec)

The innermost tracks will only hold about half this much, or about 0.5MByte. Therefore their transfer rate will be 0.5MB x 120 rotations (tracks) per second = 60MByte/sec.

These are basic geometrical facts for all hard disk drives.

 

[dgingeri]

Yes, I know that much, which is exactly what led me to believe this controller was behaving strangely. (The Intel controller with these same drives capped at 100MB/s all the way across the array.)

These hard drives have a transfer rate just above 110MB/s on the outer tracks, so 4 of them should have a theoretical total of about 440MB/s at the beginning of the array. I know there is some inefficiency, so the transfer rate should actually show up as about 380MB/s or so. It's not.

In short, I should be getting a curve like this:

..........
......
......
.......

Instead, I'm getting something more like this:

..............................
....

It's not right.

edit: bah, the formatting isn't coming out right. Stupid auto space removal. Just imagine the first being a nice gentle curve downward while the second is almost completely flat with a slight curve at the end.

 

[sminlal]

OK, I see what you're getting at now - you're not concerned about the slower transfer rate at the end of the volume, you think it should be faster at the start of the volume. If I'd read your original post a little closer I should have seen that - apologies.

Which model of controller do you have? According to the spec sheet:

http://www.3ware.com/products/pdf/9650SE_DS_101106.pdf

...the 9650SE-2LP model is a one-lane PCIe controller - if that's the one you have the bottleneck is probably the one lane on the PCIe bus. If the card itself is a 1-lane card it won't matter that you've got it plugged into a X4 slot.

 

[dgingeri]

Yep, that's the line. I have the 8 port model. it's plugged into the x4 slot on my Asus P6T motherboard. The CPU is overclocked to 4.2 GHz with a 200Mhz base clock. the QPI is running much faster than normal, so I doubt that's the issue.

oh, wait, I just thought of something. that x4 slot runs off the southbridge. the bottleneck might be the link between the southbridge and the northbridge. I need to look up x58 specs a bit. I bet that's it.

 

[dgingeri]

No, that's not it after all. the connection between the northbridge and southbridge is a 2GB/s DMI link, which should handle this traffic without any issues.

4 drives -> 4SATA connections to controller -> x4 PCIe slot -> 2GB/s DMI link -> QPI link to CPU

I'm not seeing it. maybe it's a problem with the controller itself.

 

[sminlal]

It's not obvious to me why you're hitting that limit. Is there a 3Ware support forum? Those guys may have a better handle on it.

 

[dgingeri]

Best answer selected by dgingeri.