ICH9R Matrix RAID5 write problems on Win 7/2008

[mmix]

I use onboard raid controller on Gigabyte GB-X48-DQ6 (1.1) mobo. I used Intel matrix to split the 4x750Gb WD disks (Raid edition) into one operative RAID0 (800Gb) and one storage RAID5 volume (1.45Tb). I turned off write cache on all in order to show my problem:

Read speed is what I expected and write speed on raid0 is what I expected but the write speed on the raid5 volume is just awful even for large sequential files.

And I dont know if this is a bug or not, but event though the caching is disabled everywhere the Win7 and 2008 KEEP caching the writes on big files. If I copy a 4Gb file from raid0 to raid5 it immediatelly shows 300Mb/s but its a lie as everything goes into memory) and then when the memory is filled to some degree (85%ish) the copying stalls to 7-10Mb/s and writing continues after the copying seems done as cache slowly gets drained into volume. You can imagine what kind of problems this can cause if power fails.

I need to improve the speed on raid5 as much as possible while completely removing caching on raid5. I have plenty CPU so if xor-ing is the problem can I pump up the CPU usage for raid5 somehow? I dont care if it uses 50% cpu while writing, as during copying I hardly do other things.

 

[sub mesa]

So? Any RAID5 with write-through is going to have awful write performance.

Turn of the disk write cache of a single disk, and it will have awful write performance as well. You need write-back to make this thing fast, and it comes with an inherent risk. Your RAID5 won't protect you against lost buffers; so don't think you're safe at all with that RAID5. It can provide nice speeds though, with the Intel ICHxR write caching option enabled.

 

[sub mesa]

XOR by the way, is such an easy instruction, a $20 CPU could do more than 10GB/s of XOR. So XOR is not the reason RAID5s are slow; and it doesn't have anything to do with your CPU as well.

If you want to get to details, you should first understand:
- write-back
- full stripe block and the queue depth needed to saturate it
- 1 phase writes / 2 phase writes

If you do understand those three things, you would know a RAID5 would perform awful without write-back mode.

 

[mmix]

XOR by the way, is such an easy instruction, a $20 CPU could do more than 10GB/s of XOR. So XOR is not the reason RAID5s are slow; and it doesn't have anything to do with your CPU as well.

Ok, if that is true, and I also think its true, then why all the "you need a dedicated XOR machine a.k.a. h/w raid controller for fast RAID5" dogma that is floating around internet? What exactly makes a PC unsuitable to handle controlling fast RAID5 if required resources are in abundance? I should at worst get a slightly lower results than for my raid0 (in burst).

This still doesnt answer why Windows 2008r2/ windows 7 is caching the file writes at all if I turned it off on all levels.

If you want to get to details, you should first understand:
- write-back
- full stripe block and the queue depth needed to saturate it
- 1 phase writes / 2 phase writes

If you do understand those three things, you would know a RAID5 would perform awful without write-back mode.

Ok, then educate me, Before this setup, I had everyhthing on a single RAID5 without write-back and burst write was around 70Mb/s, now its 7, same disks, same CPU, same memory, the only things changed is that now I have two volumes matrixed on the same disks and now I have windows 2008r2/ win 7 OS's (I used the switch to those OS's to recreate my volumes). I did the split because raid5 was slow on large amounts of small writes.

I know what write back (lazy write) cache is (I dont know about saturation and phase commit in raid, so if you have links I'll be glad to learn) but as far as I see that should not penalize me while copying large files to raid5.

Just for kicks I have turned on write-back cache on ICH using intel console and I got very similar results:

It just "dampened" small writes, the burst write remained very similar, topped at 13mb/s, which is crazy slow. The file copying still remained the same, windows caches it in Gb's and then slowly drains the cache to drive. I know raid5 wont protect me from buffer loss but there is a big difference between loosing few blocks and loosing 2.5Gb of pending writes especially when they include modifications to NTFS structure. I use it for redundancy and I write rarely to it (once every few days), but when I write to it I copy several large files from 1 to 4 gb's.

 

[mmix]

This forum does not allow me to edit my older messages, so here's the original image from first message reuploaded to a different site as the link is broken (if mod can edit my first message, you can replace the image there)

 

[mmix]

Ok, I found a solution for the first problem, it was in the default configuration for ICH9 raid volume. By default the disk level cache is turned off and that was killing the performance. I enabled that and I got burst speeds of 140Mb/s w/o write back and CPU util went up during copying to appx 5%, which is good and marry.

However, the second question is still a mistery. Why is Windows 2008 R2 / Windows 7 caching the copying when its turned off? I mean now the cache gets drained to storage much faster (we are now talking 6-7sec instead of 2 minutes), but it still does it. is there another switch somewhere which controls it?

 

[apache_lives]

Software RAID5 in general sucks, and you may have high speeds now but you may find that as soon as you copy/transfer a file that is larger then your system memory (used as a buffer) your transfers will go down to say ~5mb/s tops - double check the performance

btw with Intel software you can enable/disable caching and also sometimes in device manager, its off be default because of the risk of loosing data within the cache because it may report that the file transfer is complete even though its still within system memory writing to the hdd in the background - power loss will result in that data disappearing or becoming corrupted hence why by default its off etc

 

[sub mesa]

apache_lives: Software RAID5 is not inferior to hardware RAID5 in any way regarding performance; it even has a chance of outperforming hardware RAID5 because it has access to faster hardware.

Sure if you look at Windows; all RAID5 implementations just suck. Not a single good one. Only the Intel ICHxR onboard RAID drivers can do mediocre RAID5.

But if you look at Linux/BSD, software RAID should be beating hardware RAID in some key areas. Its certainly possible to get > 500MB/s sequential write with Software RAID5. You just need 8+ disks. If one disk is on PCI though, your speeds will drop considerably due to some latency issues.

RAID5 is a complicated RAID level; and you need some intelligent driver to make this work fast, one that implements write-back buffering and I/O combining. The remark apache_lives gives about the buffer is not completely true. True if you write to the buffer it'll go very fast, but most buffers aren't very large and the purpose of the buffer is not to make the first 100MB go fast; but to change the 5MB/s sequential write to 500MB/s sequential write.

This all has to do with 2-phase writes. Without buffering, the RAID5 driver needs to READ data before it can WRITE something. It needs to read-calculate parity and write again; causing many seeks and killing performance on HDD. By combining I/O in the buffer to the exact size of the full stripe block: (number of disks - 1) * <stripesize> = full stripe block. So say the full stripe block is 192KiB large. If you can make sure every I/O is exactly 192KiB large it will be very fast. A fraction off and the speeds drop to 5MB/s again.

But i'll state again: software RAID5 is by no means inferior to hardware RAID5. Its windows software which is inferior; not the RAID technology itself. The only thing hardware RAID5 controllers add is battery protected write-back buffers. But if you don't use Windows you may not need those anyway because Linux/BSD has more advanced filesystems than NTFS in this regard.

BTW even though Intel has named their option "Write Caching" this is a write-back buffercache; not a normal RAM cache! So i'd call it a buffer instead of a cache.

 

[sub mesa]

Ok, if that is true, and I also think its true, then why all the "you need a dedicated XOR machine a.k.a. h/w raid controller for fast RAID5" dogma that is floating around internet?[/quote]
Because many people are still using Windows and Windows sucks when it comes to RAID. Microsoft could have easily made a good internal RAID engine of their own; even making them bootable. But either they can't or they won't.

So its a choice between windows + hardware RAID, or Linux/BSD + software RAID. Both will get you about the same speeds i'd say. If you're using network the speeds are usually capped at gigabit speeds anyway.

What exactly makes a PC unsuitable to handle controlling fast RAID5 if required resources are in abundance? I should at worst get a slightly lower results than for my raid0 (in burst).

It has nothing to do with CPU performance; people think RAID5 uses alot of CPU, the truth is:

Simple RAID5 drivers without write-back, use very little CPU
Advanced RAID5 drivers with write-back support and I/O combining, use alot more CPU

The actual CPU usage will be memory copies (memory bandwidth) - it will NOT be XOR! XOR should take less than 5% of the CPU time the driver needs in total. XOR = supereasy instruction.

Another reason to use Hardware RAID is because they allow you to use battery backup modules; to protect the write-back against lost buffers. Since Windows does not offer any filesystem with strong journaling; only light meta-data only journaling using NTFS. You kind of need a hardware RAID controller with Battery Backup Unit to use RAID5 + write back safely on Windows/NTFS filesystems.

 

[wuzy]

sub mesa has explained what software RAID actually is quite well. What most of us are used to seeing are actually host RAID (aka fake RAID) with either unoptimised driver on Windows and in the case of RAID5 (except Intel) the lack of write-back caching.

As to why hardware RAID has been perceived as the far superior option over anything else, you can blame that solely on Windows. Once you've tried out software RAID in *nix that's been optimised for x86-64, 32-bit RISC.. etc. (whichever distro it is), they're a storage solution to be reckoned with and competing right up at the top with the most powerful hardware RAID controllers.

When using host or software RAID5 with system RAM as write-back caching, if you have a UPS that alone virtually acts as the BBU on hardware RAID controllers.

[EDITED]Probably the only major advantage hardware RAID controller has over software RAID is array migration in the case of computer failure where you can still move the array along with the battery-backed data on the controller to another computer. Never done it myself and I hope I won't ever need to do it. Considering write-back caching is essential to write performance for RAID5 and 6, usage of RAM (on a dedicated controller with BBU) is a must. Hardware RAID controllers with support for parity aren't cheap. So you see, RAID5 or 6 isn't exactly economical in terms of $/GB in some cases unless you run a massive storage array.

 

[sub mesa]

Moving an array works with software RAID too; though you're kind of bound by keeping the same operating system. But unlike hardware RAID, with software RAID you'll be 100% hardware independent. You can replace ALL hardware, including the disks themselves (raw copy contents) and still have a working RAID even though its a completely different system. Hardware RAID keeps involving the controller. If that doesn't work or breaks; redundancy is no good to you.

Generally i'd say software RAID has the potential to be superior to hardware RAID; it has access to the fastest hardware and it allows for completely hardware independent design; which is good.

And you don't always need a UPS if you're using software RAID + writeback. With strong journaling and BIO flushes you can use write back safely without using a UPS. So if the power goes down it will have lost alot of writes; the trick here is recovering from that. With heavy journaling (metadata+data) the filesystem knows what happened and how to recover. NTFS is only light journaling with only meta-data being journaled; its not enough to recover from lost buffer write-back situations.

Maybe i should put up a ZFS fileserver tutorial sometime. ZFS is so cool but its not being used by many consumers; while it could also be of great interest to them.