ICH10R - RAID 5 Failure

[wolf2]

Hello again folks. Sorry for the delay in the response to both Specialk90 and Sub mesa. I basically wanted information with teeth to it before posting back to the forum.

First of all, let me give you the update. I have completely ditched ICH10R RAID drivers after the last crash and decided to resort to good ol fashioned troubleshooting by eliminating one variable at a time. So I have loaded all the Fail Safe params in the BIOS (running F9 revision) and switched all the drivers to standard IDE not AHCI. I have dropped two drives out of the setup keeping a total of 5 x 500GB drives and 1 DVD Writer (all SATA).

The drives were setup as independent drives with no fault tolerance. I completed a fresh install of Windows XP on the first drive, installed all sound/lan drivers, and completed all the house keeping items of a fresh install. Then I put the machine to the test in the 24x7 with moderate read/write requests.

To my amazement, the machine started BSODS with the IRQL memory errors! Puzzled about this, i disconnected four drives and kept only one drive which has the OS. Then ran the machine again and was further puzzled by another BSOD within a few hours with the typical PAGE_FAULT_IN_NON_PAGED_AREA errors. Then it dawned on me what the last two posters have highlighted, the memory could be the culprit. I yanked one of my kid's DDR2 2GB 800 MHz RAM Chips out of his computer and into mine. I took out the two Corsair 2048MB 1066 MHz chips out of the server. I also reconnected all five drives back into service. Then booted the machine, it has been running non-stop for the past 4 or 5 days!

I feel that the memory is 80% of the problem right now but the acid test is a full 1-2 weeks of 24x7 operation without a failure. So I will keep you posted with what happens. Needless to say, I have no RAID setup whatsoever right now, but frankly I am really happy that I have a working system. Let's hope that it passes the two weeks mark and then I have to figure out where to go from there.

SpecialK on your question regarding cooling - I am using a hefty Gigabyte case with 2-front side fans and 2 back side fans. I paid a little extra cash at time of purchase to make sure I have plenty of fans in the machine. I am also using an Epsilon600W power supply with its own fan, so the system runs fairly cool across the board.

Sub mesa on your two questions, i believe that the answer now is related to the memory. The system seems to be fairly cool and I will double check the temperature of those chipsets over the next two weeks and let you know.

I am definitely interested in your suggestion about the dedicated RaidZ and ZFS and would appreciate any insight. I must say that having a file server that doubles up as a workstation is really nice. The file server provides media storage for the home network meanwhile its downloading files from the Internet in the background.

Thanks for all the help guys - it has really been crucial in isolating the problem and giving me hope when it was needed the most.

 

[zapf]

Hi Wolf2,
Glad to hear you sorted your ICH10R issues. We have 20+ machines running version of ICHxR and running matrix RAID. And we have found the same performance difference you have discovered between the onboard RAID5 and 10. Hence we use a 2 disk RAID1 for OS and system. And a 4 disk RAID 10 for our data. And we have been using higher end Gigabyte motherboards. Hopefully that is the end of your worries. Stupid "memory" )

Cheers
Zapf

 

[jrst]

...RAID-Z. Combining both filesystem and RAID-engine in one package allows for variable stripe sizes, so 2-phase writes of RAID5 disappear...

Going a bit OT here, but there seems to be some misconceptions about ZFS which need correcting...

ZFS does not eliminate "2-phase writes"--assuming you mean eliminate a potential low-level inconsistency due to, e.g., requiring atomic update of both the data and the parity drives, and which are typical potential failure modes of cheap RAID-5.

While ZFS doesn't eliminate it, it compensates for it with journaling, and thus makes it more robust. That applies across a ZFS vdev (i.e., it is a function of the ZFS on-disk structure, not a function of a specific file system). You could get pretty much the same from a decent/smart/BBU RAID-5 controller.

That said, I'd still prefer ZFS. And that said, the whole "RAID-5 write hole" brouhaha is due to cheap/dumb implementations (e.g., what you find with most mobo's), not because RAID-5 is fundamentally flawed. ZFS basically uses the exact same technique as RAID-5 (or RAID-6 with RAID-Z2). Variable stripe sizes are an optimization.

The need to perform atomic writes to multiple devices to ensure consistency doesn't "disappear" with ZFS--it's as constrained as anything else that has to write to disparate devices as an atomic action, but which can't ensure an atomic result because it is writing to disparate devices, any of which may fail before the entire action is completed.

The reason ZFS is resilient and immune to such low-level failures is because it maintains a journal, which allows it to ensure that "2-phase write" (i.e., an atomic update to disparate devices) either completes as an atomic operation, or ignore it as if it had never happened. Variable stripe size has nothing to do with it--you could do the same with fixed stripe sizes (and is essentially what decent BBU RAID-5 controllers do, only their "journal" is in BBU RAM on the controller).

Which is why, for best performance, you want a separate device to hold the ZFS journal/log (aka, a "logzilla").

 

[sub mesa]

As Jeff Bonwick, lead developer of ZFS, stated:

RAID-Z is a data/parity scheme like RAID-5, but it uses dynamic stripe width. Every block is its own RAID-Z stripe, regardless of blocksize. This means that every RAID-Z write is a full-stripe write. This, when combined with the copy-on-write transactional semantics of ZFS, completely eliminates the RAID write hole. RAID-Z is also faster than traditional RAID because it never has to do read-modify-write.

Whoa, whoa, whoa -- that's it? Variable stripe width? Geez, that seems pretty obvious. If it's such a good idea, why doesn't everybody do it?

The use of variable stripe sizes means RAID-Z doesn't have to do 2-phase writes (called "read-modify-write" by Jeff). Instead, by adjusting the size of the stripe so a full stripe block is exactly the size of the data you want to write, you get a "perfect fit" in which RAID5 can write very fast - so called 1-phase writes (called "full-stripe write" by Jeff).

As for journaling, that is something else. No atomic writes means you need some safeguard against dirty buffers and synchronisation between data and meta data. Filesystems can do this in various ways, and i wouldn't say ZFS does journaling. Since it uses a Copy-On-Write model, it simply writes to empty space when you overwrite something. If the system crashes meantime, it would appear as if the write never happened. Such systems differ from classic journaling, and are employed for many years in FreeBSD and other operating systems, called Soft Updates. Though each have their merits, its not really what you use, but how effective it is in preventing filesystem corruption in the case of system outages when using write-back mechanisms. NTFS fails in that, by the way. ZFS does the trick well provided its close to the disks, and can perform flush commands to the disk members, which you can't if you use some cheap hardware RAID. I haven't tested ZFS that thoroughly myself, however, i'm using it to my satisfaction.

P.S. The answer to Jeff Bonwicks question in the quote above, is that ZFS is basically a filesystem and RAID-engine combined. Only this combination can do dynamic/variable stripe sizes, because you need information from both domains to make this work. ZFS is still pretty unique, and nothing exists that compares well with it. You can read his whole entry somewhere here.

 

[jrst]

NB:

...when combined with the copy-on-write transactional semantics...

Without that it doesn't eliminate the "write hole".

...faster than traditional RAID because it never has to do read-modify-write.

Variable stripe size (always full-stripe write) is a performance benefit; it does not eliminate the write hole.

At the risk of oversimplification below...
■N = number of disks in array - 1.
■data[n] = block of data == stripe-size
■"full-stripe write" == write of stripe-size * N bytes of user data
■operations in the same step can be performed in parallel

A. RAID-5 write(new-data) < full stripe write; e.g., single block:
1. read(old-data); read(old-parity)
2. new-parity = xor(old-data,new-data,old-parity)
3. write(new-data); write(new-parity)

B. RAID-5 write(new-data) == full-stripe write:
1. new-parity = xor(new-data[1],...new-data[N])
2. write(new-data[1]); ...write(new-data[N]); write(new-parity)

C. ZFS write(new-data):
1. T = transaction-start(); new-parity = xor(new-data[1],...new-data[N])
2. allocate(new-data[1..N]+parity)
3. write(new-data[0]); ...write(new-data[N]); write(parity)
4. transaction-end(T)
...
5. commit or roll-back transaction T

Note that ZFS does the same number of writes as a RAID-5 full-strip write--one write to each disk in the array: data[1]...[N] + parity. If any of those writes fail to complete, there's an inconsistency. ZFS prevents that by using a transaction log, not by using variable stripe size.

I also like ZFS a lot , as it does a better job and does it cheaper. And among other things, tt can make very effective use of SSD's as an additional tier for the transaction log (small/fast "logzilla") and as a read cache (larger/slower "readzilla"), which can provide significant performance improvement. IMHO the best use of SSD's today. (See Adam Leventhal's Fishworks blog.)

 

[Guest]

hello,

I have a question: if I create raid matrix on motherboard with ICH9R chipset is it possible to move it then (in the future) to ICH10R chipset, if my mobo dies or something? I'm using WD drives with Raid0 for years but IDE with external controller. Have bought some time ago new mobo with ICH9R and I would like to use builtin raid. What do you think?

Mike

 

[BadCommand]

hello,

I have a question: if I create raid matrix on motherboard with ICH9R chipset is it possible to move it then (in the future) to ICH10R chipset, if my mobo dies or something? I'm using WD drives with Raid0 for years but IDE with external controller. Have bought some time ago new mobo with ICH9R and I would like to use builtin raid. What do you think?

Mike

Yes it should work fine. Intel has done a good job of keeping the ICHxR family fairly portable- especially going up. Going down (due to limitations of raid drive combinations) is another story.

 

[microking4u]

I just wanted to get something straight about the ICH10R.

Can I do this combination on the same chip.

1 - Raid 1 Mirrored Set (2 Drives) Operating System
1 - Raid 5 (3 Drives) Data or Raid 10 (4 Drives) Data

and I can do these both at the same time on the same chip?

Is there a performance hit for doing 2 sets vs 1 set on the same chip?

Thanks,

Ryedog

 

[jrst]

Can I do this combination on the same chip.
1 - Raid 1 Mirrored Set (2 Drives) Operating System
1 - Raid 5 (3 Drives) Data or Raid 10 (4 Drives) Data
and I can do these both at the same time on the same chip?
Ryedog

Yes you can; that's the "matrix" part of the "Intel Matrix RAID". As to the performance hit, I've no empirical evidence, but I'd guess (and only a guess) that the performance hit for multiple arrays vs. a single array is nil.

 

[microking4u]

I was also wondering what the performance would be like on ICH10r with just a Raid 10 with lets say some WD RE3 1TB drives in a single set of 4 drives. I know these drives have dual processors and 32mb cache. But I am wondering about maybe doing the operating system and data on the same set, maybe just partitioning them seperate. Using this configuration for a Desktop or Small Business Server?

Anyone seen benchmarks on such a config?

Thanks!

Ryedog

 

[wolf2]

Hello everyone!!

Alas, problem solved....well kind of I guess. The BSODs have stopped by replacing the 1066 speed memory with a couple of 800 speed chips. The system has been running for close to 3 weeks without a single failure, so I am stoked.

The issue now is that I have dismantled all RAID setup and back to standard SATA drives running independently. Even worse, I am not even running them in AHCI mode

Quite frankly, I am enjoying the stability of the system so I will not be re-installing to setup RAID anytime soon. I will stick to the old fashioned backup approach for a few months until an opportunity presents itself to setup RAID again.

Thanks for all those who have offered ideas in this process to identify the root cause. The interesting issue here is that I have not really discovered how reliable the ICH10R is because of the memory problem! So all this time was spent to track down memory as the Culprit and in the process I ended up removing all RAID configuration!

Anyway, I thought you all would be interested to know.

Wolf2

 

[wuzy]

I know you've already solved your problem, but I'd like to know something.
Were you running your FSB @stock e.g. 266 or 333 and RAM at its rated 533Mhz (as set by 'Auto')? It's a common mistake people make with Intel chipsets.

With Intel chipset memory controller sometimes there are certain FSB:RAM ratios, specific to your particular motherboard that are usually 'flaky'. All except 1:1. The higher that ratio is the more potential for instability. The surest way to use your RAM to its full potential is to clock your FSB up as well to keep that difference to a minimum. Or just run the RAM at lower freq., but tighter timing for around the same performance (real-world, not synthetic). A lowly stock FSB with high RAM freq. gets you nowhere.

Now, about RAID5. I'm sure you know the failing nature with mechanical HDDs and its full consequence. RAID5 is the least I can do besides regular backups to prevent against such failure for archived data with best cost efficiency. Although I use hardware RAID5 now, my short 5month venture with Matrix RAID5 w/ 3x1TB had been a fairly good one. The only thing I can complain about it are obviously slower writes than hardware RAID5 and extremely long rebuild time (19hrs when I tested before deployment), both of which are expected from software RAID5 so not really complaints. As far as reliability goes, it's one of the most solid software RAID solutions I've seen (up there with Linux IMO) compared to shocking ones like nVidia which I wouldn't even touch with a 200ft pole.

 

[charred_water]

Shadowflash,

Thanks for your insightful posts regarding RAID5!

I think I'll switch to RAID1 now.

Question about partitioning.

Is it possible to partition a RAID1 volume?

I have two WD Caviar Green 1TB drives which I would like to configure as RAID1 volume.

I normally partition into OS and DATA partitions to make it easier to image the OS volume (for emergency restore) and backup the data volume.

I'm wondering whether disk utilities like Partition Magic and True Image work with RAID volumes.

Thanks!

 

[stockstradr]

ShadowFlash,

thank you. That is a brilliant, very helpful reply .

If it is possible for the scenario (drive error recovery fallout ) you describe to also affect RAID10, then I believe that just today, one of my WD Caviar black (1TB) drives was just dropped from my RAID10 for the very reasons you outlined above. Your post helped me understand why this problem happened, as I did build the RAID (unknowingly) using the desktop edition version of the WD drives.

My system is RAID10 implemented with the on-board Intel X58/ICH10R Southbridge sitting on ASUS P6T mobo. You would be correct in guessing that I'm another amateur (on computers) who thought I could handle implementing a RAID10 on my home PC.

My strategy (for dealing with my RAID that now has dropped a drive) will be to swap in a new (same model) drive in the array and rebuild the array. They separately connect the "bad" drive as a stand-alone drive that I can then re-format around the bad sector. Then after fixing it, keep that drive as backup, thereby turning it into a "refurbished" drive that I can swap back in when this problem repeats itself with another one of those desktop edition drives.

 

[sub mesa]

Not the TLER story again

1) TLER is available on desktop drives like WD Green, you just have to enable it
2) TLER is meant for guaranteeing uptime in servers
3) TLER cannot magically solve surface errors
4) RAIDs who encounter uncorrectable surface errors will kick the disk out of the array
5) broken arrays should be resilvered with a spare disk
6) for consumer systems who have no spare disks, the RAID engine should be tweaked to allow NOT disconnecting a drive at the earliest sign of trouble, such as possible with any non-Windows based advanced software RAID
7) for serious storage needs, windows should be avoided and a dedicated NAS using advanced OS should be used

I've used desktop class disks all my life in RAID5 and RAID0 configs, and simply setting the timeout to 60 seconds does wonders; it allows the disks to fix surface errors without disconnecting them straight away leaving you with a broken array. As people don't know how to 'glue' them together, sometimes they think all their data is gone. This is one reason RAID should NOT be used to increase reliability for consumer storage; in many cases it actually decreases reliability and consumers would be better off using plain-old backups without RAID instead.

It has also to be said RAID and Windows never was a nice combination. Windows itself doesn't offer any advanced RAID, it leaves this to third-party drives like ICHxR to at least get some decent performance, while this could be native windows technology. It also has to be said Windows has no single reliable filesystem. NTFS is meta-data only light journaling and users have no option to use strong journaling.

 

[pjstar30]

Not the TLER story again

1) TLER is available on desktop drives like WD Green, you just have to enable it

I just wanted to add to the article, for those googling and finding this such as I did, that since the date of the post I've reference above WD has removed the ability to enable TLER in desktop drives. Yes, there is a TLER Utility available on the internet and yes this will allow you to enable TLER on older desktop drives. However, information that has been posted in forums across the internet has confirmed that the new WD desktop drives (including the 1TB WD Black) does NOT have the ability to enable TLER any longer.

 

[bharris291]

ICH10 Raid Failures..! Ive had plenty of them drops the raid then i get the raid back up and working for a while. Then an error wont boot up etc. Then decided to install everything IDE and noticed a little hum. Hard drive clicking for no reason . You could here the head swing back and forth like hen you first power a hard drive up. So i decided to use the Jmicron controller on my Asus Board. What do you know everything worked fine. The whiny noise started when the hard drive would write the platters. The noise would come and go usually got louder and the drives clicked more the longer the computer been on. Called Asus and they have a probblem with the 2 sata controllers ICH10 controllers overheating or running hot. So i reinstalled everything turned the air on in the house. Turned all the fans on in the case on high waited for the house to get to 73 degrees. Then noticed no whine noise. No clicking and now write errors and never dropped a drive from the raid 0 Pair. I first had Seagate 7200.11 drives and there were firmware problems with those drives. I got Seagate to replace them and the new drives the same. All this time i thought the drives were bad or wierd. It was the ICH10 southbridge getting hot. So i ended up replacing the seagates with Western Digitals. There apparently is no real fix for this except keep the house cool or replace the board. Asus Rampage ii X58 going to be replaced with a 1366 x68 chipset board like the sniper or gene.