Question RAID 5 - Intel Rapid Storage

[JornT]

Hi!

I've set up a RAID 5 array on my computer which has the following specs:

• ASUS Z370-F motherboard
• i7 8700K CPU
• 32 GB RAM.
• Samsung 970 EVO 500 GB M.2
• nVidia 1070 in the x16 slot
• 2x8 TB drives (my "old" data drives)
• ASUS Hyper x16 in the x8 slot
• 2x2 TB Intel 660p in RAID 0 for projects on the ASUS Hyper
• Dual 10 GbE SFP+ network in the x4 slot
• 4x10 TB WD RED in RAID 5

I use the onboard Intel Rapid Storage RAID, and the array contains four WD RED 10 TB (white label EMAZ drives), making a total of 30 TB available for data.

Read/write speeds were terribly slow, like 80 MB/sec read and 30 MB/sec write, but after doing an "Initialization", which took a whole week (!), I'm now getting about 600 MB/sec read and 400 MB/sec write - that is, in synthetic benchmarks.

In practice, however, write speeds often fall to 30-40 MB/sec and stay there.

Sometimes I get 200-250 MB/sec sustained throughout the transfer, and for a collection of large files (10-15 GB each), I sometimes get 400 MB/sec write.
BUT, very often, I end up with endless transfers at 30-40 MB/sec. This often happens when transferring tens of thousands of files, typically 5-10 MB JPEG images.

Is this the performance I should expect to get, or am I doing something wrong here? I've tested turning on/off the write-back cache.

BTW, I've spent all three of the full-size PCIe slots on the motherboard, so buying a dedicated RAID controller is really not an option...

Regards,
- Jørn

 

[USAFRet]

Transferring 10k small files WILL be slow, RAID or no RAID. There is a lot of overhead in moving each tiny file.

And RAID 5 is increasingly contraindicated as you go larger in drive size.
At 10TB each, you're way over rational size.

Consider a dead drive, and you replace. Rebuilding that RAID 5 will take at least several days, of continual stress on the other 3 drives in the array.

 

[JornT]

The files are not particularly small, i feel that a folder with files 5 MB in size on average, should transfer a lot faster than this, it's less than 10 files a second. Copied from a 3850 MB/sec SSD.

Did a test and copied the same files from a 8 TB drive to a slower external drive and got about twice the speed, 70-80 MB/sec. Copying the same folder to an SSD: 225 MB/sec.

Copying this folder from the SSD to the RAID: starts off at 1000+ MB/sec for a few seconds until the cache is filled, then continues at 25-40 MB/sec. What I just don't understand is that I'm sometimes getting 200-250 MB/second. But those "faster" folders might have contained RAW images instead of JPEG's, making the file sizes around 25 MB instead of 5 MB on average. Need to do some more testing I guess.

Anyone got some real world performance numbers on their "Rapid" Storage RAID 5 array?

 

[SgtScream]

What bios revision is your motherboard on?

 

[JornT]

What bios revision is your motherboard on?

It's on rev.2401 (the newest). Have had a lot of trouble with an external NAS solution as well, so everything has been patched up to the newest revisions. (Except the hard drives, but all drives are brand new from the store)

I've spent like two weeks trying to get things to work, guess I'll start a new thread concerning the NAS. 😵

It may very well be that USAFRet is right about the small file sizes and overhead, that if I were to copy a million 4k files, it would take forever on my RAID. But the four 10 TB RED drives, with write-back cache enabled, still performs slower than a 2,5" external USB drive, which I must say I didn't expect.

More detailed tests with files hand-picked or separated by size, now reveal that I get a 250 MB/sec write on some folders with 45 MB RAW files, almost ten times the speed but also ten times the file size. 10 GB files copy at 350-400 MB/sec.

...Aaaand I just copied a 20 GB folder with 7000 files, 2-3 MB average, got less than 30 MB/sec on that one. Reading it and writing it back to the SSD though, went at 550 MB/sec! So I guess, file creation is a time-consuming operation on a RAID.

 

[USAFRet]

File composition absolutely makes a difference.

Test just completed:
Source - Samsung 860EVO 500GB.
Target - QNAP NAS, 4x4TB Ironwolf, RAID 5.
Across the gigabit LAN, so ~113MB/sec theoretical max.

 

[JornT]

Thanks a lot for the benchmarks USAFRet!

Your drives seem to be rated at 180 MB/s, so about 15% slower than my 10 TB's, but still you're getting more than twice the speed, and that's with a gigabit bottleneck as well. You're getting more or less the same performance on the small files as my external Seagate 8 TB drive which is similarly rated.

So the question is whether this so-called Rapid Storage just isn't rapid at all, or maybe I've done something wrong with the setup. 🙈

I'm using 64k stripes btw, while I found out later that 128k would be better suited for media files.