[SOLVED] Awful Megaraid 9260-8i performance

[Murissokah]

I'm at my wit's end with a Raid setup, help from seasoned admins is very welcome.

Short description of the issue: Raid 10 array of 4x8TB Sata 7.2k disks performs at about 10MB/s for sustained sequential writes. Cachecade enabled for reads and writes on 2x 240GB SSD.

The setup:

Windows Server 2019
LSI Megaraid 9260-8i (512MB cache), with BBU and CacheCade key
4x Hitachi Ultrastar He8 HUH728080ALE600
2x Lexar NQ100 240GB SSD (cachecade)
1x Corsair Force MP510 2TB NVMe SSD (system disk)

VD settings:
Read Policy: No Read Ahead
Write Policy: Write Back with BBU
IO Policy: Cached IO
Access Policy: R/W
Disk Cache Policy: Disabled (tried both)
Background Init.: Enabled

LSI drivers are the most current I could find (6.714.18.0, from october 2018).

Other stuff:
Asus Crosshair VIII Formula
Ryzen 9 5900x
4x Corsair 8GB DDR4-3200

The system has no issues otherwise, and no A/V is installed (Windows Defender disabled for the test)

More elaborate description:

This Raid setup is mostly used for a home media library, but also for backup and lesser priority apps that would take too much space on the NVMe. The cachecade key and SSDs were added later in an effort to improve the performance, but they don't seem to change anything at all.

It took me some time to see the issue because smaller transfers perform incredibily fast (~1GB/s), up to about 8GB, after which it stalls to a point a can't do anything that would write to that volume (disk queue gets clogged). In closer inspection I noticed during large writes the system allocates space in RAM for the size of the initial burst (up to 15GB), which explains the quick initial burst. I don't know why it does that, though, and I'm afraid this could lead to data loss. Looking at Resource Monitor after the copy is concluded I can see the it is still running in the background at a much lower speed.

This was initially installed on a previous computer (x99 / xeon 1660v4) and had the same behaviour then. I was suspicious the issue could be with some other component or configuration and expecting the upgrade would fix it, but it didn't.

So my current understanding of the situation is:

1. Windows is somehow caching writes in system memory (naughty windows)
2. Actual write performance is about 10 MB/s
3. CacheCade changes nothing

Windows cache settings (tried enabling the second checkbox too, no change)

The tests:

Notice the actual write speed and disk queue. At this point the volume is stalled.

Here's minutes after I cancelled the transfer, still processing the pending I/O.

 

[drtweak]

First off, Don't show picks of Torrented movies with their file names. People won't help you when they see that or mods may lock the tread.

Second this is a RAID 10, not a RAID5,6,50,60 so turn off Write Back with BBU. Honestly you don't even really need a RAID card that is using cashe or BBU with a RAID 1/0/10 and also that only really helps with smaller files and also this isn't some mission critical server hosting some SQL Database and on top of that becomes pointless if you have a measly 512MB of RAM and copying around files multi gigabytes in size.

Try turning that off and see what happens. When you have WB with BBU turned on everything is going to the 512 RAM first, then to the HDD's. The reason why it goes slow is that the hard drives can only write so fast, but the memory is faster so with large files it will fill up the RAIDS RAM very fast where it can't keep up with the HDD and the speeds drop.