Need to Read/Write Massive Amounts of Log Files

[cfselen]

Trying to design a storage solution for some custom logs files on a desktop PC. Both read and write speeds are important, but 70% of operations are write.

Considering:
- Areca ARC raid card with 4x4TB SATA drives in RAID 0
- Connecting 4x4TB SATA drives directly to the motherboard and letting my program rotate through drives letters to decrease load across each drive.

What would you do?

 

[kanewolf]

How MANY log files? Are you creating new files every hour, day, etc or rotating by size? You can't deal with text files that a GB in size. That means some kind of active/archive implementation. I think you should look at high bandwidth for 2x max active size and then something that moves "yesterday's" log file to larger long term storage.

 

[cfselen]

How MANY log files? Are you creating new files every hour, day, etc or rotating by size? You can't deal with text files that a GB in size. That means some kind of active/archive implementation. I think you should look at high bandwidth for 2x max active size and then something that moves "yesterday's" log file to larger long term storage.

My software already has the timing and size handled properly. These aren't normal log files, they include data dumps like screenshots in base64 format. I just need advice on the hardware.

 

[kanewolf]

How MANY log files? Are you creating new files every hour, day, etc or rotating by size? You can't deal with text files that a GB in size. That means some kind of active/archive implementation. I think you should look at high bandwidth for 2x max active size and then something that moves "yesterday's" log file to larger long term storage.

My software already has the timing and size handled properly. These aren't normal log files, they include data dumps like screenshots in base64 format. I just need advice on the hardware.

But that sizing impacts the hardware. You don't want to pay for high bandwidth space for non-realtime I/O. What I was going to suggest was an Optane PCIe disk for your high I/O high write endurance. BUT it depends on sizing.

 

[cfselen]

Gotcha. The log files range in size of 5 MB to 50 MB. They get purged in different intervals, it gets complicated, depends on usage and if a log data file get scheduled for review. So how many files really depends on if the disks need space or not.

 

[kanewolf]

SSDs are really expensive for high write endurance. A 1TB Optane SSD is expensive, but not outrageous -- https://www.cdw.com/product/Intel-Optane-SSD-905P-Series-solid-state-drive-960-GB-PCI-Express-3.0/5094564 BUT it has a 41.0 PB write endurance. The Intel P4500 is much cheaper but has a MUCH lower write endurance.

 

[cfselen]

Sorry, I thought SATA was widely accepted to mean HDD. I should clarify - I can't afford SSDs need to stick with disk drives.