Need to Read/Write Massive Amounts of Log Files

Oct 6, 2018
5
0
10
0
Trying to design a storage solution for some custom logs files on a desktop PC. Both read and write speeds are important, but 70% of operations are write.

Considering:
- Areca ARC raid card with 4x4TB SATA drives in RAID 0
- Connecting 4x4TB SATA drives directly to the motherboard and letting my program rotate through drives letters to decrease load across each drive.

What would you do?
 

kanewolf

Titan
Moderator
How MANY log files? Are you creating new files every hour, day, etc or rotating by size? You can't deal with text files that a GB in size. That means some kind of active/archive implementation. I think you should look at high bandwidth for 2x max active size and then something that moves "yesterday's" log file to larger long term storage.
 
Oct 6, 2018
5
0
10
0


My software already has the timing and size handled properly. These aren't normal log files, they include data dumps like screenshots in base64 format. I just need advice on the hardware.
 

kanewolf

Titan
Moderator


But that sizing impacts the hardware. You don't want to pay for high bandwidth space for non-realtime I/O. What I was going to suggest was an Optane PCIe disk for your high I/O high write endurance. BUT it depends on sizing.
 
Oct 6, 2018
5
0
10
0
Gotcha. The log files range in size of 5 MB to 50 MB. They get purged in different intervals, it gets complicated, depends on usage and if a log data file get scheduled for review. So how many files really depends on if the disks need space or not.

 
Oct 6, 2018
5
0
10
0
Sorry, I thought SATA was widely accepted to mean HDD. I should clarify - I can't afford SSDs need to stick with disk drives.
 

ASK THE COMMUNITY

TRENDING THREADS