Question What storage solution or a data intense hobby project?

Jan 5, 2020

for a hobby project (limited budget that is ;) ), I'm in need for a storage solution.
The project is about archiving and analyzing websites.
Every month, I'm generating around 100GB of zipped files which I need to analyze two times soon after downloading and again later on, maybe.

At the moment, I use my Laptop for that and then, manually move the archived files to two external 2.5 HDD that I got out of old laptops (2 times for having a backup).

I'm fine with a slow transfer speed
The storage doesn't have to be online all the time
I would like to be able to scale easily

I thought about buying 4 - 8 500GB 2nd hand drives and plugging them into an existing tower and installing FreeNAS and switch the tower on on demand - mostly for learning sth about FreeNAS and using a pool of disks.
Alternatively, the price per GB for 2nd hand 4TB is only a bit higher and I'd need only two disks which would make things probably easier.

What would be your suggestion for a non critical / hobby / learn project?
What would be your suggestion, if that project would scale up and even generate some money later on?


FreeNAS is an excellent choice if you have the time and patience, although I would buy new drives. If you take your time and shop around you can usually find some pretty good deals. I would start with at least 3 drive so that you can use at least single parity (z1), which is similar to RAID 5 in conventional adapter based arrays.

If you don't have the time or patience for FreeNAS look at something like an Adaptec 6805T/. I buy most of my adapters out of Hong Kong brand new for very cheap prices and they are the real thing, like THIS new one for around $30. You will then need one or two fanout cables (SFF-8087 to 4 SATA or SAS connectors depending on the drives you buy, most likely SATA). Each cable can handle 4 drives so you could start with one. I buy those out of China too for about $20 each, you just need to be sure to buy the correct one. Then just add 3 or 4 preferably identical drives and with some reading build the array. Any old parts with a free PCIe x8 or x16 running at x8 usually using internal graphics and a little ram and a small old SSD for the OS is sufficient. I use real old stuff for that usually.

And then there is the easiest solution -- buy a prebuilt NAS box, but there is not much learning experience there and it will cost more. Still quite viable.

While I do some of this for real money, I do it for a hobby at home and my latest build a week ago that is still loading data was an Adaptec 8805 with 8 12TB SAS WD enterprise drives. The total cost was $2300 and a few spare old parts for 66TB of dual parity storage. I spent a few months shopping to get everything as cheap as possible. My 4th home array as I am a real backup kind of guy with cold storage of everything. So be careful -- it can become an expensive hobby. :)
Reactions: Chris8080