Photographer here - looking for the best storage solution for large amount of files.

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680
Hi all, I'm a photographer by trade, and as such, have terabytes upon terabytes of files that I need to keep stored and backed up on a regular basis. I am using a desktop PC, not a MAC, as my main editing computer.

I'm wondering what you think is the best solution for this? I have been doing some research, and it seems a RAID is popular, but I hear a lot of people saying this isn't ideal for a variety of reasons, and a lot of people having errors and issues with them. The other option I was thinking of, is getting maybe 6 of the following:

https://www.bhphotovideo.com/c/product/1268951-REG/seagate_steb8000100_8tb_expansion_desktop.html

This would be a more simple and cheaper method than a RAID by the looks of it, and I can just regularly back up my images (3 hard drives plugged into my PC as main storage + 1 backup drive for each main storage drive).

The other main thing to keep in mind is that I would be editing these images from these hard drives. The editing software is stored on my main SSD drive, but the images would be stored on the external hard drives or RAID.

I would really appreciate any feedback as I'm a bit lost right now.
 
I don't think RAID is your best option. For example if you edit and save a file and later realize that you don't like the results, with RAID, the original file is gone, unless you backed up the original to somewhere else. Or if you mistakenly delete a file, it is gone with RAID.

RAID allows you to continue working in the event one of the drives fails, and allows you to replace the drive and let the RAID array rebuild itself, but isn't really a backup solution. More useful for a business server that can't afford any down time during the work day.

You say you will be editing the images on the external hard drives. Does that mean the original unedited images will be stored on your computer's internal drives? In that case the "backup" would be on your computer's drive(s).
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


Thank you for this information. No, the original, unedited files would be for example on one of those external storage solutions I linked above and plugged into my PC. For each one of these drives that I have, I would have a corresponding "backup drive" that would not be connected to the PC except for when I'm conducting the backup.

I am definitely open to other suggestions that you think may be better.

Also, for clarification, I never replace the original files. New files are created when the editing process begins and are saved separately so I can always go back and re-edit the original file if needed.
 
A true backup would require multiple copies of a file. If you use your computer as a holding place(not a backup) for your unedited pictures, you would still want a backup solution for those files - somewhere other than that computer, and maybe even off-site. Get an additional external drive and backup up to it, and keep it unplugged and somewhere else except when you are actually backing up.

Raid is different than backup. Raid has it's uses, but it is mainly a way to keep operating during a hdd failure - not for backup.
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


Thanks for the input. Sounds like RAID isn't the way to go. Yes, I would also have separate cold storage drives in a safety deposit box. For my needs, would you just suggest I purchase hard drives with as high as possible RPM that I can afford and put them into an individual hard drive enclosure rather than have them all just stacked on top of each other and taking up so many USB slots?
 
Here is my backup procedure for my ProTools session (audio) files. Originally the audio tracks are recorded to my "D" drive of my audio PC. At the end of each day, I plug in an external USB drive and copy the sessions that I worked on during the day. (No need to use backup software if you are organized and can keep track of what you worked on).

Then I copy the session files over my home network to my "non-audio" computer's "D" drive. So at this point I have the original and 2 backups of the session files. Say the next day I do some editing on a few of the sessions, and I backup to the external drive, overwriting the previous versions, but I don't backup to the drive on my non-audio pc, so that copy contains the original unedited audio files. Only when I am 100% sure that the song is completely finished would I overwrite the files on the non-audio pc.

A couple of times this method has let me down, because I mistakenly deleted some files in the working session on my audio pc (doing clean-up after the song was done to free up space on the drive), and then backed up to the external and non-audio pc, so the original session file didn't exist anymore. Luckily I had already mixed the song down to stereo and saved that file to a different place where I store the finished tracks, but I lost the ability to do any further editing on the missing tracks. In my case, I could have recorded the missing tracks again, if necessary, but I don't think that would work with photos.

The main thing is to have a backup plan, execute it consistently, and keep very organized. I don't know how long you have to keep your raw photo files after you finish your work and deliver the final product to your client, but that may impact your backup plan. Some even advise keeping a second backup of everything off site, in the event your office burns down, floods, gets burglarized, etc.

Edit: I didn't see your last response while I was typing, but yes, I would go with an external multiple hard drive bay like this one:

https://www.amazon.com/Mediasonic-ProBox-HF2-SU3S2-SATA-Enclosure/dp/B003X26VV4/ref=sr_1_3?ie=UTF8&qid=1517432933&sr=8-3&keywords=3+bay+hard+drive+enclosure

I am not recommending that specific product, it's just an example. 7,200 rpm HDDs like WD Blacks would be a good choice for the drives.
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


Really appreciate your assistance, it's helped a lot. Ok, looks like I'm going with 3.5" drives and an enclosure. Do you know if it's possible for a hard drive enclosure to cause damage to a hard drive? The reason I ask, is I'm considering going 2 X 4 bay enclosures and keep the original on enclosure A, with its corresponding backup on enclosure B in case of a malfunction with the enclosure. Otherwise, I may as well get 1 X 8 bay enclosure.
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


Really appreciate your assistance, it's helped a lot. Ok, looks like I'm going with 3.5" drives and an enclosure. Do you know if it's possible for a hard drive enclosure to cause damage to a hard drive? The reason I ask, is I'm considering going 2 X 4 bay enclosures and keep the original on enclosure A, with its corresponding backup on enclosure B in case of a malfunction with the enclosure. Otherwise, I may as well get 1 X 8 bay enclosure.
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


Really appreciate your assistance, it's helped a lot. Ok, looks like I'm going with 3.5" drives and an enclosure. Do you know if it's possible for a hard drive enclosure to cause damage to a hard drive? The reason I ask, is I'm considering going 2 X 4 bay enclosures and keep the original on enclosure A, with its corresponding backup on enclosure B in case of a malfunction with the enclosure. Otherwise, I may as well get 1 X 8 bay enclosure and save the hassle of 2 separate units.
 
My storage system is a custom NAS (running FreeNAS) with 4x4TB drives. Basically a PC in a small case with 4 HDDs and one SSD. FreeNAS supports ZFS, which is kinda like RAID except it does redundancy at the file level instead of at the drive level. This eliminates the dreaded problem where your RAID array drops a disk because of a single read error, then drops another disk during rebuild because of another read/write error thus destroying the array. With ZFS, only the file which experienced the error(s) is marked as bad. Every 35 days, it'll also check every file on the NAS against its checksum, and use the redundancy data to repair bit rot errors (where a bit in the file flips due to cosmic rays or some other random occurrence). It's got a bunch of other nifty features including compression, snapshots, overprovisioning. I'd say its biggest drawback is that it's difficult to enlarge your storage size by adding more drives or swapping in larger drives. But that isn't easy with RAID either.

For best effectiveness FreeNAS needs a fairly beefy processor (go for an Intel or AMD processor, not ARM), ECC RAM (which means a server motherboard and limited choice of CPUs), and 4+ GB of RAM. FreeNAS actually recommends 8GB, but you won't need that much if you turn off deduplication, which is a CPU hog and of little use for photos so you'll want to turn it off anyway.

I run Lightroom on my main computer. It supports photo libraries stored on network drives. The preview for all these photos (used during editing) is stored on the local drive so NAS speed is only issue during the final export. When I get back from a photo shoot, I dump the photos into a temp folder on my computer's main drive (SSD). Then import them into Lightroom. Since it's a SSD, the indexing and preview generation happens quickly. I usually do my editing while it's on the SSD as well. When I'm finished and ready to archive the photos, I simply move the album from the temp folder to my NAS via Lightroom. From that point on, Lightroom still shows the album as part of its library, but it knows the album photos are now stored on the NAS.

The nice thing about Lightroom is that instead of saving the changed photo, it only saves your editing steps. So all the work I've done on a photo only results in a small file a few kB in size. Occasionally I have a photo where Lightroom isn't enough. In that case I'll export it to TIFF and edit it in Photoshop, which does generate new photo files. But Adobe has done a great job adding the most common photo editing tools to Lightroom. I only have to fire up Photoshop every 200 photos or so.

I backup my NAS to a Sans Digital RAID enclosure with a bunch of old drives (I think right now it's 4GB + 3x3GB). The enclosure has a JBOD mode where all the drives show up as a single virtual drive, but data is stored sequentially (no RAID going on) so I get the full capacity of all the disks. That's really all you need. Yes I'll lose files if one of the drives dies. But this is the backup for the NAS, so presumably I'll still have the file on my NAS.
 

USAFRet

Titan
Moderator
Solution

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


Thanks for the suggestion. You guys have been incredibly helpful. My budget is preferably under $1500 for about 32tb (16 main/16 backup). I can actually get an enclosure and 4 x 12tb Barracuda Pro HDDs for under $1500. What would the advantage of something like this be vs just getting a hard drive enclosure and stuffing 4 x 8tb Seagate Barracuda Pro 3.5" drives in?
 

USAFRet

Titan
Moderator


A dedicated NAS box is much more robust than just a simple drive enclosure.
Either something you build with FreeNAS as the OS, or something like that QNAP (or Synology).
It is its own PC, with its own Linux based OS.

Low power consumption, stable, lots of features.
I've had zero problems with mine. Coming up on a year of 24/7 operation.
It holds the backups from the house PC's, music and movie lib.

It could just as easily be the default storage location for photo work.
 
I would suggest hiring a professional. In a professional setting any "online" storage level will likely be in a raid. You need someone to pick out your hardware/software and who can come fix it. Below is a handy link for backup drive failures from a large backup provider.

If you really want to DIY go with a QNAP or Synology NAS box with some enterprise storage drives.
Buying drives is fairly complicated. Sites are filled with recerts, "white label". After you pick a model to go with make sure you know what you are buying. I use recerts in a non-professional manner, they are inexpensive, but you have to deal with DOA and risking the extra failure rates.

https://www.backblaze.com/blog/hard-drive-failure-rates-q1-2017/
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680




What do you think of this?

https://www.amazon.com/Synology-DiskStation-DS1817-8GB-Diskless/dp/B06Y4TJL54/ref=sr_1_2?ie=UTF8&qid=1517438840&sr=8-2&keywords=Synology+DiskStation+DS1817

Coupled for now with 4 x 8tb Seagate Barracuda Pro HDDs. This way, using a NAS, I can access my files remotely when I'm overseas which I couldn't do with just HDDs and an enclosure.
 

USAFRet

Titan
Moderator

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680


One thing I'm failing to understand is that shouldn't an enclosure with direct USB 3 access to the hard drives be quicker than remote access on a NAS? I can't have my system slowing down while editing my photos.

If this is the case, am I essentially paying more for different accessibility options, at the expense of slower read speeds?
 

USAFRet

Titan
Moderator


I'll do some performance tests later tonight with RAW pics from my Fuji X-T1. See what specific differences there are.
 

matthewh133

Distinguished
Jun 20, 2009
211
0
18,680

Thank you, I'll be interested to see your results. My files are from a 42mp Sony A7R III, both raw files and 4k video files so I can't have slow drive speeds.
 

USAFRet

Titan
Moderator


Short answer:
The NAS was faster than the USB HDD. Slower than the internal SSD.

Long answer:
Camera - Fuji X-T1 16mp
RAW files
PC - specs in the avatar to the left
NAS - QNAP TS-453a, 4 x 4TB Seagate ronwolf, RAID 5.

NAS, Internal SSD, USB 3.0 3TB Seagate HDD
Lightroom 5.7

9 individual images each run
31MB each image

Import into LR:
NAS - 19.1 sec
Internal SSD - 17 Sec
USB HDD - 21 sec

Editing:
Contrast +40
Scale 125
Rotate 10.0

No perceptable difference. This all happens in RAM anyway.

Export to disk:
Exporting TIFF files, to original dimensions of 4896 x 3264
Resulting TIFF file size 91.4MB for each.

Single image export
NAS - 5.6 sec
Internal SSD - 3.8 sec
USB 3.0 HDD - 6.8 sec

-------------------------
EDIT: I just realized...the NAS box was in the middle of its weekly update to its external space while I was doing this test...:)
smhYgsl.png
 

It depends how you want to be able to access your photo library. I'm usually on a laptop so hate having to plug things into it. So having the files available over the network is a lot more convenient since I can access them over wireless. My WiFi speed is about 35 MB/s, which is good enough most of the time. If I'm doing a big batch job and want faster speed, I'll plug in an ethernet cable (100-125 MBs). Unless you've got SSDs or one of the newer big HDDs (which can top 200 MB/s), USB isn't going to be much faster. And like I said, because I use Lightroom, all the editing is done on previews of the photos. The RAW files are only accessed when I initially create the album (when the files are on my SSD anyway), and when I'm exporting an album that I've archived. The latter operation isn't interactive - I can just start it, then go browse the web or something.

The NAS is also better if you want multiple computers to be able to access your files simultaneously. And too many times friends and clients have asked me to recover files from an external HDD which "just stopped working" when they unplugged the drive. Usually it ends up being the partition table somehow got corrupted when the USB cable was unplugged. USB just isn't as robust or safe as a network share in my experience.

But if you're the only person who needs to access the photos, and the USB enclosure will never be disconnected, then yeah you could use a USB RAID enclosure instead of a NAS.
 

SoNic67

Distinguished
The work drive/enclosure HAS to be a hardware RAID 5 (or 6). RAID is better than just a bunch of drivers, especially now with the quality of drives being so iffy. That would limit the usable space to n-1 drives for RAID 5 and n-2 drives for RAID 6.
I am using this solution for 10 years now, with one 3ware controller, and three drives in RAID 5, I have upgraded my drives recently to gain more space. I have bought a spare controller and spare cable too. The controller can accept up to 12 drives (on SAS) internal or external.

Then you need a cold storage where you keep the backups - it can be an attached external drive. Backup can be done manually or automatically, an e-SATA or USB 3.0 connected drive is a must IMO. USB 2.0 at this level of backup data it's a joke.

For editing photos - someone was hilarious above! Obviously he never used a professional editing software, he would know that you never edit the RAW files, you always save in a intermediary different format, with layers and "undo" levels, and then the finished product is saved in a different folder. You don't manage the work flow with the back-ups!

As a note, the drives need to be the top of the line, today even Seagate and WD have many quality levels and you don't want the cheapest ones. I never had a "rebuild" issue on my RAID using drives dedicated for non-stop use in a NAS/RAID.
 

USAFRet

Titan
Moderator


That's precisely what Lightroom is for. Nondestructive work on the base RAW file.
Then export to the format of your choice, for further editing in photoshop, PSP, or the editor of choice.