Looking for advice...

riccochet

Distinguished
Aug 6, 2006
61
0
18,630
Hey everyone,
First off, i apologize if this question isn't suited to this forum but i have been searching around and really have no idea where to ask about this.
The conundrum is this. I have a file server thats about 10TB and growing, filled with... pretty much everything over the last few years... music, videos, whatever. Its on a windows home server 2011, with a drobo pro connected via iSCSI... i'm starting to consider what i might be losing if i lost a couple hard drives (i have it in single disk redundancy - essentially drobo version of RAID 5). What my friend and i thought is that it would be nice if there was someway to do a remote backup of this to a system and each other's house. Now we've been trying to come up with the best way to do this but i haven't really been able to come up with any specific products that might be able to do the trick.
We did stumble on something called "Transporter", which would have done the trick if it wasn't an appliance that would only sync files that are on its local storage. (http://filetransporterstore.com/)
What we want is pretty simple, to use our existing hardware, and get an appliance or software solution that will let us sync our two systems in a reliable and relatively secure manner. If its an appliance then i would like it to use some kind of vpn like encryption... If its a software solution then we can get a cheap VPN appliance for that purpose.
I considered that something like this would be great with DFS, but getting a windows server (2003 or 2008) seems a bit excessive. I believe it also means we would have to setup a domain between our 2 locations, but i'm not really an expert in DFS. My buddy suggested just doing a cron job of rsync over NFS, over a VPN but that just seems so... ghetto. 😛
Has anyone ever heard or seen (or even done) a solution to an issue like this. Obviously this is something that is pretty normal for enterprise level networks, but i've never seen or heard of anything for personal use.
We did consider the regular cloud solutions but the amount of space we would need make that not very feasible.
Also, if i really shouldn't be in this forum with this question, please let me know.
 
Solution
There are so many ways to do this. Hard to say which works best. I normally do not link wiki pages but this is one of the better ones. It is somewhat limited on the commercial software but it may be omitting software that does many more things one of which is file replication.

http://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software

VPN is pretty easy either with router feature or with software based vpn. The key issue you have to watch out for is that you want the lan subnets at the 2 location to be different. They can be the same it just gets kinda complex in some cases.

I suspect you are going to be limited by your ISP upload bandwidth so it is not going to take much of a machine to run both the sync...
There are so many ways to do this. Hard to say which works best. I normally do not link wiki pages but this is one of the better ones. It is somewhat limited on the commercial software but it may be omitting software that does many more things one of which is file replication.

http://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software

VPN is pretty easy either with router feature or with software based vpn. The key issue you have to watch out for is that you want the lan subnets at the 2 location to be different. They can be the same it just gets kinda complex in some cases.

I suspect you are going to be limited by your ISP upload bandwidth so it is not going to take much of a machine to run both the sync software and the VPN.

I suspect the initial sync is going to be a little painful with that much data, will likely take many days.
 
Solution
We sync over 30TB nightly - but differences only. Still takes 5-6 hours over a dedicated 32Mbit connection
For the initial sync we put the two units in one location... and have an 8TB portable devices for when there is a big data refresh. This portable unit was bought after having a sync run for nearly two weeks and crippling everything else running on the link...
 
Urban, is that in an enterprise level setting? We won't have nearly as much incremental changes than an enterprise business will. I don't even really expect there to be more than a few GB of changes every week.
Thanks for the info John, I am going to search through this to see if we can find what might be the best solution for us. The initial sync we're going to do locally so that really shouldn't be a problem. We're also planning on upgrading both our internet connections to a 250mbit connection to help with that.
 
Yep, enterprise kit, part that my day job - two NetApp filers...
if it's ohnly a few GB then the issues will not be data volume, but capturing and managing the differences - on high volumes of data, many of the home tools run into issues trying to work out what has changed...
Good luck