Question Best way to do backups

[Misgar]

A very good point. I hadn't thought about malware that could overwrite a closed Blu-ray disc. I suppose I could use my Combo drive that only supports reading from Blu-ray discs, but malware could still use the longer wavelength DVD laser to wipe existing data. It's a good thing I keep multiple backups on other devices and media, but if a giant meteorite hits the earth and wipes out civilisation like Chicxulub, losing a few files won't be important.

 

[ZugTheDragon]

Both at home and at work, I practice backup routines that is both versioning (similar to incremental backup) and partially offline. The versioning has already saved me a couple of times from data loss caused by software/user errors leading to destroyed/deleted files.

For my own case, having scheduled a backup task is out of the question because it require that the backup location must either be accessible all times, or have some sort of timed service to open access at certain times.
Reason for this is to minimize loss of backup in case of crypto virus.

My solution at work - Windows
I've used the software called FreeFileSync, because it support file versioning. Backup is mainly done once a day, takes a couple of minutes from connecting a usb drive until disconnecting.

Before I used FreeFileSync, I used robocopy - but I soon discovered the weakness that whenever a file got changed in a way that I later regret - after backup was performed, there was no way of getting the original file back.


My solution at home - Linux
I have two backup locations. First backup is just a usb hdd. Secondary backup is another computer in the basement that use an SSH server so that I can use rsync to perform backup. I also do use rsync to backup to the usb hdd.
Versioning is accomplished by a backup parameter in rsync combined by the bash shell ability to use date and time as any other variable.


Why and what is versioning
For a backup scheme where it exist a source and a destination directory for your backup, in addition there will be a third folder for versioning purpose.
I personally tend to name this folder so that imply what backup and that it is for versioning, so for example "versioning_photos" would be pretty common.

Each time a backup is performed, and a file in backup directory is to be replaced, updated or deleted - then the original file in the backup dir - instead of being removed - is moved to the versioning folder, usually in a sub folder that have incremental number or date+time. I prefer date+time scheme.

This way - If I have a project in progress and do something stupid to a document with irreversible changes, I can now easily go back and retrieve an older version of that file, or group of files.


[edit]
Forgot to mention - For both backup at work and at home, I also have off-site backups in case of fire or other all-destroying accidents.

This is what I'd need!
However, having to manually remember to connect the hard drive and remove it constantly sounds like a pain, and straight up impossible sometimes.. as well as leaving plenty of possibility for human error of me forgetting to / not feeling like connecting it.
And how would error correction be performed on an external drive?

 

[ZugTheDragon]

There is no one size fits all method.

Some folks use an image...others a clone...others the cloud or maybe copy a few files to a flash stick.

It's what fits your needs.

I am trying to find something that'd fit my needs, but I am struggling to find something quire right for me.

 

[ZugTheDragon]

What can't be automated?

Except for my End of the World offsite backup drives, my routine IS all automated.
The PCs run a nightly or weekly Full or Incremental, as needs dictate.
All hands off, takes zero seconds out of my day.

"Drives not accessible by anything"
If they're not accessible by anything, how can anything be backed up to them?
And what about having a 2nd lan network?
Another issue, I've discovered that the QNAP device I might get only supports it's own vulnerability ridden OS, which only supports EXT4 (not ZFS), so it'll probably end up using it's own version of RAID which might or might not do error correction, and may not even be able to recover the HDD contents if the QNAP NAS dies

 

[USAFRet]

"Drives not accessible by anything"
If they're not accessible by anything, how can anything be backed up to them?
And what about having a 2nd lan network?
Another issue, I've discovered that the QNAP device I might get only supports it's own vulnerability ridden OS, which only supports EXT4 (not ZFS), so it'll probably end up using it's own version of RAID which might or might not do error correction, and may not even be able to recover the HDD contents if the QNAP NAS dies

"Not accessible" are the 2 drives in a desk drawer at work.
Refreshed quarterly or so.

As for the NAS, the contents of the main volumes get backed up to other drives in a USB connected pod.
These are formatted NTFS, and if I pull them out, accessible from any Windows system. Yes, I've tested this.
Also, access to the drives in that pod are accessible only from a NAS user account. In normal operations, the WIndows PCs can't see them at all.

 

[ZugTheDragon]

"Not accessible" are the 2 drives in a desk drawer at work.
Refreshed quarterly or so.

As for the NAS, the contents of the main volumes get backed up to other drives in a USB connected pod.
These are formatted NTFS, and if I pull them out, accessible from any Windows system. Yes, I've tested this.
Also, access to the drives in that pod are accessible only from a NAS user account. In normal operations, the WIndows PCs can't see them at all.

I don't use windows anyway, but if the NAS uses it's own RAID system I will not be able to read the drives on anything probably
And your solution would require having a whole extra drive
and what about having some of the backups on the 2nd more secure network?

 

[USAFRet]

I don't use windows anyway, but if the NAS uses it's own RAID system I will not be able to read the drives on anything probably
And your solution would require having a whole extra drive
and what about having some of the backups on the 2nd more secure network?

Don't have to use RAID in the NAS.

RAID is for continued uptime, not for backups or data security.

"a whole extra drive" ? Thats pretty much the point of a backup. A second (or third) copy. Somewhere to recover your data from in the case of disaster.

2nd network? Sure. This requires a whole other system and drive set.

 

[ZugTheDragon]

Don't have to use RAID in the NAS.

RAID is for continued uptime, not for backups or data security.

"a whole extra drive" ? Thats pretty much the point of a backup. A second (or third) copy. Somewhere to recover your data from in the case of disaster.

2nd network? Sure. This requires a whole other system and drive set.

I wanted to use RAID as protection against data corruption (which is why I'd prefer to use ZFS, as it's really good at detecting and correcting errors)
I cannot fit all my drives in one system, and would need to split them across several systems anyway
The problem is that I assume I cannot make incremental weekly backups on a server that's on the main network, if the main monthly backup is on the system that's on the secure network?

 

[USAFRet]

I wanted to use RAID as protection against data corruption (which is why I'd prefer to use ZFS, as it's really good at detecting and correcting errors)
I cannot fit all my drives in one system, and would need to split them across several systems anyway
The problem is that I assume I cannot make incremental weekly backups on a server that's on the main network, if the main monthly backup is on the system that's on the secure network?

My layers:

What is on the PC
Incremental backups of all the house PCs to the NAS, nightly or weekly.
NAS backups up to another volume, in a USB connected 4 bay thing.
Offsite backups, manually copied once a quarter or so, and then returned to their offsite location.

The NAS devices:
4 bay QNAP TS-453a. 4 individual drives, no RAID. 8TB, 14TB, 18TB, 480GB SSD (system drive).
4 bay QNAP TR-004, 4x 4TB, JBOD, This accepts a backup of the main data in the NAS, on a schedule
4 bay MediaSonic, USB connected. 4 drives, NTFS. This accepts weekly backups of the NAS data. The drives in here are only accessible via the NAS, unless I physically remove them and connect to some other PC.

There is no single perfect method.
Whatever works for you is far far better than most people, who only think of data recovery 5 minutes after they actually need it.

 

[Grobe]

And how would error correction be performed on an external drive?

This is a good question.

I'd suggest you in some cases may want to use the --checksum argument in order to make rsync compare file content.
Maybe intermediately after a regular sync, and in combination with the --dry-run argument (to prevent the healthy file being overwritten by a corrupted file).

 

[Firestone]

This is how I handle it

- Backblaze running on all systems that I care about (macOS / Windows)
- single file storage server on the network with large RAID1 volume (accessible over network via ssh / SMB)
- all systems that I care about have a basic bash script that runs `rsync -vrthP -e ssh` with a number of exclude/include args to sync the PC/laptop home directory (/Users/myname on macOS, /home/myname on Linux, C://Users/myname on Windows) over to a dedicated location on the network file server's storage volume ; I run this manually when I import data to a system for processing, or whenever I feel like it ; on Windows I use WSL2 for this btw
- network file server (running macOS on a Mac Mini) also has its entire backup volume backed up with Backblaze
- all Mac computers on the network also have Time Machine enabled, backing up to a separate storage volume on the file server

been working pretty well

 

[Firestone]

This is a good question.

I'd suggest you in some cases may want to use the --checksum argument in order to make rsync compare file content.
Maybe intermediately after a regular sync, and in combination with the --dry-run argument (to prevent the healthy file being overwritten by a corrupted file).

This. AFAIK, you only need to run `rsync` twice; the first time with "normal" settings, and the second time with "checksum" enabled.

iirc default rsync would not catch a data corruption if the reciever already has a copy of the file with an appropriate filesize and timestamp, you would need checksum to catch that.

if you are real paranoid and have time to spare I guess you could just run rsync with checksum twice instead.

 

[Firestone]

honestly, a lot of blood and sweat is expended over Linux and BSD storage solutions, but macOS has really been doing just fine for me for a long time now.

- RAID support built in
- native POSIX *nix style terminal interface built in
- Time Machine built in
- Backblaze unlimited backups supported
- solid hardware that I did not have to DIY-myself
- Apple Remote Desktop works well from my other Mac's
- its running macOS so you get all the "it just works" benefits for free
- Mac Mini form factor is fantastic, and power draw for the system itself is about 7W idle, a little over 10W~ish under load
- Docker and virtualization support is a given, though you'll want an Intel model for best compatibility

obviously its not for everyone and has some clear drawbacks (such as lack of internal storage expansion and reliance on external physically attached storage devices over USB/Thunderbolt, no options for ECC mem, no fancy filesystems), but I have yet to find myself in a situation where I actually needed all the extra stuff that comes with DIY Linux solutions or Unraid/TruNas, etc..

just plug in a big reliable external storage device, enable SMB and ssh access, and keep on truckin'

 

[Firestone]

but if a giant meteorite hits the earth and wipes out civilisation like Chicxulub, losing a few files won't be important.

you might be interested in https://en.wikipedia.org/wiki/M-DISC
its been on my to-do list for a while, have not tried it just yet though

 

[Grobe]

This. AFAIK, you only need to run `rsync` twice; the first time with "normal" settings, and the second time with "checksum" enabled.

I was thinking running it with "checksum" enabled some few times a year just to be able to identify files being corrupted.

However - I believe it's more clever to store the data on a file system that has built-in crc check as it's normally faster to do this on local disk than over network. Btrfs is one file system that can do this.
That said - probably file system on both sides should have this ability to avoid doing crc comparison of files over network.

 

[WhatIsAMobo]

First off, kudos for being proactive with this.
Far too many people aren't.

As you note, RAID is not a backup.
A RAID 1 or 5 provides physical drive redundancy, to support continued Uptime. It does little or nothing for actual data security.

The basic concept is 3-2-1
3 copies, on at least 2 different media, at least 1 offsite or otherwise inaccessible.

The first post here is mine, somewhat changed since I wrote this:

What is your backup situation at home?

What is your backup situation at home? And if you don't do that, why not? Every single day, I read multiple threads here of "How do I get my stuff back?" or "That drive had 5 years of photos of my kids!!" Be it a dead drive, dropped phone, virus, accidental deletion, formatting the wrong...

forums.tomshardware.com

Hi!
I know this is not my thread, but I ve got a question: Is just using the backup feature in Windows 10's Settings enough for me to create backups of my data to an external drive? I was thinking about not only buying a new HDD to replace my dying one, but also buy an Ext Drive to backup stuff that matters to me and then keep it in a padded case. Main idea is backing up videos, some games and I ve heard you can save an Image file with your OS at the point where you created that image (wonder which free programs can do that).

Is this a good idea?

 

[USAFRet]

Hi!
I know this is not my thread, but I ve got a question: Is just using the backup feature in Windows 10's Settings enough for me to create backups of my data to an external drive? I was thinking about not only buying a new HDD to replace my dying one, but also buy an Ext Drive to backup stuff that matters to me and then keep it in a padded case. Main idea is backing up videos, some games and I ve heard you can save an Image file with your OS at the point where you created that image (wonder which free programs can do that).

Is this a good idea?

Please start a NEW thread for this. (pm me the link)

The ways and means of backups are varied.