Backup solutions

[Per E. Johannessen]

Looking for a backup solution that after an inital full backup can be configured to only back up new and
changed files in futural back ups. (Backing up to an external SATA or USB3, both JFS.)
I have PM Backup Suite but cannot find any setting for backing up only new or changed files.
Any recommendtions?

[Dariusz Piatkowski]

Hi Per,

Well, the first thing that pops to mind is RSYNC...have you looked at that?

I had previously used dSync as well, but RSYNC is just more feature rich and being a Unix world creation has plenty of documentation and setup examples out there.

In fact, I have developed a RSYNC based REXX solution that I use here to backup both LOCAL and REMOTE devices. If you are comfortable with REXX and wish to pursue the RSYNC route let me know and I'll happily share the script.

[Per E. Johannessen]

Thanks Dariusz,
yes, I had a quick look at RSYNC but before testing I'd like to know more about what is being used these
days by others. At least I'm not uncomfortable with REXX and would like to see the script.   

[Dariusz Piatkowski]

...At least I'm not uncomfortable with REXX and would like to see the script...

Take a peek at the attached REXX script, re-named to rsync_backup.txt from rsync_backup.cmd per attachment rules.

Anyways, I have several instances of that script scheduled through CRON/2 utility, each one being a call to a stand alone CMD script.

Here is what that looks like for a LOCAL SOURCE & TARGET,

Code: [Select]

[G:\util\misc]type rsync_backup_F_H.cmd
@echo off
rsync_backup.cmd LOCAL "F:/" "H:/"

...and here is what I do to copy a LOCAL directory over to my NAS box:

Code: [Select]

[G:\util\misc]type rsync_backup_AUTOMOTIVE.cmd
@echo off
rsync_backup.cmd REMOTE "G:/DESKTOP/Automotive" "V:/public/documents"

Keep in mind that the NAS box is mapped through NetDrive, so it has a real OS/2 drive letter assigned to it. Of course you don't need to do this, but that's just how I implemented the config on my machine.

Typical LOCAL to LOCAL backup for my full OS/2 volume (boot, all apps and data) takes about 1/2 hr with this RSYNC approach, that's about a 150Gig of data:

==========================================================================
RSYNC started:     10/27/21 - 01:25:05...

SOURCE_FOLDER => G:/
TARGET_FOLDER => I:/

RSYNC CLI =>--archive --delete --progress --stats --whole-file --xattrs --os2-perms --human-readable --quiet  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_LOCAL_backup.log G:/ I:/

...whole bunch of file listing detail snipped here...

2021/10/27 01:51:39 [795] Number of files: 330651
2021/10/27 01:51:39 [795] Number of files transferred: 298
2021/10/27 01:51:39 [795] Total file size: 148.54G bytes
2021/10/27 01:51:39 [795] Total transferred file size: 3.34G bytes
2021/10/27 01:51:39 [795] Literal data: 3.34G bytes
2021/10/27 01:51:39 [795] Matched data: 0 bytes
2021/10/27 01:51:39 [795] File list size: 30.11M
2021/10/27 01:51:39 [795] File list generation time: 0.060 seconds
2021/10/27 01:51:39 [795] File list transfer time: 0.000 seconds
2021/10/27 01:51:39 [795] Total bytes sent: 3.37G
2021/10/27 01:51:39 [795] Total bytes received: 29.69K
2021/10/27 01:51:39 [795] sent 3.37G bytes  received 29.69K bytes  2.11M bytes/sec  elapsed 00:26:35
2021/10/27 01:51:39 [795] total size is 148.54G  speedup is 44.08
-----------------------------------------

RSYNC completed:   10/27/21 - 01:51:39...

RSYNC took 1594.320000 seconds to complete the LOCAL backup task!
==========================================================================

The various RSYNC options are controlled through the REXX script itself, just an array of 15 parameters that you can set to ON/OFF, but my config for the LOCAL copy is to do a FULL file as opposed to the incremental changes. I figured these are volumes on two local disks, so plenty of pipe to move whole files.

You can of course exclude certain files or directories, the script provides that option and you simply list those in the EXCLUDE file (which I do as well).

[Dave Yeo]

I simply use a script containing,

Code: [Select]

rsync.exe -a --links --stats --progress --no-perms --delete --log-file=C:/rsync/C_USB.LOG C:/ j:/cdrive/
simple and works. J: is a USB drive

[Doug Bissett]

I ZIP whole drives, then RSYNC those ZIP files to other places (NAS, USB). A typical ZIP command is:

Code: [Select]

if exist CDRIVE.ZIP DEL CDRIVE.ZIP
zip.exe -9yrS CDRIVE.ZIP C:\* -x@exclude.list 2>&1 | tee.exe CDRIVE.LOGThe file exclude.list contains a list of files that I don't need to back up. The ZIP file, and the log file, go into the current (Working) directory.

[Andi B.]

LuckyBackup (frontend for rsync) for syncing to other backup drives. Sometimes I even use the 'keep the last xx changed file versions' LuckyBackup offers.

RAR for daily backup of changed files. With RAR there is no problem with >2GB files and I can split the backup files to <2GB chunks. I've packed the parameters in a .cmd (including attrib -a -s) and use the "Terminplaner" properties tab (Scheduler? don't know how it is called in EN versions). Think this comes with Rich Drag Text package. 

ZIP usually for backing up the boot partitions. ZIP is way faster than RAR and my boot partitions are always <2GB.

Edit: Here is a sample rar command line which creates a backup file of changed files named with the current date and excluding some unnecessary files

Code: [Select]

REM -ac clear achive bit
REM -ag Generate archive name using the current date
REM -ao Add files with Archive attribute set

mode 80,100

o:
cd \Mufasa\TäglichesBackup
rar a -r -ao -ac -m1 -v2000m -ag+YYYYMMDD-NN -x\delete\* -xE:\Seamonkey\Andreas2\mozilla-media-cache\* -xE:\Mozilla\Andreas\dtadmfoc.slt\Cache\* -xE:\Seamonkey\Andreas\zddd4y2q.slt\OfflineCache\* -xE:\Mozilla\Andreas\dtadmfoc.slt\Cache.Trash\* -xE:\Seamonkey\Andreas\zddd4y2q.slt\startupCache\* _E_AutoBak e:\* 2>AutoBackup_E_error.log 1>AutoBackup_E.log


[Doug Bissett]

ZIP does files greater than 2 GB (in and out), as long as you use the ZIP/UNZIP supplied by RPM/YUM (my biggest ZIP backup file is 27 GB). I did try RAR, years ago, and found that ZIP is more user friendly (and does get updated occasionally). It is important to use the "y" parameter, so it handles symlinks properly, and the "S" parameter for hidden, and system, files. No "special" handling is required for either boot drives (even when you are booted from them), or data drives. The same command does work properly. You can also split it into more than one file, if you want to (the -s <size> parameter). I use the RAMDISK as a temporary storage place, while it is ZIPing, if there is enough room (the -b <temp location> parameter). That eliminates a lot of seek time. It still needs to copy the final result to the final location, but it is faster overall (especially when using spinning disks, but even with a SSD, it is a little faster - NVME can be a little faster than a RAMDISK). I do multiple ZIPs, at the same time (up to 6 of them). Some planning is required, for that to work with minimal interference, but it is always faster than doing them individually.

To restore a drive, or just individual files, I use the Arca Noae Archive Tool (it is also called ZIPPY, I think). For a full drive restore, I format it first, but the tool offers various ways to do a restore, while keeping existing files. I have done restore many times (even for multiple drives, at the same time, and usually not for a failure), with no problems. You still need to manage your backups, and you will still have problems if bad files get into your backups (with no other, older, backup to recover them from), but that has nothing to do with using ZIP as a backup tool.

I gave up trying to manage incremental, and differential, backups. It is always a case of "get it done now", and trying to figure out what needs to be restored, from multiple files, takes a lot longer than just doing a full restore. Of course, I also have a nightly backup (scheduled by the Drag Text Schedule feature), using RSync, where I can recover files that were changed after the full ZIP backup was done. UNZIP can be done using the ArcaOS installer System Management tools, if you don't have another operational (or partly operational) boot system.

[Mo]

There has been a discussion for another rsync based solution. Unfortunately the links in the postings seem to be broken.

https://www.os2world.com/forum/index.php?topic=1119.0

Bests