Author Topic: RSYNC - local drive backups  (Read 495 times)

Dariusz Piatkowski

  • Hero Member
  • *****
  • Thank You
  • -Given: 14
  • -Receive: 20
  • Posts: 821
  • Karma: +4/-0
    • View Profile
RSYNC - local drive backups
« on: October 20, 2020, 08:10:54 pm »
So I now have about two weeks of nightly backups (both LOCAL dirve and networked NAS box), things seems to be working well. Spent a little time this weekend tuning the exception list to avoid things like Firefox cache, etc.

Now that I'm looking at the LOCAL backups, so disk to disk, I'm wondering how to interpret the metrics RSYNC is reporting.

Case in point  (these logs are a bit lengthy, so scroll around if you need to see the full thing please):

1) this is the boot drive to a backup copy (basically all OS/2 things run on this SSD with JFS FS), but using the '--whole-file' option since I figured in a local disk setting this would be by far the safest thing to do

Code: [Select]
===============================================================================
RSYNC started:     10/20/20 - 11:26:51...

SOURCE_FOLDER => G:/
TARGET_FOLDER => I:/

RSYNC CLI =>--archive --delete --progress --stats --whole-file --xattrs --os2-perms --human-readable --quiet  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_LOCAL_backup.log G:/ I:/

-----------------------------------------
2020/10/20 11:26:51 [420] building file list
2020/10/20 11:27:26 [420] .d..t...... HOME/.config/psi/profiles/Home/
2020/10/20 11:27:26 [420] >f..t.....x HOME/.config/psi/profiles/Home/accounts.xml
2020/10/20 11:27:26 [420] >f..t.....x HOME/.config/psi/profiles/Home/accounts.xml.backup
2020/10/20 11:27:26 [420] >f..t.....x HOME/.config/psi/profiles/Home/options.xml
2020/10/20 11:27:26 [420] >f.st.....x HOME/.config/psi/profiles/Home/options.xml.backup
2020/10/20 11:27:29 [420] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/
2020/10/20 11:27:29 [420] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/SiteSecurityServiceState.txt
2020/10/20 11:27:29 [420] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cookies.sqlite
2020/10/20 11:27:29 [420] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cookies.sqlite-wal
2020/10/20 11:27:30 [420] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/places.sqlite
2020/10/20 11:27:30 [420] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/places.sqlite-wal
2020/10/20 11:27:30 [420] >f..t.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/xulstore.json
2020/10/20 11:27:32 [420] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cache2/
2020/10/20 11:27:32 [420] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cache2/index
2020/10/20 11:27:32 [420] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cache2/doomed/
2020/10/20 11:27:32 [420] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/datareporting/
2020/10/20 11:27:32 [420] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/datareporting/aborted-session-ping
2020/10/20 11:27:34 [420] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/extension-data/
2020/10/20 11:27:35 [420] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/extension-data/ublock0.sqlite
2020/10/20 11:27:37 [420] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/
2020/10/20 11:27:37 [420] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/recovery.bak
2020/10/20 11:27:37 [420] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/recovery.js
2020/10/20 11:27:42 [420] .d..t...... MPTN/ETC/
2020/10/20 11:27:43 [420] .d..t...... OS2/
2020/10/20 11:27:43 [420] >f..t...... OS2/OS2.!!!
2020/10/20 11:27:43 [420] >f.st...... OS2/OS2.INI
2020/10/20 11:27:43 [420] >f..t...... OS2/OS2SYS.!!!
2020/10/20 11:27:43 [420] >f.st...... OS2/OS2SYS.INI
2020/10/20 11:28:34 [420] >f.st...... apps/tcpip/PMMail3/Accounts/logfile.log
2020/10/20 11:28:37 [420] .d..t...... apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/
2020/10/20 11:28:37 [420] *deleting   apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/qii7lc00.msg
2020/10/20 11:28:37 [420] *deleting   apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/qii5ql00.msg
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/FOLDER.UCT
2020/10/20 11:28:37 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/folder.bag
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/folder.ini
2020/10/20 11:28:37 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/folderix.xml
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/folderpr.xml
2020/10/20 11:28:37 [420] .d..t...... apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/FOLDER.UCT
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/folder.ini
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/folderpr.xml
2020/10/20 11:28:37 [420] .d..t...... apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/Union_0.FLD/
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/Union_0.FLD/FOLDER.UCT
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/Union_0.FLD/folder.ini
2020/10/20 11:28:37 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/ePost_0.FLD/Union_0.FLD/folderpr.xml
2020/10/20 11:29:20 [420] .d..t...... apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/dropbox/
2020/10/20 11:29:53 [420] .d..t...... apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/qigkto00.msg
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/qibdr700.msg
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/qi9i4j00.msg
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/FOLDER.UCT
2020/10/20 11:29:53 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/folder.bag
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/folder.ini
2020/10/20 11:29:53 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/folderix.xml
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/INBOX.FLD/folderpr.xml
2020/10/20 11:29:53 [420] .d..t...... apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/TRASH.FLD/
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/TRASH.FLD/FOLDER.UCT
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/TRASH.FLD/folder.bag
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/TRASH.FLD/folder.ini
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/TRASH.FLD/folderix.xml
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/LILLIA0.ACT/TRASH.FLD/folderpr.xml
2020/10/20 11:29:53 [420] .d..t...... apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/qicsmw00.msg
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/qibhb500.msg
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/qib5q900.msg
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/qiaw0204.msg
2020/10/20 11:29:53 [420] *deleting   apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/qiavje00.msg
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/FOLDER.UCT
2020/10/20 11:29:53 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/folder.bag
2020/10/20 11:29:53 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/folder.ini
2020/10/20 11:29:54 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/folderix.xml
2020/10/20 11:29:54 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/INBOX.FLD/folderpr.xml
2020/10/20 11:29:54 [420] .d..t...... apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/TRASH.FLD/
2020/10/20 11:29:54 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/TRASH.FLD/FOLDER.UCT
2020/10/20 11:29:54 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/TRASH.FLD/folder.bag
2020/10/20 11:29:54 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/TRASH.FLD/folder.ini
2020/10/20 11:29:54 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/TRASH.FLD/folderix.xml
2020/10/20 11:29:54 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/PIATKO0.ACT/TRASH.FLD/folderpr.xml
2020/10/20 11:29:54 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/acct.snp
2020/10/20 11:29:54 [420] .d..t...... apps/tcpip/PMMail3/Accounts/gmail_00.act/dropbox/
2020/10/20 11:30:06 [420] .d..t...... apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/
2020/10/20 11:30:06 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/folder.bag
2020/10/20 11:30:06 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/folder.ini
2020/10/20 11:30:06 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/folder.uct
2020/10/20 11:30:07 [420] >f.st.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/folderix.xml
2020/10/20 11:30:07 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/folderpr.xml
2020/10/20 11:30:07 [420] >f+++++++++ apps/tcpip/PMMail3/Accounts/gmail_00.act/inbox.fld/qiiadt00.msg
2020/10/20 11:30:09 [420] .d..t...... apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/
2020/10/20 11:30:09 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/folder.ini
2020/10/20 11:30:09 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/folder.uct
2020/10/20 11:30:09 [420] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/folderpr.xml
2020/10/20 11:38:28 [420] >f..t...... apps/tcpip/web2/counter.dat
2020/10/20 11:38:31 [420] >f.st...... apps/tcpip/web2/web.log
2020/10/20 11:42:17 [420] >f.st...... tmp/log/rsync_LOCAL_backup.log
2020/10/20 11:42:17 [420] >f.st...... tmp/log/smartd.log
2020/10/20 11:46:17 [420] Number of files: 304299
2020/10/20 11:46:17 [420] Number of files transferred: 65
2020/10/20 11:46:17 [420] Total file size: 132.66G bytes
2020/10/20 11:46:17 [420] Total transferred file size: 322.71M bytes
2020/10/20 11:46:17 [420] Literal data: 322.71M bytes
2020/10/20 11:46:17 [420] Matched data: 0 bytes
2020/10/20 11:46:17 [420] File list size: 26.88M
2020/10/20 11:46:17 [420] File list generation time: 0.065 seconds
2020/10/20 11:46:17 [420] File list transfer time: 0.000 seconds
2020/10/20 11:46:17 [420] Total bytes sent: 349.65M
2020/10/20 11:46:17 [420] Total bytes received: 20.14K
2020/10/20 11:46:17 [420] sent 349.65M bytes  received 20.14K bytes  299.76K bytes/sec  elapsed 00:19:26
2020/10/20 11:46:17 [420] total size is 132.66G  speedup is 379.39
-----------------------------------------

RSYNC completed:   10/20/20 - 11:46:17...

RSYNC took 1165.220000 seconds to complete the LOCAL backup task!
===============================================================================

2) and here is the same disk but not using the '--whole-file' option, and therefore relying on the RYSNC algorithm

Code: [Select]
===============================================================================
RSYNC started:     10/20/20 - 12:58:13...

SOURCE_FOLDER => G:/
TARGET_FOLDER => I:/

RSYNC CLI =>--archive --delete --progress --stats --xattrs --os2-perms --human-readable --quiet  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_LOCAL_backup.log G:/ I:/

-----------------------------------------
2020/10/20 12:58:13 [447] building file list
2020/10/20 12:58:48 [447] >f.st...... HOME/.config/ATutils/QE.ini
2020/10/20 12:58:48 [447] .d..t...... HOME/.config/psi/profiles/Home/
2020/10/20 12:58:48 [447] >f..t.....x HOME/.config/psi/profiles/Home/accounts.xml
2020/10/20 12:58:48 [447] >f..t.....x HOME/.config/psi/profiles/Home/accounts.xml.backup
2020/10/20 12:58:48 [447] >f.st.....x HOME/.config/psi/profiles/Home/options.xml
2020/10/20 12:58:48 [447] >f..t.....x HOME/.config/psi/profiles/Home/options.xml.backup
2020/10/20 12:58:51 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/
2020/10/20 12:58:51 [447] >f..t.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/SiteSecurityServiceState.txt
2020/10/20 12:58:51 [447] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cookies.sqlite
2020/10/20 12:58:51 [447] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cookies.sqlite-wal
2020/10/20 12:58:51 [447] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/healthreport.sqlite
2020/10/20 12:58:51 [447] >f.st...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/healthreport.sqlite-wal
2020/10/20 12:58:52 [447] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/places.sqlite
2020/10/20 12:58:52 [447] >f.st...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/places.sqlite-wal
2020/10/20 12:58:52 [447] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/prefs.js
2020/10/20 12:58:52 [447] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/xulstore.json
2020/10/20 12:58:54 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cache2/doomed/
2020/10/20 12:58:54 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/datareporting/
2020/10/20 12:58:54 [447] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/datareporting/aborted-session-ping
2020/10/20 12:58:57 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/extension-data/
2020/10/20 12:58:57 [447] >f.st...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/extension-data/ublock0.sqlite
2020/10/20 12:58:57 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/safebrowsing/
2020/10/20 12:58:59 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/
2020/10/20 12:58:59 [447] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/recovery.bak
2020/10/20 12:58:59 [447] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/recovery.js
2020/10/20 12:58:59 [447] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/thumbnails/
2020/10/20 12:59:00 [447] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/thumbnails/b71d3175a96a4405b46164f93e028d6f.png
2020/10/20 12:59:04 [447] .d..t...... MPTN/ETC/
2020/10/20 12:59:05 [447] .d..t...... OS2/
2020/10/20 12:59:05 [447] >f..t...... OS2/OS2.!!!
2020/10/20 12:59:06 [447] >f.st...... OS2/OS2.INI
2020/10/20 12:59:56 [447] >f.st...... apps/tcpip/PMMail3/Accounts/logfile.log
2020/10/20 12:59:59 [447] .d..t...... apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/
2020/10/20 12:59:59 [447] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/FOLDER.UCT
2020/10/20 12:59:59 [447] >f.st.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/folder.bag
2020/10/20 12:59:59 [447] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/folder.ini
2020/10/20 12:59:59 [447] >f.st.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/folderix.xml
2020/10/20 12:59:59 [447] >f..t.....x apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/folderpr.xml
2020/10/20 12:59:59 [447] >f+++++++++ apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/INBOX.FLD/Car_St0.FLD/qiibzv00.msg
2020/10/20 13:00:41 [447] .d..t...... apps/tcpip/PMMail3/Accounts/DARIUS0.ACT/dropbox/
2020/10/20 13:01:15 [447] .d..t...... apps/tcpip/PMMail3/Accounts/gmail_00.act/
2020/10/20 13:01:15 [447] >f.st.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/acct.snp
2020/10/20 13:01:15 [447] >f.st.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/acctnet.log
2020/10/20 13:01:31 [447] .d..t...... apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/
2020/10/20 13:01:31 [447] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/folder.ini
2020/10/20 13:01:31 [447] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/folder.uct
2020/10/20 13:01:31 [447] >f..t.....x apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/folderpr.xml
2020/10/20 13:01:31 [447] >f+++++++++ apps/tcpip/PMMail3/Accounts/gmail_00.act/trash.fld/trnsient.fld/head0031.hdr
2020/10/20 13:09:47 [447] >f..t...... apps/tcpip/web2/counter.dat
2020/10/20 13:09:50 [447] >f.st...... apps/tcpip/web2/web.log
2020/10/20 13:13:35 [447] >f.st...... tmp/log/rsync_LOCAL_backup.log
2020/10/20 13:13:36 [447] >f.st...... tmp/log/smartd.log
2020/10/20 13:16:55 [447] >f.st.....x util/misc/rsync_backup.cmd
2020/10/20 13:17:03 [447] >f.st...... util/upsmon/upsparm.log
2020/10/20 13:17:35 [447] Number of files: 304301
2020/10/20 13:17:35 [447] Number of files transferred: 40
2020/10/20 13:17:35 [447] Total file size: 132.66G bytes
2020/10/20 13:17:35 [447] Total transferred file size: 307.12M bytes
2020/10/20 13:17:35 [447] Literal data: 307.12M bytes
2020/10/20 13:17:35 [447] Matched data: 0 bytes
2020/10/20 13:17:35 [447] File list size: 26.88M
2020/10/20 13:17:35 [447] File list generation time: 0.062 seconds
2020/10/20 13:17:35 [447] File list transfer time: 0.000 seconds
2020/10/20 13:17:35 [447] Total bytes sent: 334.07M
2020/10/20 13:17:35 [447] Total bytes received: 19.63K
2020/10/20 13:17:35 [447] sent 334.07M bytes  received 19.63K bytes  287.14K bytes/sec  elapsed 00:19:23
2020/10/20 13:17:35 [447] total size is 132.66G  speedup is 397.08
-----------------------------------------

RSYNC completed:   10/20/20 - 13:17:35...

RSYNC took 1161.180000 seconds to complete the LOCAL backup task!
===============================================================================

Granted, neither run has a LOT of stuff to backup, but what I'm curious about is that 'Total transferred file size: 307.12M bytes' metric. I know for a fact that none of the updates add up to this total.

I suspect this is a sum of all the filesystem data compares that happened...but is it?

I ask b/c even the full disk runs which had all the darn FF cache entries (before I tossed them into the exception list) didn't cause this number to increase much. So does this represent the RSYNC data overhead of managing a disk-to-disk copy on my system?

I'm curious what some of you who run RSYNC backups are seeing for your working OS/2 volumes?

Thanks,
-Dariusz

Dariusz Piatkowski

  • Hero Member
  • *****
  • Thank You
  • -Given: 14
  • -Receive: 20
  • Posts: 821
  • Karma: +4/-0
    • View Profile
Re: RSYNC - local drive backups
« Reply #1 on: October 20, 2020, 09:11:13 pm »
...one more point noting, here are the results when using the '--inline' '--inplace' option:

Code: [Select]
===============================================================================
RSYNC started:     10/20/20 - 14:32:32...

SOURCE_FOLDER => G:/
TARGET_FOLDER => I:/

RSYNC CLI =>--archive --delete --progress --stats --whole-file --xattrs --os2-perms --human-readable --quiet --inplace  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_LOCAL_backup.log G:/ I:/

-----------------------------------------
2020/10/20 14:32:32 [475] building file list
2020/10/20 14:33:06 [475] .d..t...... HOME/.config/psi/profiles/Home/
2020/10/20 14:33:06 [475] >f..t.....x HOME/.config/psi/profiles/Home/options.xml
2020/10/20 14:33:06 [475] >f..t.....x HOME/.config/psi/profiles/Home/options.xml.backup
2020/10/20 14:33:09 [475] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/
2020/10/20 14:33:09 [475] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cookies.sqlite
2020/10/20 14:33:09 [475] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cookies.sqlite-wal
2020/10/20 14:33:09 [475] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/places.sqlite-wal
2020/10/20 14:33:09 [475] >f..t.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/xulstore.json
2020/10/20 14:33:11 [475] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/cache2/doomed/
2020/10/20 14:33:11 [475] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/datareporting/
2020/10/20 14:33:11 [475] >f..t.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/datareporting/aborted-session-ping
2020/10/20 14:33:14 [475] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/extension-data/
2020/10/20 14:33:14 [475] >f..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/extension-data/ublock0.sqlite
2020/10/20 14:33:16 [475] .d..t...... HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/
2020/10/20 14:33:16 [475] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/recovery.bak
2020/10/20 14:33:16 [475] >f.st.....x HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/sessionstore-backups/recovery.js
2020/10/20 14:33:20 [475] .d..t...... MPTN/ETC/
2020/10/20 14:33:22 [475] .d..t...... OS2/
2020/10/20 14:33:22 [475] >f..t...... OS2/OS2.!!!
2020/10/20 14:33:22 [475] >f..t...... OS2/OS2.INI
2020/10/20 14:33:22 [475] rsync: open "I:/OS2/OS2.INI" failed: Permission denied (13)
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/SCENTER.CFG
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock0.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock1.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock10.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock11.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock12.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock13.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock14.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock15.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock2.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock3.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock4.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock5.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock6.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock7.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock8.cfg
2020/10/20 14:33:28 [475] >f..t...... OS2/DLL/dock9.cfg
2020/10/20 14:34:12 [475] >f.st...... apps/tcpip/PMMail3/Accounts/logfile.log
2020/10/20 14:44:03 [475] >f..t...... apps/tcpip/web2/counter.dat
2020/10/20 14:44:06 [475] >f.st...... apps/tcpip/web2/web.log
2020/10/20 14:44:09 [475] >f..t...... code/source/REXX/os2utils/REXX_os2utils.vtg
2020/10/20 14:44:09 [475] >f.st...... code/source/REXX/os2utils/rsync_backup.cmd
2020/10/20 14:47:52 [475] >f.st...... tmp/log/rsync_LOCAL_backup.log
2020/10/20 14:47:52 [475] >f.st...... tmp/log/smartd.log
2020/10/20 14:47:54 [475] .d..t...... tmp/vslick/backup/code/source/REXX/os2utils/
2020/10/20 14:47:54 [475] >f.st...... tmp/vslick/backup/code/source/REXX/os2utils/rsync_backup.cmd
2020/10/20 14:51:12 [475] >f.st.....x util/misc/rsync_backup.cmd
2020/10/20 14:51:20 [475] >f.st...... util/upsmon/upsparm.log
2020/10/20 14:51:52 [475] Number of files: 304306
2020/10/20 14:51:52 [475] Number of files transferred: 39
2020/10/20 14:51:52 [475] Total file size: 132.66G bytes
2020/10/20 14:51:52 [475] Total transferred file size: 252.45M bytes
2020/10/20 14:51:52 [475] Literal data: 252.45M bytes
2020/10/20 14:51:52 [475] Matched data: 0 bytes
2020/10/20 14:51:52 [475] File list size: 26.88M
2020/10/20 14:51:52 [475] File list generation time: 0.060 seconds
2020/10/20 14:51:52 [475] File list transfer time: 0.000 seconds
2020/10/20 14:51:52 [475] Total bytes sent: 279.40M
2020/10/20 14:51:52 [475] Total bytes received: 19.57K
2020/10/20 14:51:52 [475] sent 279.40M bytes  received 19.57K bytes  240.57K bytes/sec  elapsed 00:19:21
2020/10/20 14:51:52 [475] total size is 132.66G  speedup is 474.77
2020/10/20 14:51:52 [475] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1168) [sender=3.0.9.1]
-----------------------------------------

RSYNC completed:   10/20/20 - 14:51:52...

RSYNC took 1159.850000 seconds to complete the LOCAL backup task!
===============================================================================

Given the RSYNC documentation the '--inline' should be the preferred choice for local disk-to-disk copies.

I am curious though whey the failure on the OS2.INI in the target drive? It is a JFS drive, and since no one is accessing that drive even the H-idden and S-ystem attributes should not cause a failure, right?

EDIT
====

Corrected the mistake Doug pointed out in his response below: there is no '--inline' option, the right reference is to use '--inplace'.
« Last Edit: October 20, 2020, 10:31:54 pm by Dariusz Piatkowski »

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 60
  • Posts: 1383
  • Karma: +0/-0
    • View Profile
Re: RSYNC - local drive backups
« Reply #2 on: October 20, 2020, 10:17:28 pm »
Quote
I'm curious what some of you who run RSYNC backups are seeing for your working OS/2 volumes?

I haven't looked through all of this, but this is what I found, when I was looking for myself.

First, RSync is not a good backup method for boot volumes (doesn't matter if they are HPFS, or JFS, drives), unless you are booted from an alternate boot system (the drive is not being used). There are just way too many things that are open, and some files have other flags set <see below>.

For boot volumes, I ZIP them up, using the parameters -r9yS (the parameter order doesn't matter). This works well, even when booted from the drive (you should close as much as possible). Then, I RSync the ZIP files to the backup device for safe keeping. You probably still need to filter things like log files out, because many programs keep them open, and ZIP can't work with them either. Old log files usually aren't much good anyway. I notice that some programs do not keep log files open, and that seems to be because they open the file, write to it, then close it (which makes more sense than keeping them open all the time, because there is less chance that CHKDSK will "fix" it after a system crash).

For my Data/Programs drive (which also contains my ZIPed boot volume backups) the ZIP (using the same commands) files are kept along with the boot volume ZIP files.

For RSYNC a FAT32 drive to a locally attached USB drive, I use:
Code: [Select]
rsync.exe -a --stats --no-perms --delete --log-file=W:/Utility/rsync/RSYNC2.LOG --filter=._W:/Utility/rsync/filters.list d:/ I:/main/DDrive/
For JFS (and HPFS, if I used it), I use:
Code: [Select]
rsync.exe -aX --stats --no-perms --delete --log-file=W:/Utility/rsync/RSYNC2.LOG --filter=._W:/Utility/rsync/filters.list w:/ I:/main/WDRIVE/
I also use RSync, to my ArcaOS machine, that I use as a NAS box. The command, that I use for a FAT32 drive is:
Code: [Select]
rsync.exe -a --progress --stats --no-perms --delete --log-file=W:/Utility/rsync/RSYNC3.log --filter=._W:/Utility/rsync/filters.list d:/ 192.168.0.108::IREBBS7/DDrive/
For JFS (and HPFS, if I used it), I use:
Code: [Select]
rsync.exe -aX --progress --stats --no-perms --delete --log-file=W:/Utility/rsync/RSYNC3.log --filter=._W:/Utility/rsync/filters.list w:/ 192.168.0.108::IREBBS7/WDRIVE/
File names, and paths need to change to match your preferences. RSYNC, in daemon mode, runs on the remote system (the "NAS" box). Filters.list contains the omitted list. It is the result of selecting failures, and determining if the file is really worth saving (many aren't). If it is an important file, you need to decide exactly how to handle it.

Quote
...one more point noting, here are the results when using the '--inline' option:

I don't see any '--inline' option. There is '--inplace', which is probably not a good idea, if the target drive has lots of room. I do use '--inplace' when the target drive doesn't have enough free space to have two copies of the largest file (plus a bit). Normally, RSYNC copies the file to a temporary name, then deletes the old one, and renames the new one. IMO, that is a LOT safer than deleting the original, then copying the new one.

Quote
I am curious though whey the failure on the OS2.INI in the target drive?

It is probably because it has the System, and Read Only,  flags, which effectively prevents programs from writing over it. It is one of the reasons why I don't bother using RSYNC with boot volumes.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Thank You
  • -Given: 14
  • -Receive: 20
  • Posts: 821
  • Karma: +4/-0
    • View Profile
Re: RSYNC - local drive backups
« Reply #3 on: October 22, 2020, 03:36:51 am »
Hey Doug,

...
Quote
I am curious though whey the failure on the OS2.INI in the target drive?

It is probably because it has the System, and Read Only,  flags, which effectively prevents programs from writing over it. It is one of the reasons why I don't bother using RSYNC with boot volumes.

Well, it's a little more complicated than that, and based on my results I actually suspect it's either a bad combination of parameters, or an outright bug, here is why.

1) the '--inplace' option produces the failure which I had already highlighted

2) the '--whole-file' option on the other hand appears to successfully handle the file

Code: [Select]
...
RSYNC CLI =>--archive --delete --progress --stats --whole-file --xattrs --os2-perms --human-readable --quiet  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_LOCAL_backup.log G:/ I:/

-----------------------------------------
2020/10/21 01:25:06 [512] building file list
2020/10/21 01:25:40 [512] .d..t...... HOME/.config/psi/profiles/Home/
...
2020/10/21 01:25:59 [512] .d..t...... OS2/
2020/10/21 01:25:59 [512] >f..t...... OS2/OS2.!!!
2020/10/21 01:25:59 [512] >f.st...... OS2/OS2.INI
...

which produces the following results:

Code: [Select]
[G:\os2]dirm os2.ini
2020-10-21 21:23         regular file    3234470 OS2.INI

[G:\os2]i:

[I:\os2]dirm os2.ini
2020-10-20 23:20         regular file    3234750 OS2.INI

Therefore, as you can tell the I: drive does contain a most recent OS2.INI file as of the boot drive's G: availability of that file from around midnight last night.

Anyways, RSYNC has a lot of flexibility, but I suspect we really would need someone on our end to get down to business with the porting effort and work through whatever issues we report. Neither option is likely to happen anytime soon for us I am afraid.


Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 11
  • -Receive: 195
  • Posts: 2720
  • Karma: +16/-0
    • View Profile
Re: RSYNC - local drive backups
« Reply #4 on: October 22, 2020, 05:28:05 am »

Anyways, RSYNC has a lot of flexibility, but I suspect we really would need someone on our end to get down to business with the porting effort and work through whatever issues we report. Neither option is likely to happen anytime soon for us I am afraid.

Have you checked the readme for who is maintaining it? I've had issues solved fairly quick but don't want to name names here.
As for OS2.INI, is it possible that sometimes you've tried transferring it when open? IIRC, the system often has it open but not always.