RSYNC - what to put in EXCLUDE list?

[Dariusz Piatkowski]

Alright...so with the New Year comes a resolution of mine: "get rsync up and running"!

I used to run dSync back in the day to run a local (disk to disk) nightly backup of my partitions (mostly Maintenance and OS2). I then moved off of HPFS386 to JFS, made some changes to hardare (went to SSD) and just never got around to re-configuring the backup routines. Now, dSync had some issues, so I decided to move to RSYNC.

Alright, so with a few spare days on my hands I was able to make a bit of progress. Spent the day yesterday actually re-learning all the RSYNC stuff and managed to do a whole bunch of testing with the --dry-run option (so no actual changes are written out). Again, this is still a LOCAL rsync, so if I understood the functionality correctly it does it all as if the '--whole-file' parameter was specified, which is all fine here (since on a local storage it really would not make sense to do just the incremental, unless of course it's some massively huge data file - maybe dbms???). Anyways, once I have this rsync configured correctly I will execute nightly through the CRON/2 deamon process I have installed on my machine.

Now here comes the core of my post: for a complete backup of a full OS2 partition (one that has boot and apps on it) I am seeing three exceptions on my machine, they are:

1) SWAPPER.DAT

Code: [Select]

2020/01/01 00:48:07 [587] [sender] os2getxattr: DosOpenL(./SWAPPER.DAT) failed with error 32 at lib/sysxattrs.c(426)
2020/01/01 00:48:07 [587] rsync: get_xattr_names: llistxattr("g:/./SWAPPER.DAT",1024) failed: Permission denied (13)

2) Mozilla => parent.lock

Code: [Select]

2020/01/01 00:48:36 [587] [sender] os2getxattr: DosOpenL(HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/parent.lock) failed with error 32 at lib/sysxattrs.c(426)
2020/01/01 00:48:36 [587] rsync: get_xattr_names: llistxattr("g:/HOME/mozilla.org/userprofiles/Mozilla/Firefox/Profiles/p5gjgk21.Testing/parent.lock",1024) failed: Permission denied (13)

3) PMMail => mua.mtx

Code: [Select]

2020/01/01 00:49:38 [587] [sender] os2getxattr: DosOpenL(apps/tcpip/PMMail3/Accounts/mua.mtx) failed with error 32 at lib/sysxattrs.c(426)
2020/01/01 00:49:38 [587] rsync: get_xattr_names: llistxattr("g:/apps/tcpip/PMMail3/Accounts/mua.mtx",1024) failed: Permission denied (13)

Therefore, the question I have is: can I skip all of the above files?, or if I should NOT skip them are there RSYNC options to allow me to deal with such a situation (force the archive past the 'Permission denied' error)?

I think SWAPPER.DAT is an easy one, yes, I can skip it. If I ever have to recover from a backup image that will get re-created during boot.

The 'Mozilla => parent.lock' is most likely due to FF being up and running, which is going to be the situation, so I can not avoid this. Likewise, the 'PMMail => mua.mtx' is a semaphore I think, it is always present as long as PMMail is up and running, so that one I can not avoid as well.

As always, appreciate any and all feedback you can provide! Thanks...

[ivan]

Hi Dariusz,

Not saving the first 2 is not a problem, in fact I have SWAPPER.DAT out on a ram disk and I regularly delete parent.lock to stop ff popping up the 'do you want to renew' message.

mua.mtx must be something from the newer versions of PMMail, I don't see it on my old 2.20.2382 version so I can't help there.   

[Doug Bissett]

This is my Exclude list:

Code: [Select]

- $live$
- Cache/
- Cache.Trash/
- umix.log
- DSS_RMT.PID
- /PROCDUMP/
- /CDTemp/
- Recycled
- /TEMP/
- /SPOOL/
- /SWAPPER/
- System Volume Information
- rsync*.log
- clamd.log
- logfile.log*
- stunnel.log
- WP?ROOT.?SF
- WP?SHARE.?SF
- var/log/cups/
- /SYSDumps/
- *thumbnails/
- /WATCOM/binp/nmpbind.exe
- /bluegriffon/
- /Kompozer/
- /Seamonkey/
- /thunderbird/
- /TRASH/
- $RECYCLE.BIN
- TestAccts/
- /Download/RSyncCrash/updvbox.cmd
- chklgjfs*.log
- PSTAT.$$$
- webmonx.log
- /HOME/OpenOffice/4/user/extensions/shared/log.txt
- /HOME/OpenOffice/4/user/extensions/tmp/
- /MOZPROFILES/MOZILLA/Firefox/Profiles/wzf5p2q3.default-1485581936463/
- parent.lock
- socktidy.log
- CHKDSK.*
- volumes.cfg
- helpdesk/host/etc/passwd
- static.key

Most of it is just for my convenience, but none of it is actually required in a backup. You need to determine that, for yourself.

This is a typical command file to backup one local disk to another (USB) local disk:

Code: [Select]

ECHO --->RSYNC.LOG
ECHO Backup started>>RSYNC.LOG

ECHO ---/D:\>>RSYNC2.LOG
rsync.exe -a --stats --delete --log-file=RSYNC2.LOG --filter=._filters.list d:/ I:/main/DDrive/
ECHO ---/D: Done>>RSYNC2.LOG

ECHO ---/W:\>>RSYNC2.LOG
rsync.exe -aX --stats --delete --log-file=RSYNC2.LOG --filter=._filters.list w:/ I:/main/WDRIVE/
ECHO ---/W: Done>>RSYNC2.LOG
You do need to change this, to match your system. I: is the USB HDD (JFS). D: is a FAT32 drive, and W: is a JFS drive (my main programs and data drive). This basically just smart copies D:\* to I:\main\DDrive, and W:\* to I:\main\WDrive, leaving out what is listed in filters.list. You can make separate exclude, and include, lists, depending on what you are trying to do.

I normally shut down things like Firefox, PMMail, and OpenOffice, over night, and this runs at about 4 AM (I use the scheduler in DragText). Swapper.DAT and parent.lock are pretty obvious things to omit. Look through my list, above, for other ideas. I built the list by studying the logs, to see what it has trouble with, then I decide what to do about it. Most of the time, I just add them to the list. It is always a good idea to see if the affected program has a problem, by renaming the file, before you commit to leaving it out. Some of it is just stuff that isn't really needed, to cut down on the time, and storage room, that is required.

I have SWAPPER.DAT out on a ram disk

Just an observation about this. I have more than 2 GB in most of my systems. It has been YEARS, since I saw the swap file actually used. I usually just leave it as it was installed. Even my 1 GB machine, running ArcaOS, and used as a NAS device (with some extras), never uses the swap file. If your RAMDISK is in memory above about 3.5 GB, it doesn't matter. If it is below the 3.5 GB mark, you would be better off to have the swap file on a real disk, and use the memory as program memory.

mua.mtx must be something from the newer versions of PMMail, I don't see it on my old 2.20.2382 version so I can't help there. 

mua.mtx is new, and it can be omitted.

[ivan]

Hi Doug,

All my machines have 8GB of memory installed including those running OS/2 - ArcaOS so ram disk space is not a problem

[Tom]

(...)  I regularly delete parent.lock to stop ff popping up the 'do you want to renew' message.

Deleting parent.lock is only a temporary solution, as you have discxovered: it needs to be repeated from time to time.

A permanent solution is to create a new boolean key with the name

browser.disableResetPrompt

through about:config and give it the value True .

[Dave Yeo]

I have hit the swap file pretty good with 2GB's of ram and after a ram module died and I had 1.5GB's, I managed to find out what happens when the swap file grows to 2GB's, namely the system died with a swap file full error. Guess they used a signed long somewhere which made sense when files were limited to 2GB's.
This was linking xul.dll around FF24 IIRC, also needed VIRTUALADRESSLIMIT set to 3072 at the time. After the swapfile full error I made sure not to be doing anything when linking xul.
In theory, on a 32bit i386 system the swap file could grow to a terabyte IIRC but our ancient system seems to use signed longs and unsigned longs in too many places so pretty hard to get swapping with 4GB's of ram installed.

[Dariusz Piatkowski]

Hi Doug!

This is my Exclude list:

Code: [Select]

- $live$
- Cache/
- Cache.Trash/
...snip...


Ahh...excellent idea, toss the EXCLUDE items in a file as opposed to passing them in the command line. You are absolutely right about what should and should not go there, these are all very "system specific", no doubt. For now the three I listed are what is being rejected. Since this is a LOCAL rsync run, disk=>disk, the speed is fast so I'm actually thinking of doing basically everything that isn't being a problem for rsync to handle.

...mua.mtx is new, and it can be omitted...

OK, good to know that. I will exclude.

Thanks Doug!

[Dariusz Piatkowski]

Hi Tom!

(...)  I regularly delete parent.lock to stop ff popping up the 'do you want to renew' message.

Deleting parent.lock is only a temporary solution, as you have discxovered: it needs to be repeated from time to time.

A permanent solution is to create a new boolean key with the name

browser.disableResetPrompt

through about:config and give it the value True .

OK, excellent reminder. I have put that very key in my FF install quite some time ago to avoid that repeated reminder, but...I did not physically delete the file. Checking up on it right now I find it has the following date/time stamp:

 7-14-17   3:16p         0           0  parent.lock

Which I'm assuming is maybe the original install file given that I only have about 3 other files with that very same date/time stamp.

I will give this a try by re-naming the file and see if it ends up being re-created. I believe it should not given the current date/time stamp I see on my system, although just about every on-line note on this topic states that the file is automatically re-created each time FF starts (it enforces a single profile use).

Thanks!

[Doug Bissett]

I have hit the swap file pretty good with 2GB's of ram and after a ram module died and I had 1.5GB's, I managed to find out what happens when the swap file grows to 2GB's, namely the system died with a swap file full error. Guess they used a signed long somewhere which made sense when files were limited to 2GB's.

You are an exception. Not many people try to build XUL.DLL (which is a totally ridiculous piece of software to begin with).

I think the swap file can be up to the size that the disk space will allow, or up to the size of what OS/2 can address minus the size of real memory. There may be some other limit, as you describe. After all, it is all addressable memory space, so it is limited to what OS/2 can address.

pretty hard to get swapping with 4GB's of ram installed.

That should be impossible, since all of the addressable memory will be in real memory.

A permanent solution

Well, that is "permanent" until something changes in Firefox, or somebody creates a new profile, or any number of other reasons. I simply delete the file at each boot. Problem solved.

[Dariusz Piatkowski]

...I will give this a try by re-naming the file and see if it ends up being re-created. I believe it should not given the current date/time stamp I see on my system, although just about every on-line note on this topic states that the file is automatically re-created each time FF starts (it enforces a single profile use)...

Well, I suppose as it was to be expected the parent.lock file is created each time FF is started. So deleting, renaming, etc. does me no good. I will toss that thing into my EXCLUDE list.

BTW: I haven't read up on this yet, but what exactly is the difference (functionality wise) between 'filter' & 'exclude'?

Also, in the 3.0.9.1 readme there following is present:

Code: [Select]

eCS/OS2 specific changes:
- Supports 2 second native file time resolution
  Defaults --modify-window to 1 second
  Disable with --modify-window=0

How is this relevant to us? I do not know what is the actual time resolution on OS/2...haven't tried an experiment to figure this out...but does anyone know if I need to utilize this?

[Dave Yeo]

@Dariusz, I believe OS/2 (and DOS) has only 2 second resolution on file times, whereas Linux and such has 1 second resolution. I'd guess that generally you should leave it alone, IIRC Steven added this and is fairly knowledgeable about this stuff.

@Doug, I had lots of room when the swap file error happened. It makes sense it was written with a 2GB limit as that was the limit of files before JFS.
The 4GB limit is how much memory a process can access, not the CPU. At that, while the 386 could only access 4GB's of real memory(ram), the i686 and newer can use PAE to access 64GB's or more of real memory(ram) so could have 16 processes in memory each accessing 4GB's of address space and no swapping. It's actually more complicated with shared memory, PCI address space and such. With swapping, you can have even more processes access their own address space, even with less real ram. It's actually more complicated due to limits on the page tables, overhead and such though the CPU does have fixes like changing from a 4kb page to a 2MB page.
Our limits are due to the internal structures of OS/2, an OS originally targeting the 286 and originally limited to 1GB (minus 512MB for the OS) per process as that was the limit for a 286's virtual memory.

[Dariusz Piatkowski]

@Dave, I decided to not mess around with the time resolution option for now. To test this out I need to prep a couple of use-cases with the help of 'touch' and see how this really plays out.

Alright, regarding the overall config - I still haven't finished up my stuff here, got side-lined with other things, but so far here is the CLI I have:

Code: [Select]

RSYNC --dry-run -aX2 --delete --progress --stats --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_local_f_abr-1.log G:\ I:\

Now there is just one more thing for me to take care of before I have my local disk backups up and running again. How are you guys handling the logging part of things?

What I mean by that is the following:

1) when I leave things as you see them above the CLI window scrolls a listing of each and every directory that's being parsed, that's of course massive, heck the LOG file in such cases is about 2Meg in size and since I intend on running this nightly I have no desire to accumulate such logs

2) when I include the '-q' (quiet) parameter, well, it becomes a one-liner (as it's intended to be when scheduled through something like cron, etc.)...umm, not the results I need since I still want to see at least a high level summary, thus the --stats option

3) basically I am looking for the equivalent of --info=name0 which unfortunately does not exist in our 3.0.9 version of rsync (I think that's a > 3.1.x feature)

So literally, is there a param option/combo that allows me to capture the stats generated with '-verbose -v -q' (see below) and yet skipping the listing of all the directories?

Code: [Select]

...
2019/12/31 20:17:59 [554] total: matches=0  hash_hits=0  false_alarms=0 data=0
2019/12/31 20:17:59 [554] rsync[554] (sender) heap statistics:
2019/12/31 20:17:59 [554]   arena:       33000662   (bytes from sbrk)
2019/12/31 20:17:59 [554]   ordblks:            0   (chunks not in use)
2019/12/31 20:17:59 [554]   smblks:             0
2019/12/31 20:17:59 [554]   hblks:              0   (chunks from mmap)
2019/12/31 20:17:59 [554]   hblkhd:             0   (bytes from mmap)
2019/12/31 20:17:59 [554]   allmem:      33000662   (bytes from sbrk + mmap)
2019/12/31 20:17:59 [554]   usmblks:            0
2019/12/31 20:17:59 [554]   fsmblks:            0
2019/12/31 20:17:59 [554]   uordblks:    33000662   (bytes used)
2019/12/31 20:17:59 [554]   fordblks:    47619544   (bytes free)
2019/12/31 20:17:59 [554]   keepcost:           0   (bytes in releasable chunk)
2019/12/31 20:17:59 [554] Number of files: 299374
2019/12/31 20:17:59 [554] Number of files transferred: 34201
2019/12/31 20:17:59 [554] Total file size: 125717278064 bytes
2019/12/31 20:17:59 [554] Total transferred file size: 46983542944 bytes
2019/12/31 20:17:59 [554] Literal data: 0 bytes
2019/12/31 20:17:59 [554] Matched data: 0 bytes
2019/12/31 20:17:59 [554] File list size: 25896274
2019/12/31 20:17:59 [554] File list generation time: 0.065 seconds
2019/12/31 20:17:59 [554] File list transfer time: 0.000 seconds
2019/12/31 20:17:59 [554] Total bytes sent: 26812730
2019/12/31 20:17:59 [554] Total bytes received: 916455
2019/12/31 20:17:59 [554] sent 26812730 bytes  received 916455 bytes  26908.48 bytes/sec  elapsed 00:17:10
2019/12/31 20:17:59 [554] total size is 125717278064  speedup is 4533.75 (DRY RUN)
...