OS/2, eCS & ArcaOS - Technical > Applications

Rsync and eaclean

(1/3) > >>

Neil Waldhauer:
I keep my data on a single JFS drive, including my PMMail test mailstore, which has over 1 million messages. I back up this drive (2 TB) to several other drives on other computers using Rsync in client/server mode.

Due to a drive failure on my main computer, I had to use Rsync to transfer the entire drive, which took 30 hours. This seemed long to me, so I experimented with things and found that the PMMail mailstore was taking most of the time. Not only that, but the server was so loaded down that I could not use it.

Further experimentation showed me that it was the Extended Attribute (EA) processing that was taking so long. I used eaclean, by David Azarewicz to remove the EA from my mailstore message files and the time to transfer the mailstore using rsync changed from 9 hours to 45 minutes. The server was no longer impacted and the user interface ran normally.

You can obtain eaclean from David's Software Download Center

https://88watts.net/software.html

eaclean was able to remove unnecessary EA over one million files in a few minutes.

Doug Bissett:
Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Dave Yeo:

--- Quote from: Doug Bissett on March 31, 2023, 03:54:51 am ---Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

--- End quote ---

Isn't PMMail now compiled with GCC/Libc? Libc does add EA's to everything it touches.

Andi B.:

--- Quote ---I don't know why that would cause RSync to use more time to transfer the files.
--- End quote ---
I don't know the details either but Steven explained it a while ago. Since then I switched off EA support for all my samba shares. I seem to remember there's also a switch with our rsync to disable this EA processing.

Neil Waldhauer:

--- Quote from: Doug Bissett on March 31, 2023, 03:54:51 am ---Using RSync to copy the mail store does add Extended Attributes (EAs) to every file.

I don't know why that would cause RSync to use more time to transfer the files.

--- End quote ---

Rsync does not add EAs to every file when I use it. I'm pointing my rsync to an rsyncd running on the server.

Rsync is very fast, but the OS/2 code to read EAs is very slow, and apparently hogs critical system resources. My server machine appears to hang for up to 10 minutes, but it recovers and continues. This is unacceptable in a server. I need backups to work well. A recent server hard drive crash motivated me to improve my backup strategy.

But some EAs are useful to OS/2, so I want to keep some of them. eaclean gave me a way to remove the EAs that  I didn't need.

This is my call to rsync.


--- Code: ---commandString = 'bin\rsync' /* name of the executable */,
                '-a',
                '-t',
                '--os2-perms',
                '--human-readable',
                '--del' /* delete files on target that no longer exist on source */,
                '--ignore-errors' /* delete files even if there are I/O errors */,
                '--xattrs' /* support extended attributes */,
                '--stats' /* show statistics at end */,
                '--itemize-changes' /* what rsync is doing with each file */

--- End code ---

What adds the UID, GID, MODE, INO, RDEV, GEN, FLAGS extended attributes? I suspect SAMBA 3.6. Maybe Dave is right that it's libc.


Navigation

[0] Message Index

[#] Next page

Go to full version