Rsync and eaclean

[Neil Waldhauer]

I keep my data on a single JFS drive, including my PMMail test mailstore, which has over 1 million messages. I back up this drive (2 TB) to several other drives on other computers using Rsync in client/server mode.

Due to a drive failure on my main computer, I had to use Rsync to transfer the entire drive, which took 30 hours. This seemed long to me, so I experimented with things and found that the PMMail mailstore was taking most of the time. Not only that, but the server was so loaded down that I could not use it.

Further experimentation showed me that it was the Extended Attribute (EA) processing that was taking so long. I used eaclean, by David Azarewicz to remove the EA from my mailstore message files and the time to transfer the mailstore using rsync changed from 9 hours to 45 minutes. The server was no longer impacted and the user interface ran normally.

You can obtain eaclean from David's Software Download Center

https://88watts.net/software.html

eaclean was able to remove unnecessary EA over one million files in a few minutes.

[Doug Bissett]

Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

[Dave Yeo]

Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Isn't PMMail now compiled with GCC/Libc? Libc does add EA's to everything it touches.

[Andi B.]

I don't know why that would cause RSync to use more time to transfer the files.

I don't know the details either but Steven explained it a while ago. Since then I switched off EA support for all my samba shares. I seem to remember there's also a switch with our rsync to disable this EA processing.

[Neil Waldhauer]

Using RSync to copy the mail store does add Extended Attributes (EAs) to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Rsync does not add EAs to every file when I use it. I'm pointing my rsync to an rsyncd running on the server.

Rsync is very fast, but the OS/2 code to read EAs is very slow, and apparently hogs critical system resources. My server machine appears to hang for up to 10 minutes, but it recovers and continues. This is unacceptable in a server. I need backups to work well. A recent server hard drive crash motivated me to improve my backup strategy.

But some EAs are useful to OS/2, so I want to keep some of them. eaclean gave me a way to remove the EAs that  I didn't need.

This is my call to rsync.

Code: [Select]

commandString = 'bin\rsync' /* name of the executable */,
                '-a',
                '-t',
                '--os2-perms',
                '--human-readable',
                '--del' /* delete files on target that no longer exist on source */,
                '--ignore-errors' /* delete files even if there are I/O errors */,
                '--xattrs' /* support extended attributes */,
                '--stats' /* show statistics at end */,
                '--itemize-changes' /* what rsync is doing with each file */

What adds the UID, GID, MODE, INO, RDEV, GEN, FLAGS extended attributes? I suspect SAMBA 3.6. Maybe Dave is right that it's libc.

[Paul Smedley]

Those EAs are added by libc afair

[Dave Yeo]

It's easy to test, create a file, lets say a text file with e.exe. It has no EA's. Use a libc program on it, lets say cp to new file, the copy has the EA's.
Knut had plans for the EA's and was very resistant to removing them, unluckily he seems to have lost interest.

[Neil Waldhauer]

Would there be a downside to cleaning off those libc-based EA from my entire drive? If libc puts them in, there must be an advantage to having them.

[Doug Bissett]

Would there be a downside to cleaning off those libc-based EA from my entire drive? If libc puts them in, there must be an advantage to having them.

I wouldn't do the whole drive. Icons are kept in EAs. As I said, PMMail does not use EAs (Libc may create them), so it is perfectly safe to remove them from the PMMail mail store.

[Dave Yeo]

Would there be a downside to cleaning off those libc-based EA from my entire drive? If libc puts them in, there must be an advantage to having them.

The UUID etc being cleaned should be fine. Have to remember that symlinks and permissions are also kept in EA's. While only some things (Samba?) care about the permissions, there are lots of symlinks that are important.

[Neil Waldhauer]

The UUID etc being cleaned should be fine. Have to remember that symlinks and permissions are also kept in EA's. While only some things (Samba?) care about the permissions, there are lots of symlinks that are important.

The eaclean program used with the -c parameter does not clean the symlink EA. It only removes the ones listed.

 UID, GID, MODE, INO, RDEV, GEN, FLAGS

This sounds very like flags needed for SAMBA or for sharing files with any multi-user system. Is there any way to test SAMBA in this case? If I got my Ubuntu system working again, could I use the Ubuntu SAMBA client to detect the difference?

[Neil Waldhauer]

Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Isn't PMMail now compiled with GCC/Libc? Libc does add EA's to everything it touches.

When I run PMMail, it creates these EA for new messages: UID, GID, MODE, INO, RDEV, GEN, FLAGS. These are exactly the ones targeted by eaclean -c.

[Dave Yeo]

Most all programs linked against kLIBC/libcn will create them. Use a GCC compiled unzip, use cp, etc and those EAs will be added to a file or be there on file creation.
Be thankful that they are no longer critical EAs as they were in some early libc's.