Author Topic: Rsync and eaclean  (Read 7422 times)

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1045
  • Karma: +25/-0
    • View Profile
    • Blonde Guy
Rsync and eaclean
« on: March 31, 2023, 01:53:25 am »
I keep my data on a single JFS drive, including my PMMail test mailstore, which has over 1 million messages. I back up this drive (2 TB) to several other drives on other computers using Rsync in client/server mode.

Due to a drive failure on my main computer, I had to use Rsync to transfer the entire drive, which took 30 hours. This seemed long to me, so I experimented with things and found that the PMMail mailstore was taking most of the time. Not only that, but the server was so loaded down that I could not use it.

Further experimentation showed me that it was the Extended Attribute (EA) processing that was taking so long. I used eaclean, by David Azarewicz to remove the EA from my mailstore message files and the time to transfer the mailstore using rsync changed from 9 hours to 45 minutes. The server was no longer impacted and the user interface ran normally.

You can obtain eaclean from David's Software Download Center

https://88watts.net/software.html

eaclean was able to remove unnecessary EA over one million files in a few minutes.
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Doug Bissett

  • Hero Member
  • *****
  • Posts: 1596
  • Karma: +4/-2
    • View Profile
Re: Rsync and eaclean
« Reply #1 on: March 31, 2023, 03:54:51 am »
Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4871
  • Karma: +103/-1
    • View Profile
Re: Rsync and eaclean
« Reply #2 on: March 31, 2023, 07:00:59 am »
Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Isn't PMMail now compiled with GCC/Libc? Libc does add EA's to everything it touches.

Andi B.

  • Hero Member
  • *****
  • Posts: 842
  • Karma: +11/-2
    • View Profile
Re: Rsync and eaclean
« Reply #3 on: March 31, 2023, 09:53:14 am »
Quote
I don't know why that would cause RSync to use more time to transfer the files.
I don't know the details either but Steven explained it a while ago. Since then I switched off EA support for all my samba shares. I seem to remember there's also a switch with our rsync to disable this EA processing.

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1045
  • Karma: +25/-0
    • View Profile
    • Blonde Guy
Re: Rsync and eaclean
« Reply #4 on: March 31, 2023, 03:41:53 pm »
Using RSync to copy the mail store does add Extended Attributes (EAs) to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Rsync does not add EAs to every file when I use it. I'm pointing my rsync to an rsyncd running on the server.

Rsync is very fast, but the OS/2 code to read EAs is very slow, and apparently hogs critical system resources. My server machine appears to hang for up to 10 minutes, but it recovers and continues. This is unacceptable in a server. I need backups to work well. A recent server hard drive crash motivated me to improve my backup strategy.

But some EAs are useful to OS/2, so I want to keep some of them. eaclean gave me a way to remove the EAs that  I didn't need.

This is my call to rsync.

Code: [Select]
commandString = 'bin\rsync' /* name of the executable */,
                '-a',
                '-t',
                '--os2-perms',
                '--human-readable',
                '--del' /* delete files on target that no longer exist on source */,
                '--ignore-errors' /* delete files even if there are I/O errors */,
                '--xattrs' /* support extended attributes */,
                '--stats' /* show statistics at end */,
                '--itemize-changes' /* what rsync is doing with each file */

What adds the UID, GID, MODE, INO, RDEV, GEN, FLAGS extended attributes? I suspect SAMBA 3.6. Maybe Dave is right that it's libc.


Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Paul Smedley

  • Hero Member
  • *****
  • Posts: 2139
  • Karma: +166/-0
    • View Profile
Re: Rsync and eaclean
« Reply #5 on: March 31, 2023, 07:03:08 pm »
Those EAs are added by libc afair
« Last Edit: March 31, 2023, 08:11:03 pm by Paul Smedley »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4871
  • Karma: +103/-1
    • View Profile
Re: Rsync and eaclean
« Reply #6 on: March 31, 2023, 07:34:30 pm »
It's easy to test, create a file, lets say a text file with e.exe. It has no EA's. Use a libc program on it, lets say cp to new file, the copy has the EA's.
Knut had plans for the EA's and was very resistant to removing them, unluckily he seems to have lost interest.

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1045
  • Karma: +25/-0
    • View Profile
    • Blonde Guy
Re: Rsync and eaclean
« Reply #7 on: April 01, 2023, 03:35:58 pm »
Would there be a downside to cleaning off those libc-based EA from my entire drive? If libc puts them in, there must be an advantage to having them.
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Doug Bissett

  • Hero Member
  • *****
  • Posts: 1596
  • Karma: +4/-2
    • View Profile
Re: Rsync and eaclean
« Reply #8 on: April 01, 2023, 04:02:45 pm »
Would there be a downside to cleaning off those libc-based EA from my entire drive? If libc puts them in, there must be an advantage to having them.

I wouldn't do the whole drive. Icons are kept in EAs. As I said, PMMail does not use EAs (Libc may create them), so it is perfectly safe to remove them from the PMMail mail store.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4871
  • Karma: +103/-1
    • View Profile
Re: Rsync and eaclean
« Reply #9 on: April 01, 2023, 05:34:26 pm »
Would there be a downside to cleaning off those libc-based EA from my entire drive? If libc puts them in, there must be an advantage to having them.

The UUID etc being cleaned should be fine. Have to remember that symlinks and permissions are also kept in EA's. While only some things (Samba?) care about the permissions, there are lots of symlinks that are important.

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1045
  • Karma: +25/-0
    • View Profile
    • Blonde Guy
Re: Rsync and eaclean
« Reply #10 on: April 02, 2023, 03:32:35 pm »
Quote
The UUID etc being cleaned should be fine. Have to remember that symlinks and permissions are also kept in EA's. While only some things (Samba?) care about the permissions, there are lots of symlinks that are important.

The eaclean program used with the -c parameter does not clean the symlink EA. It only removes the ones listed.

 UID, GID, MODE, INO, RDEV, GEN, FLAGS

This sounds very like flags needed for SAMBA or for sharing files with any multi-user system. Is there any way to test SAMBA in this case? If I got my Ubuntu system working again, could I use the Ubuntu SAMBA client to detect the difference?
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1045
  • Karma: +25/-0
    • View Profile
    • Blonde Guy
Re: Rsync and eaclean
« Reply #11 on: April 06, 2023, 04:14:20 am »
Just for the record, PMMail does not use EAs. Using RSync to copy the mail store does add EAs to every file.

I don't know why that would cause RSync to use more time to transfer the files.

Isn't PMMail now compiled with GCC/Libc? Libc does add EA's to everything it touches.

When I run PMMail, it creates these EA for new messages: UID, GID, MODE, INO, RDEV, GEN, FLAGS. These are exactly the ones targeted by eaclean -c.
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4871
  • Karma: +103/-1
    • View Profile
Re: Rsync and eaclean
« Reply #12 on: April 06, 2023, 07:05:22 am »
Most all programs linked against kLIBC/libcn will create them. Use a GCC compiled unzip, use cp, etc and those EAs will be added to a file or be there on file creation.
Be thankful that they are no longer critical EAs as they were in some early libc's.