SAMBA, RSync, and ZIP.

[Doug Bissett]

This is a report of a problem that I have seen a few times. I would like some suggestions about what to report, and where to report it. I think that SAMBA causes the problem that causes RSync, and ZIP to crash when they stumble upon the "defective" file. Other programs don't seem to mind the file.

First, I will state that RSync, and ZIP, should NOT crash, and they need to be fixed. Who is looking after those programs these days, and where does one report problems? Then, I don't think that SAMBA should be creating files with EAs that contain a blank, empty, EA in the middle of other EAs.

Anyway, what I did: I use a small batch file to update the files distributed by Virtual Box Extensions, for OS/2. It is pretty simple (and far from complete). I keep the script on my main machine, so I can access it from any SAMBA attached eCS system in a VM. Recently, I had need to change the drive letter in that file, so I Attached the share in SAMBA, copied the file (from the remote directory to the remote directory) to a new name, then edited it using the E editor (EA.EXE), saved it, and it worked fine. When trying to reproduce the problem, it seems that the problem does not always happen, but I have seen it a few times over the last few months.

Later, RSync came along, and crashed when it got to the file. even later, it did it again. After much messing around, I isolated the problem to that file (simply listing the file with RSync causes the crash). Then, I tried to ZIP up the folder, so I wouldn't lose it, and ZIP crashed when it got to that file. ARJ doesn't seem to mind (but I haven't verified that the problem was retained).

Using FM/2 (latest), I looked around on the file, and discovered a problem in the EAs. There is a blank EA, with no data, in the list. If I remove the EAs, the crash no longer happens.

I attached RSyncCrash.arj, for those who might be interested in looking closer.

[Dave Yeo]

Steven Levine seems to be in charge of rsync, at least when I had a crash with it caused by connecting to a Linux machine that had xattrs turned off.
It'd be interesting to test rsync (with xattrs enabled) with this file on Linux.
As far as I know, no-one is looking after unzip and zip as, at least the last time I looked, they just built and worked out of the box. You could always test the RPM/YUM versions and if it still crashes, open a bug report at netlabs/rpm.
Samba seems to be being looked after by bitwise and I believe the issue tracker is also at netlabs so an issue can be opened there as well.

[Doug Bissett]

It'd be interesting to test rsync (with xattrs enabled) with this file on Linux.

I don't know enough about Linux to know if xattrs are enabled or not. The file always causes the crash, even when simply listing the file using RSync, so it appears that it is RSync on the local machine that has the problem, not a remote machine. RSync also fails copying to the local machine, but I expect that is because it can't read the EAs without crashing. I suspect that ZIP has the same problem, and it could even be the same cause, if a common library is involved. ZIP works, if I use -X as a parameter. I need to try the equivalent with RSync.

I attached RSyncCrash.arj, for those who might be interested in looking closer.

It appears that ARJ just dropped the EAs, so the extracted file doesn't have the problem. It also seems that the SAMBA server just drops the defective EAs when a remote machine copies the file. I expect that other programs will do the same, unless they crash.

You could always test the RPM/YUM versions and if it still crashes, open a bug report at netlabs/rpm.

It appears that the RPM/YUM version of ZIP is the same as the one that I use normally, Zip 3.0 (July 5th 2008), and yes it does crash. I will see if they will accept a problem ticket, but I can't seem to find a way to package the file to send it to them, and it seems that the problem does not always happen, so it isn't easy to reproduce the problem.

Samba seems to be being looked after by bitwise and I believe the issue tracker is also at netlabs so an issue can be opened there as well.

I got tired of reporting problems there. Every time I did, I was told that I was the only person to have the problem, so they closed my tickets. Eventually, they did fix enough things that enough problems went away, magically, that I could actually use SAMBA (it still has problems, which seem to be in EVFSGUI, or associated software). I will try again, but I need to find a way to package the bad EAs - perhaps EAUTIL will do something, but I really hate that program, and there is always the possibility that it is E.EXE (EA.EXE), or something else, that is causing the problem.

Steven Levine seems to be in charge of rsync

I think I will start there. I hate to bother him about trivial things, because he is so busy, but he is the most likely to understand what might be happening. It would be really nice if somebody else could duplicate the problem, somehow. It makes me wonder how many backups are failing silently, because ZIP or RSync are just crashing. Unless you look at the log, you would never know.

[Doug Bissett]

It appears that ARJ just dropped the EAs, so the extracted file doesn't have the problem.

Okay. EAUTIL does split, and restore, the EAs. After you restore the EAs, try to ZIP the folder that contains VMUpdateY.cmd. RSYNCCRASH.zip attached. Read Crash.txt for details.

[Doug Bissett]

You could always test the RPM/YUM versions and if it still crashes, open a bug report at netlabs/rpm.

Wrong place. It should be http://trac.netlabs.org/ports/report (I think). Anyway, it is reported, for ZIP, and RSync. Now, to see what comes of that. I haven't bothered SHL, yet.   

I am still not too sure exactly what creates the blank, empty, EA, so more investigation is required for that. At the moment, it looks like some part of EVFSGUI is doing it, but I can't prove it.

[Dave Yeo]

R:\tmp\crash>eautil VMUpdateY.cmd VMUpdate.cmd.EAs /J /O
SYS0254: An incorrect extended attribute name was used.

R:\tmp\crash>help sys0254

SYS0254: An incorrect extended attribute name was used.

EXPLANATION: The extended-attribute name contains a character that is
incorrect.

ACTION: Correct the extended-attribute name and retry the operation.

Perhaps try using rar to save the file.

[Doug Bissett]

Interesting. I was able to unzip the file, then use EAUTIL to put the defective EAs back, with no errors. RSync and ZIP failed as they did on the other system.

I am running eCS 2.1, with the latest Arca Noae fixes, on the first machine, and eCS 2.2 b2, with the Arca Noae fixes on the second machine. The second machine has RPM/YUM, and Yumie. The first is updated by hand.

Your errors seem to indicate that it wants the EA file to be called VMUpdateY.cmd.EAs. Not having the "Y" was a typo, but it didn't seem to matter, for me, so I left it.

[Doug Bissett]

Perhaps try using rar to save the file.

It tells me "ERROR: Cannot get extended attributes of W:\Download\RSyncCrash\VMUpdateY.c" ("cmd" was cut off), but it didn't crash.

[Dave Yeo]

I tried a couple of versions of eautil.exe and renamed VMUpdateY.cmd.EAs with the same result. This is also eCS 2.1 with all the Arca Noae updates. R: is a tmpfs volume.
Hmm, same error on a HPFS volume but it worked on a JFS volume, strange, perhaps it really is an incorrect EA name, as in having a non ascii character as JFS uses unicode.
Using the zip that is included with eCS 2.1 (3.0 compiled with GCC 4.4.6) zip simply fails with zip error: Interrupted (aborting)
rsync does crash.
Using Unimaint to view the EAs, they look fine (of course the binary ones are questionable) and testing them reports that they are valid.
Using the zip (2.2 compiled Nov 3rd 1997) that is in my Warp V4 tcpip\bin works fine to zip it., You should test a zip that was not compiled with kLIBC.
Possibly it is an error in one of the EAs that kLIBC adds so kLIBC (only recent?) causes the program to fail.
I'll attach the one I zipped, perhaps the EAs got altered here.
edit: it now unzips and zips fine so I probably changed something. I have to go to bed now.

[Doug Bissett]

same error on a HPFS volume but it worked on a JFS volume,

I use JFS. I don't even have a HPFS volume. I use FAT32 without EAs, so simply moving the file there fixes the problem.

You should test a zip that was not compiled with kLIBC.

I don't keep old versions around, after I am satisfied that the newer one works.

I'll attach the one I zipped, perhaps the EAs got altered here.

The EAs in your ZIP are quite different from the ones that cause the problem.

SHL e-mailed me this:

I'll see what I can do to avoid the trap.  Rsync traps because it exposes a defect in the DosEnumAttribute API when multiple EAs are requested.  The list returned by the API is corrupted.  Fm/2 does not trap because it requests only one EA per call.

He also suggested:

please enter a ticket in the Rsync project on Paul's mantis.

Now, to figure that out.

I am still not too sure, exactly, what is causing the bad EA, but it seems to have something to do with EVFSGUI or SAMBA, adding EAs to files that are transferred. Perhaps it would be best to simply not enable EA support. EAs are almost never needed anyway, but the problem should be fixed.

[Doug Bissett]

please enter a ticket in the Rsync project on Paul's mantis.

Well, that got a response from SHL. He says:

This turns out to be a kernel defect in DosEnumAttribute. Given the broken EAs in the testcase, FEA2 list is not returned as documented, if more than 1 EA is returned. The result is a trap when the code attempts to dereference the oNextEntry pointer. I will update rsync to report this as an error which would have been the case if the kernel had detected the malformed EA data.

That should get RSync fixed. Hopefully the report at Netlabs UNIX ports will get ZIP fixed.

I am still not too sure what is causing the bad EAs in the first place.

[Greggory Shaw]

Really cool if we could get Rsync and OpenSSh to work together too !