Author Topic: Shared folders IFS for VBox Additions, the test version  (Read 106196 times)

Valery Sedletski

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +2/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #75 on: June 24, 2019, 09:08:11 pm »
BTW, I released a new r199 version of VBox Additions today:

Quote
ftp://osfree.org/upload/vbox/additions/VBoxAdditions-os2-x86-5.0.51.r199.zip

Many fixes were added, including timestamp fixes: now "touch" utility gives the current time as a timestamp
for affected files. "Make" utility no more loops around, so that, building sources with "make" should work
correctly. Also, I adjusted returned file listings, so file find operation (with DosFindFirst/Next API's) no more
returns an empty filename, so now "copy" command works correctly in both CMD and 4OS/2. Also, now
ZIP and WIC utilities work correctly, so that I was able to create ZIP and WarpIN archives over a shared
folder. Also, I fixed some traps and memory leaks, so now I was able to check out VBox and fat32.ifs and
osFree sources, and build them over a shared folder without any problems. VBox sources take about
1 GB of space on disk, and compile for about 6-8 hours in a VBox machine under a Linux host. So,
no memory leaks occured that time and everything was built without problems. Git and Svn clients
work fine over a shared folder too.

So, I'll appreciate any testing.

PS: This version also has fixed wildcard matching, the DOS way (instead of the UNIX way).

PPS: Now timestamps are adjusted to local timezone. For that, TZ environment variable in config.sys
is parsed to calculate a time zone offset. Time zone offset calculating is done in VBoxService.exe
"Timesync" service, and set a "timezone" field in Global infoseg (via DosSetDateTime/DosGetDateTime
API's). Currently, only simple syntax like "TZ=MSK-3" works. (I'll implement a full timezone parser in the future)

PPPS: Also, I implemented logging to OS/4 / QSINIT / ArcaLDR log buffer. There are unfinished docs (README
and DEBUG documents) added. Please read DEBUG doc for details on debugging techinques (using the log
buffer etc.).
« Last Edit: June 24, 2019, 09:25:47 pm by Valery Sedletski »

ivan

  • Hero Member
  • *****
  • Posts: 1558
  • Karma: +17/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #76 on: June 24, 2019, 09:49:17 pm »
Now all we need is a vBox host for OS/2 that works as well as your additions.

Valery Sedletski

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +2/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #77 on: June 24, 2019, 09:59:27 pm »
OS/2 host? It works, isn't it? OS/2 guest over OS/2 host with hardware-assisted virtualization works fine on my machines, too.

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 331
  • Karma: +23/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #78 on: June 24, 2019, 10:19:22 pm »
I have mostly implemented returning empty EA list in some IFS functions, so, in theory, WPS should show a folder listing in "Drives" object. In practice, it shows it sometimes, and sometimes it shows a "No objects were found that  matched the specified match criteria" dialog. Some other problem occurs. No idea yet, why.

Just to confirm: if your IFS receives a request for the total size of all EAs (FIL_QUERYEASIZE), and you want to indicate that their are no EAs, are you returning '4' (and not '0')? Returning zero is the most common coding error when handling EAs.

Valery Sedletski

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +2/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #79 on: June 24, 2019, 11:13:20 pm »
Quote from: Rich Walsh
Just to confirm: if your IFS receives a request for the total size of all EAs (FIL_QUERYEASIZE), and you want to indicate that their are no EAs, are you returning '4' (and not '0')? Returning zero is the most common coding error when handling EAs.

What do you mean? cbList field? It is always returned to be 0. (It is so in vboxfs.ifs, fat32.ifs and ramfs.ifs/emsfs.ifs). Why it should be 4?

I used the working code from fat32.ifs as a template, so it should work. I see no problems with displaying disks with Drives WPS object
in fat32.ifs. So, the problem is with something other.
« Last Edit: June 24, 2019, 11:22:12 pm by Valery Sedletski »

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 331
  • Karma: +23/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #80 on: June 25, 2019, 04:01:08 am »
Quote from: Rich Walsh
Just to confirm: if your IFS receives a request for the total size of all EAs (FIL_QUERYEASIZE), and you want to indicate that their are no EAs, are you returning '4' (and not '0')? Returning zero is the most common coding error when handling EAs.

What do you mean? cbList field? It is always returned to be 0. (It is so in vboxfs.ifs, fat32.ifs and ramfs.ifs/emsfs.ifs). Why it should be 4?

First, I assume that the 'cbList' field you are referring to is the one at the end of the FILESTATUS4 structure. In a properly functioning IFS, the following call will return a value of 4 in 'cbList' if 'pFilename' refers to a file with no EAs:

DosQueryPathInfo(pFilename, FIL_QUERYEASIZE, &fs4, sizeof(FILESTATUS4));

The documentation for this function and DosQueryFileInfo() is misleading. You would think it says that the size of the EAs will be returned - but that is incorrect. It actually returns the size of the FEA2LIST structure that is needed to hold all EAs. That structure is the size of all EAs *PLUS* a ULONG (named 'cbList') that identifies the total size of the variable-length structure. Thus, the correct size of an FEA2LIST for a file with no EAs is 0 + 4.

I knew this is a common error because I had to fix it in ISOFS in order to install ArcaOS from an ISO image file. The problem showed up when the installer tried to run IBM's ancient 'unpack.exe'. Even though it knew a given pack file had no EAs, it still queried them using the undocumented Level 4 argument (FIL_QUERYALLEAS). It interpreted a value of zero as 65536 and complained that the EAs were too large (since they can be no more than 65535 bytes). I doubt anyone had ever seen this error before because I know of no other program stupid enough to ask for something it knows doesn't exist.

Quote
So, the problem is with something other.

That may be true, but returning 0 in cblist is still incorrect. Write a simple test program and point it to a file on a JFS, HPFS, or even FAT drive to confirm this. BTW... AN's version of FAT32 also returns 4.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #81 on: June 25, 2019, 06:51:26 am »
Oh yes, EAs.

That's the worst designed programming interface I have ever come across. I am sure MS invented this bullshit in its darkest days. It's a steady resource of misinterpretation (as we all can see from this discussion).

But wait, parts of MMPM are just as bad.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #82 on: June 25, 2019, 07:03:36 am »
By the way: how does this go together with the fact that if you ask for a specific EA that then, the IFS does have to return an FEA(2) structure, basically returning the EA name that was requested (that is: szName is filled and cbName properly set to the length) but with EMPTY content (that is: cbValue is 0) ?
Returning nothing in this case is definitely WRONG.

That's why in the past I suggested to return a "sensible" size for exactly that FEA(2)LIST structure that would allow to return at least a couple of empty FEA(2) structures but Gregg stated that 0 should be returned.

As I said: a horrible programming interface.
« Last Edit: June 25, 2019, 08:17:27 am by Lars »

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #83 on: June 25, 2019, 10:30:01 am »
I just checked the FAT32 source code:

FS_FILEINFO (that should cover API DosGetFileInfo) returns 4 (for FIL_QUERYEASIZE and FIL_QUERYEASIZEL) if there are no EAs.

FS_PATHNFO (that should cover API DosGetPathInfo) returns 4 (for FIL_QUERYEASIZE and FIL_QUERYEASIZEL) if there are no EAs.


FS_FINDFIRST/FS_FINDNEXT (that should cover APIs DosFindFirst/DosFindNext) return 0 (for FIL_QUERYEASIZE and FIL_QUERYEASIZEL) if there are no EAs. This latter one was what Gregg requested as the response for files that do not have EAs.

Read this nonsense about "DosFindFirst" (quoting from the original programmer's documentation):
<quote>
For Level 2 File Information (ulInfoLevel == FIL_QUERYEASIZE) : On output, pfindbuf contains the FILEFINDBUF4 data structure without the last two fields: cchName and achName. This is used with EAs.

The cbList field contains the size, in bytes, of the file's entire EA set on disk. You can use this field to calculate the maximum size of the buffer needed for Level 3 file information. The size of the buffer required to hold the entire EA set is less than or equal to twice the size of the EA set on disk.
</quote>


« Last Edit: June 25, 2019, 10:47:58 am by Lars »

Valery Sedletski

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +2/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #84 on: June 25, 2019, 11:28:46 am »
2Rich Walsh & Lars Erdmann:

vboxfs.ifs was created looking at fat32.ifs and ramfs.ifs. So, they should behave similar way. Looking at fat32.ifs, I see:
1) In FS_PATHINFO and FS_FILEINFO: filestatus.cbList = sizeof(filestatus.cbList); (i.e., 4 for one ULONG).
2) In FS_FINDFIRST worker,

Code: ("C") [Select]
                     if (!f32Parms.fEAS || !HAS_EAS( pDir->fEAS ))
                        /* HACK: what we need to return here
                           is the FEALIST size of the list
                           that would be produced by usGetEmptyEAs !
                           for the time being: just tell the user
                           to allocate some reasonably sized amount of memory
                        */   
                        pfFind->cbList = 0;
                     else
                        {
                        rc = usGetEASize(pVolInfo, pFindInfo->pInfo->rgClusters[0], NULL,
                                         szLongName, &pfFind->cbList);
                        if (rc)
                           /* HACK: what we need to return here
                              is the FEALIST size of the list
                              that would be produced by usGetEmptyEAs !
                              for the time being: just tell the user
                              to allocate some reasonably sized amount of memory
                           */   
                           pfFind->cbList = 0;
                        rc = 0;
                        }

so, in absence of EA's, it returns cbList = 0; Yes, I looked primarily at FS_FINDFIRST. Rich, you seem to be right.
At least, FS_FILEINFO and FS_PATHINFO do thye same.

Quote
Quote
    So, the problem is with something other.

That may be true, but returning 0 in cblist is still incorrect. Write a simple test program and point it to a file on a JFS, HPFS, or even FAT drive to confirm this. BTW... AN's version of FAT32 also returns 4.

Yes, but the error occurs when FS_FINDFIRST (but not FS_FILEINFO / FS_PATHINFO) is called. It returns 0 in cbList in both vboxfs.ifs and fat32.ifs. Though, I see no problems at all
with fat32.ifs.
« Last Edit: June 25, 2019, 11:42:53 am by Valery Sedletski »

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #85 on: June 25, 2019, 02:34:33 pm »
ok, FS_FINDFIRST / FS_FINDNEXT should therefore also return pfFind->cbList = sizeof(pfFind->cbList) in case of no EAs (!HAS_EAS(pDir->fEAS)) or not activated EAs (!f32Parms.fEAS).

And then, I also wonder if "usGetEASize" returns the correct size. My gut feeling is that you will also need to add 4 on top of what it returns. And I have not yet understood if it really counts the bytes of all the FEA structures (appended by the EA name and the EA data) that it needs to create or only the bytes of the "naked" EA data which, again, would be wrong.

By the way: FS_PATHINFO and FS_FILEINFO do not honour the (!HAS_EAS(pDir->fEAS)) case (that is: the case where no EAs exist).
Unless "usGetEASize" properly handles this case ...

Here is the back and forth discussion regarding FS_FINDFIRST/FS_FINDNEXT:
http://trac.netlabs.org/fat32/ticket/55
« Last Edit: June 25, 2019, 02:47:35 pm by Lars »

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 331
  • Karma: +23/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #86 on: June 25, 2019, 07:35:51 pm »
Here is the back and forth discussion regarding FS_FINDFIRST/FS_FINDNEXT:
http://trac.netlabs.org/fat32/ticket/55

It's hard to understand why there should ever have been a "back and forth" in the first place: none of this is subject to debate.

You write a test program that incorporates the various scenarios, run it against one or more files on an *HPFS* formatted disk, then alter your own IFS to produce the same results as HPFS. Period, end of discussion. It doesn't matter if the results you get are good, bad, or just stupid - they are what the OS and programs like 'unpack.exe' were built to expect.

Note: I consider HPFS to be the reference IFS as it is the only filesystem that is pure IBM and has not been altered by well-meaning but potentially ill-informed third parties.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #87 on: June 26, 2019, 07:18:04 am »
It's hard to understand what good it does to return "4 Bytes" on a FIL_QUERYEASIZE if the return value is to serve as a measure of how many bytes you need to allocate when you want to actually request an EA.
As I said, you need an FEALIST with at least one entry plus the extra space for the requested EA Name to return the result. even if the EA is empty/does not exist. That is most certainly more than 4 bytes.

And guess what: obviously JFS.IFS returns 0 bytes on a DosFindFirst with FIL_QUERYEASIZE (that's what I read from Gregg's comments). But seemingly it would return 4 bytes on a DosQueryPathInfo with FIL_QUERYEASIZE. In short: the results are not even consistent in itself.

Who is using HPFS (and therefore testing with it) these days ? Nobody.

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 331
  • Karma: +23/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #88 on: June 26, 2019, 10:54:28 am »
It's hard to understand what good it does to return "4 Bytes" on a FIL_QUERYEASIZE if the return value is to serve as a measure of how many bytes you need to allocate when you want to actually request an EA. As I said, you need an FEALIST with at least one entry plus the extra space for the requested EA Name to return the result. even if the EA is empty/does not exist. That is most certainly more than 4 bytes.

How is the OS supposed to know that you're going to waste your time requesting some EA when it's telling you that there are NO EAs?? Returning '4' says "Ain't nothin' here, don't bother". If you're going to ignore what it says, then you'll just have to put forth some effort to calculate the appropriate size because it won't assist you in pursuing something that's pointless.

Quote
Who is using HPFS (and therefore testing with it) these days ? Nobody.

What kind of excuse is that for not bothering to find out how things are supposed to work? I have no use for willful ignorance, so I *did* try to find out. These are the results I got from running a simple DosFindFirst() against files with no EAs on various filesystems. The number is the value of FILEFINDBUF4.cbList:

FAT - 4
HPFS - 4
CDFS - 4
ISOFS - 4
JFS - 0
FAT32 - 0

You know what this shows? JFS & FAT32 are BROKEN. There was a _reason_ I suggested using HPFS as a reference: "[it] has not been altered by well-meaning but potentially ill-informed third parties" the way bootable JFS has been.

Valery Sedletski

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +2/-0
    • View Profile
Re: Shared folders IFS for VBox Additions, the test version
« Reply #89 on: June 26, 2019, 03:52:20 pm »
2Rich Walsh:
Yes,  indeed. I created the test example for DosFindFirst, and now see that different IFS'es return the following results:

HPFS - 4
JFS - 0
FAT32 - 0
RAMFS/EMSFS - 0
CDFS - 4
NDFS32 - 0

So yes, original IBM's IFS-es return 4, you're right