XDF disk extractor

[RickCHodgin]

Valery Sedletski, I can't figure out the algorithm to get the free diskspace.  I figured I would just count the number of free slots in the FAT12 table, but it's not working out.  This particular disk shows 265 free 512-byte slots in the FAT table, which results in about 60K more than it should have for the disk.

What is the algorithm?

[Valery Sedletski]

2RickHodgin: FAT32 has a special field in FSInfo sector (the next sector after the boot sector), which
contains the count of free clusters. exFAT has similar field, IIRC, too. But FAT12/FAT16
has no such field and it calculates free space each time it mounts/accesses the medium.

The free clusters are just counted, from the FAT table. You just run a loop through all FAT
entries, from 2, to TotalClusters + 2. FAT entries 0 and 1 are reserved, so the 1st data cluster
is 2.

TotalClusters = (TotalSectors - StartOfData) / SectorsPerCluster;

TotalSectors and SectorsPerCluster are the fields from BPB.

StartOfData = ReservedSectors + SectorsPerFAT *
NumberOfFats + (RootDirEntries * sizeof(DIRENTRY)) / BytesPerSector +
((RootDirEntries * sizeof(DIRENTRY)) % BytesPerSector) ? 1 : 0;

Where sizeof(DIRENTRY) == 32. -- This is as I see the code from fat32.ifs (FAT case).
All the above variables are BPB fields. So, when you calculated the number of free
clusters, you should multiply it by SectorsPerCluster, and divide by BytesPerSector,
so you get the free space in bytes.

[RickCHodgin]

2RickHodgin: FAT32 has a special field in FSInfo sector (the next sector after the boot sector), which
contains the count of free clusters. exFAT has similar field, IIRC, too. But FAT12/FAT16
has no such field and it calculates free space each time it mounts/accesses the medium.

The free clusters are just counted, from the FAT table. You just run a loop through all FAT
entries, from 2, to TotalClusters + 2. FAT entries 0 and 1 are reserved, so the 1st data cluster
is 2.

TotalClusters = (TotalSectors - StartOfData) / SectorsPerCluster;

TotalSectors and SectorsPerCluster are the fields from BPB.

StartOfData = ReservedSectors + SectorsPerFAT *
NumberOfFats + (RootDirEntries * sizeof(DIRENTRY)) / BytesPerSector +
((RootDirEntries * sizeof(DIRENTRY)) % BytesPerSector) ? 1 : 0;

Where sizeof(DIRENTRY) == 32. -- This is as I see the code from fat32.ifs (FAT case).
All the above variables are BPB fields. So, when you calculated the number of free
clusters, you should multiply it by SectorsPerCluster, and divide by BytesPerSector,
so you get the free space in bytes.

Hmmm.  That is what I'm doing, but it doesn't yield the same free space as what DOS 7 reports.  The number of free sectors seen in the FAT actually exceeds the disk capacity by about 60 KB.

Now, it's time to figure out what I'm doing wrong. :-)

UPDATE:  I figured out what I was doing wrong here, and it was an error in my logic.  But another problem remains (see the next post).

[RickCHodgin]

The free clusters are just counted, from the FAT table. You just run a loop through all FAT
entries, from 2, to TotalClusters + 2. FAT entries 0 and 1 are reserved, so the 1st data cluster
is 2.

TotalClusters = (TotalSectors - StartOfData) / SectorsPerCluster;
TotalSectors and SectorsPerCluster are the fields from BPB.

Note:  I use u8,u16,u32 for unsigned 8-bit, 16-bit, and 32-bit integers.  And s8,s16,s32 for signed 8-bit, 16-bit, and 32-bit integers.  I also use f32 and f64 for 32-bit and 64-bit floating point values.

I've worked and re-worked this algorithm several times.  I can't get it to match up with what DOS 7 reports.

Here's what I have right now.  I'm taking total_sector_count from the BPB's 16-bit value at offset 19.  The fat12 block is an unsigned char memory buffer that's loaded with the data beginning at sector 1 and running forward for the number of sectors specified at BPB's 16-bit value at offset 22 (sectors_per_fat).  These have been loaded from the XDF file image, and exist in memory.

My thinking is I take the total_sector_count provided in the BPB, subtract the 33 which are used by the FAT12 layout (boot + two FATs + root directory), and then take off the two reserved slots at the start of the FAT, and then proceed through the FAT counting until I reach my max offset, examining each 3-byte (2x 12 bit) entry I find for a 0 value (indicating it's unused / available / free) to get that count.  Then, to get total bytes, multiply that free sector count by 512 for bytes per free sector:

Code: [Select]

// Gets the free diskspace
s32 iiGetFreeDiskspace(u8* fat12, s32 total_sector_count)
{
    s32  lnThisSector, lnFreeSectors, lnMaxSector;
    u32  doubleSector;
    u8*  sectorPtr;

    // Initialize
    lnThisSector    = 0;
    lnFreeSectors   = 0;
    lnMaxSector     = total_sector_count - 33 - 2;

    // Iterate through the FAT
    for (sectorPtr = fat12 + 3; lnThisSector < lnMaxSector; lnThisSector += 2, sectorPtr += 3)
    {
        // Read this entry
        doubleSector = *(u32*)sectorPtr;

        // Check the two 12-bit values
        lnFreeSectors += ((doubleSector & 0x000fff) == 0);
        lnFreeSectors += ((doubleSector & 0xfff000) == 0);
    }

    // Indicate our count
    return(lnFreeSectors * 512);
}
This algorithm yields 159 sectors free, which is 81,408 bytes, a difference of 1,536 (3 sectors) from what DOS 7 is reporting.  So, where am I making my mistake?

[RickCHodgin]

I have the XDF extractor working.  Does anybody have an XDF image with sub-directories I can try?  The only ones I have are IBM PC DOS 7 XDF disks 2 thru 5, and they only have a few files in the root directory.

Where can I get some XDF images that have sub-directory contents?

[RickCHodgin]

I looked through my OS/2 diskettes and they all only have files in the root directory.  I found a few online and the same thing.

So, I think I'll break this project up into two projects.  I'll publish and release xdflibsf for an XDF file image to disk file extractor as a simple one-way utility.  It works on Windows right now.  I'll get the OS/2 version working next.  I'll need to know how to publish something on Hobbes.

xdflibsf source code:  http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/UTILS

Next, I'll begin work on an xdfmaker utility, which will be a file manager type system that mounts an XDF disk image and allows files to be added, edited, deleted, extracted, manage deleted files, volume labels, etc.

And, if I could get Arca Noae's support/help, I would be happy to write an xdfmount utility which would mount an XDF image as a RAM disk, allowing it to be changed/updated as needed, and then when it's unmounted it would write the changed contents back to the XDF disk image so it can be re-written to a real floppy disk if need be.  Also, an option to save it to another XDF filename.

[RickCHodgin]

I have it compiling and running in OS/2.  It displays the directory contents properly, but some things are aligned different.  It also doesn't work with the paths the way it does in Windows.  It seems like Borland C++ and/or OS/2 process things differently than Microsoft Visual C++ and/or Windows does.  Am tracking them down...

UPDATE:  I have it working.  Here's sample output from disk 2 of IBM PC DOS 7.0 XDF image:

Code: [Select]

[D:\XDFLIBSF] xdflibsf -x e:\xdf\144us2.xdf d:\XDF\DISK2\
XDF/LibSF for OS/2 -- XDF Extractor Utility

 Volume in XDF is DISK      2
 Directory of e:\xdf\144us2.xdf

 DOS1                522,033 -----a- 11-17-1994 13:00:00, extracted
 DOS2                479,333 -----a- 11-17-1994 13:00:00, extracted
 SHELL2              190,695 -----a- 11-17-1994 13:00:00, extracted
 PEN1                 84,299 -----a- 11-17-1994 13:00:00, extracted
 WUNDEL               99,615 -----a- 11-17-1994 13:00:00, extracted
 BACKUP1             409,193 -----a- 11-17-1994 13:00:00, extracted
        6 file(s)        1,785,168 bytes
                            77,312 bytes free

[D:\XDFLIBSF] _

I'll publish on Hobbes as soon as I figure out how.

[Valery Sedletski]

2Rick Hodgin: Nice, so you got the free space calculation to work without my help?

And, if I could get Arca Noae's support/help, I would be happy to write an xdfmount utility which would mount an XDF image as a RAM disk, allowing it to be changed/updated as needed, and then when it's unmounted it would write the changed contents back to the XDF disk image so it can be re-written to a real floppy disk if need be.  Also, an option to save it to another XDF filename.

I think, there is a better option than to mount an XDF file as a ramdisk. In fact, I already added
the support for different disk images to fat32.ifs. FAT32.IFS now is able to mount the FAT/FAT32/exFAT
disk images at the subdirectory on another FAT/FAT32/exFAT file system, serving as a mount point. Also, with my
loop.add block device driver, I can mount any filesystem, contained in a file, to a drive letter. This include
not only FAT/FAT32/exFAT filesystems, but any other FS having an IFS driver for OS/2. Except for
FAT/FAT32/exFAT, it can mount the CD ISO images, floppy images, any "raw" disk images and different
VM images, like .vhd's from VirtualPC, .vdi's from VirtualBox, VMWare's .vmdk images, etc (see details
in fat32.inf file from FAT32.IFS). For supporting VM disk images, I ported a library from QEMU, qemuimg.dll.
This DLL is loaded by cachef32.exe daemon, and the daemon executes read/write operations by request of
the IFS or of loop.add driver. Currently, it works fine for mounting FAT/FAT32/exFAT/CDFS file systems. JFS
and HPFS inside disk images are mounted successfully too, with the help of loop.add, but there are currently
problems with I/O request via strat2/strat3 routines: when you start copying files from/to the HPFS or JFS
image, it starts copying, but when cache is flushed, it hangs. Not resolved it yet. But for FAT/FAT32/exFAT/CDFS,
which don't use start2 and caching, everything is working fine, without hangs.

So far, the qemuimg.dll lib supports the following disk image formats:

1) RAW disk images, including diskette and CD ISO images
2) Bochs VM images
3) Linux CLOOP (Cryptoloop) images, used by Knoppix LiveCD
4) Macintosh .dmg
5) VirtualPC .vhd
6) VMWare .vmdk
7) Parallels
QEMU vvfat, qcow, qcow2 formats
9) VirtualBox .vdi

Also, I planned to try adding support for compressed .dsk floppy images from IBM.
So, I'd suggest you to add the XDF support to this lib too. So, qemuimg.dll lib could
be reused by many different programs in theory, not only fat32.ifs/loop.add.

[RickCHodgin]

2Rick Hodgin: Nice, so you got the free space calculation to work without my help?

I don't think so.  I think there's something happening I don't know about.  Either that or I've identified a flaw in DOS's expression of XDF free space on floppy disks.  I am counting the number of unused sectors, and multiplying by 512, and it yields a value different than DOS 7 is reporting.  So, either there's something they know that I don't, or I've exposed a bug/flaw in their stuff.  I'm guessing my knowledge is lacking. :-)

And, if I could get Arca Noae's support/help, I would be happy to write an xdfmount utility which would mount an XDF image as a RAM disk, allowing it to be changed/updated as needed, and then when it's unmounted it would write the changed contents back to the XDF disk image so it can be re-written to a real floppy disk if need be.  Also, an option to save it to another XDF filename.

I think, there is a better option than to mount an XDF file as a ramdisk. In fact, I already added
the support for different disk images to fat32.ifs. FAT32.IFS now is able to mount the FAT/FAT32/exFAT
disk images at the subdirectory on another FAT/FAT32/exFAT file system, serving as a mount point. Also, with my
loop.add block device driver, I can mount any filesystem, contained in a file, to a drive letter. This include
not only FAT/FAT32/exFAT filesystems, but any other FS having an IFS driver for OS/2. Except for
FAT/FAT32/exFAT, it can mount the CD ISO images, floppy images, any "raw" disk images and different
VM images, like .vhd's from VirtualPC, .vdi's from VirtualBox, VMWare's .vmdk images, etc (see details
in fat32.inf file from FAT32.IFS). For supporting VM disk images, I ported a library from QEMU, qemuimg.dll.
This DLL is loaded by cachef32.exe daemon, and the daemon executes read/write operations by request of
the IFS or of loop.add driver. Currently, it works fine for mounting FAT/FAT32/exFAT/CDFS file systems. JFS
and HPFS inside disk images are mounted successfully too, with the help of loop.add, but there are currently
problems with I/O request via strat2/strat3 routines: when you start copying files from/to the HPFS or JFS
image, it starts copying, but when cache is flushed, it hangs. Not resolved it yet. But for FAT/FAT32/exFAT/CDFS,
which don't use start2 and caching, everything is working fine, without hangs.

So far, the qemuimg.dll lib supports the following disk image formats:

1) RAW disk images, including diskette and CD ISO images
2) Bochs VM images
3) Linux CLOOP (Cryptoloop) images, used by Knoppix LiveCD
4) Macintosh .dmg
5) VirtualPC .vhd
6) VMWare .vmdk
7) Parallels
QEMU vvfat, qcow, qcow2 formats
9) VirtualBox .vdi

Also, I planned to try adding support for compressed .dsk floppy images from IBM.
So, I'd suggest you to add the XDF support to this lib too. So, qemuimg.dll lib could
be reused by many different programs in theory, not only fat32.ifs/loop.add.

You probably already have support.  The XDF file format is simply 23 sectors per track, and a normal floppy (2 heads, 80 tracks).  It is a literal FAT12 format with logical sector numbers provided for from the start of data through to the end (per the BPB's total sectors value).

If you extended any existing floppy driver to recognize and mount those formats for 1.2 MB, 1.44 MB and 2.88 MB, it would probably already work with your existing logic.

[Valery Sedletski]

PS: I have some XDF images, but only those available on OS/2 installation disks (Merlin, Warp3).
These are the same you have, so they doesn't contain subdirectories.

[RickCHodgin]

PS: I have some XDF images, but only those available on OS/2 installation disks (Merlin, Warp3).
These are the same you have, so they doesn't contain subdirectories.

I appreciate your help, Valery.  It's been nice to have that little extra insurance to nudge me along. :-)

[Dave Yeo]

It's easy to publish something on Hobbes. Download the README.TEMPLATE from /pub/incoming, adjust the lines that need adjusting, rename it to the same name as your package, with txt rather then zip as a suffix and use ftp to upload both the zip and txt files to /pub/incoming. Best to stick to lower case as well.

You might want to also look at OpenWatcom for a compiler, Borland is old, doesn't support certain things easily and I don't think it works as a cross compiler.

[Valery Sedletski]

2Rick Hodgin: Indeed, I tried two 1.8 MB diskette images from Merlin, I mounted them
successfully with fat32.ifs+loop.add. I was able to copy the diskette contents out of
the image! So, it is working, though, copying is slow, for some reason. Maybe, this is
because of sector interleave?

Also, as far as i understand, these XDF diskettes have 128 bytes per sector? So, it's
working with such sectors! (I also successfully tested 2048-byte sectors. FAT32 can
mount CD's with FAT/FAT32/exFAT file system. And its FORMAT/CHKDSK/SYS utilities
also work with CDRW/DVD+-RW/BD-RE disks in packet mode. So, it is tested by me
too and is working).

But strange why then windows utilities have problems with XDF images? Is it only
because sector size is hardcoded in most of them to 512 bytes?

[David Graser]

You might want to also look at OpenWatcom for a compiler, Borland is old, doesn't support certain things easily and I don't think it works as a cross compiler.

Latest beta can can be downloaded from

ftp://ftp.netlabs.org/incoming/open-watcom-c-os2-2.0-beta3.zip

[Valery Sedletski]

2Rick Hodgin:

> I appreciate your help, Valery.  It's been nice to have that little extra insurance to nudge me along. :-)

Thanks!