OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Utilities => Topic started by: RickCHodgin on November 02, 2018, 12:49:58 am

Title: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 12:49:58 am
Is there an XDF disk image extractor?  I've tried WinImage, WinEMT, DiskExplorer, and WinXDF.  DiskExplorer was able to read it, but it wouldn't read directories and wasn't able to reliably extract the files.  I couldn't find any tool to read the XDF format and export out the files.

I tried xdfcopy in OS/2, but I can't see where it extracts to a folder, but only to a physical floppy drive/disk.

I've ordered an external floppy drive for my laptop.  It will be here possibly this weekend, but probably Monday or Tuesday.
Title: Re: XDF disk extractor
Post by: David Graser on November 02, 2018, 01:04:23 am
Rick

Have you tried vfdisk.  Virtual floppy disk 6.0 and necessary control programs.

http://hobbes.nmsu.edu/download/pub/os2/system/drivers/filesys/vfdisk60.zip
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 01:50:49 am
Have you tried vfdisk.  Virtual floppy disk 6.0 and necessary control programs.

http://hobbes.nmsu.edu/download/pub/os2/system/drivers/filesys/vfdisk60.zip

I got it to work.  But, I can't find the settings to set the drive letter.  It automatically assigns the next available letter.  I assume it's a device=...vfdisk.sys option.  I tried "A: 1" but that didn't do it.  It still goes to the next letter.  Does anyone know?

UPDATE:  When I run xdfcopy with H:, it fails.  If I run it with A: it asks me to put the disk in the drive.  It doesn't seem to recognize the virtual floppy as a physical device.
Title: Re: XDF disk extractor
Post by: David Graser on November 02, 2018, 01:54:42 am
I know when I installed WPS security on my system, it originally wanted a floppy disk drive.  I tried VFDisk and even though the virtual drive was not drive A or B, it had no problem finding and using the Virtual floppy disk to install itselp.
Title: Re: XDF disk extractor
Post by: Neil Waldhauer on November 02, 2018, 02:13:14 am
This device driver provides support for eXtended Density Format (XDF) diskettes.

BASEDEV= ── XDFLOPPY.FLT ──────────────────────────────────────┤


Remarks

This device driver is required to use the XDFCOPY, command and to install this version of OS/2 using the OS/2 Installation diskettes.

Parameters

This device driver has no parameters.

Related Commands: BASEDEV, DEVICE, IBM1FLPY.ADD, IBM2FLPY.ADD, XDFCOPY.   
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 02, 2018, 02:17:32 am
There's a couple of other utilities to deal with xdf disks on Hobbes, http://hobbes.nmsu.edu/h-search.php?sh=1&button=Search&key=xdf&stype=all&sort=type_name&dir=%2F (http://hobbes.nmsu.edu/h-search.php?sh=1&button=Search&key=xdf&stype=all&sort=type_name&dir=%2F) a port of mtools and another virtual disk I believe.
Note that A: and B: are hardwired to the physical floppies and can't be used for other things so we're limited to C:-Z:
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 04:19:29 am
I think for my first OS/2 utility, I'm going to write a decent XDF disk extractor / converter.

Does anyone have knowledge of the OS/2 console window API?  Or an example of a simple text-based utility using one?  I want to make it a full-screen console app.  Is it like DOS in that you just write to 0xb8000, or is it like Windows with AllocConsole() and WriteLn(), etc, all API-function based?

XDF is 23 logical 512-byte sectors per track (http://www.os2museum.com/wp/the-xdf-diskette-format/), with 39 128-byte physical sectors indicated for the track, broken out into an 8KB / 2 KB / 1 KB / 512 B format per track.  Additional info for physical access. (http://www.fdutils.linux.lu/Fdutils.html#SEC47)

I think I could write an XDF floppy driver for ES/1 with relative ease as it works even with BIOS.  Will try that on my IBM PC DOS 7 machine (80486DX 33).
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 04:42:25 am
There's a couple of other utilities to deal with xdf disks on Hobbes, http://hobbes.nmsu.edu/h-search.php?sh=1&button=Search&key=xdf&stype=all&sort=type_name&dir=%2F (http://hobbes.nmsu.edu/h-search.php?sh=1&button=Search&key=xdf&stype=all&sort=type_name&dir=%2F) a port of mtools and another virtual disk I believe.

I tried to install mtools, but couldn't get it installed.

Quote
Note that A: and B: are hardwired to the physical floppies and can't be used for other things so we're limited to C:-Z:

I'll have to change that for ES/2.  I actually want to give aliases to hard pathnames, so you can refer to things simply and as you want to.
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 02, 2018, 05:08:06 am
mtools.exe seems to run fine here. I take it you have libc installed? The symlinks won't work from cmd.exe.
The console window API should be similar to windows. If you have the toolkit installed, there is an example program, consolio which creates a full screen console.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 06:00:17 am
mtools.exe seems to run fine here. I take it you have libc installed? The symlinks won't work from cmd.exe.
The console window API should be similar to windows. If you have the toolkit installed, there is an example program, consolio which creates a full screen console.

Thank you, Dave.  Will check it out tomorrow.  My floppy drive won't arrive until Tuesday, so maybe my XDF extractor tool could be completed before then.

I wonder why IBM didn't just use file compression?  Seems a better solution to save disk quantities.
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 02, 2018, 06:41:30 am
Other then on the first few boot disks, most of the files are compressed. Couldn't get too fancy on a 386 with compression though. Use unpack.exe to decompress them.
I take it you're using an ISO. Wouldn't be easier to just examine the disk images in OS2IMAGE rather then the floppy images?
Title: Re: XDF disk extractor
Post by: Bogdan on November 02, 2018, 09:00:47 am
Is there an XDF disk image extractor?  I've tried WinImage, WinEMT, DiskExplorer, and WinXDF.  DiskExplorer was able to read it, but it wouldn't read directories and wasn't able to reliably extract the files.  I couldn't find any tool to read the XDF format and export out the files.
Most disk image tools work only with fixed 512 bytes/sector.

Quote
I tried xdfcopy in OS/2, but I can't see where it extracts to a folder, but only to a physical floppy drive/disk.
It copies from physical drive to an image or vice versa.

Quote
I've ordered an external floppy drive for my laptop.  It will be here possibly this weekend, but probably Monday or Tuesday.
External USB floppy drives will not work, because of 512 bytes/sector limitation used by their transfer method. External floppy drives connected through the dedicated external floppy connector found on ThinkPad port replicators or docking stations will do fine.

The only OS/2 tool that handles XDF images virtually is SVDISK: https://www.bmtmicro.com/BMTCatalog/os2/svdisk.html

Under Windows you can simply use Virtual PC which supports at least XDF image files - but not physically.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 01:49:09 pm
I take it you're using an ISO. Wouldn't be easier to just examine the disk images in OS2IMAGE rather then the floppy images?

I've recently learned about something called TTD (Time Travel Debugging (https://www.youtube.com/watch?v=BVslyei0804), which I think I mentioned elsewhere).  It's a system where the entire state of the CPU is captured (Microsoft uses a "soft machine" (like Bochs) to capture it, but it can also be captured using a facility/feature of the x86 CPU, which is the ability to single-step every instruction by setting a flag value to signal an INT 1 interrupt).

I'd like to ultimately develop a set of primitive tools to enable this capture ability (initially across a COM port at bootup, then eventually to local disk files once drivers are loaded), to allow everything the CPU does to be intercepted, with all memory reads/writes being captured, and then an optional bit of storage to capture physical register changes as well as flags.

By doing this, any OS to be examined for security research would only need to be booted and arrive at the user interaction state.  It would capture the entire range of everything in memory, how it got there, which instruction did it, and everything.  This would allow for a comprehensive post-mortem analysis of the system after the fact.  Every aspect of which logic was chosen at each branch, how structures were updated, etc., would be captured.

It would then become a process of mining that operation to follow branches algorithmically, to categorize them, prioritize them as items of interest (oft-called, central branching points, etc.), and to extrapolate from the code being run exactly what is happening where, why, when, and how often.

I've never done this before, so I'm charting new waters.

I would prefer to modify a version of Bochs to handle this all virtually for me, but I've never been able to get Bochs setup to properly run / boot / install OS/2.  It's always reported a fatal error at some point.

In the alternative, Microsoft uses a custom-built virtual machine for this purpose.  It is integrated seamlessly with Windows and allows the full host of host machine attributes being exposed tot he virtual machine, so that when you create a Window it actually displays on the screen, you can interact with keyboard, mouse, network, etc.  It's quite a thing.

In any event, my goal is that instruction-by-instruction capture.  If I can do it with INT 1 it would be great because that facility would work on any machine, but bet slow (probably 200-500x slower than normal, so a normal 45 second boot would take over 6 hours).  But, it would only need to be done one time on each fixpack revision once everything is setup properly.

We'll see.  I'm in uncharted waters here (at least with regards to my own experience / expertise).  I proceed now only operating in theory, and it will require trial and error.  I will post my efforts / attempts / results as I go though.

Remember also, the goal here is security research.  I'm looking for where there are fatal flaws that could impact users resulting in data corruption, or the ability for some malicious software to take over / damage the system in some way.
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 02, 2018, 03:31:04 pm
Interesting, going to be a lot of data to go through :) Still not sure about the need to unpack XDF files.
Someone did suggest the diunpack program which is referenced here, https://wiki.gentoo.org/wiki/QEMU/OS2WarpV3_guest (https://wiki.gentoo.org/wiki/QEMU/OS2WarpV3_guest). Search for fastkick141.zip on the net.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 03:35:47 pm
Interesting, going to be a lot of data to go through :) Still not sure about the need to unpack XDF files.
Someone did suggest the diunpack program which is referenced here, https://wiki.gentoo.org/wiki/QEMU/OS2WarpV3_guest (https://wiki.gentoo.org/wiki/QEMU/OS2WarpV3_guest). Search for fastkick141.zip on the net.

Right now I am looking for unpacked XDF files for IBM PC DOS 7 disks (https://winworldpc.com/product/pc-dos/7x) for my guest machine partition in VirtualBox.  I'm going to extract them, then try to create them as 2.88 MB floppies so they'll load like normal and have enough space for their weird format.

If I get my XDF extractor written, I can probably just truncate some of the files, recompress them, add them to the floppies by number, and then add an extra step at the end to copy over the portions which were truncated, so it would install on regular 1.44 MB floppies.  As long as it's just copying and not running the programs it installs, truncating a few hundred bytes on a few files here and there would work, knowing they'll be copied over later.
Title: Re: XDF disk extractor
Post by: xynixme on November 02, 2018, 03:43:31 pm
Is it like DOS in that you just write to 0xb8000, or is it like Windows with AllocConsole() and WriteLn(), etc, all API-function based?

https://www.os2world.com/forum/index.php/topic,1859.msg18915.html#msg18915 (https://www.os2world.com/forum/index.php/topic,1859.msg18915.html#msg18915)
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 02, 2018, 03:47:52 pm
Is it like DOS in that you just write to 0xb8000, or is it like Windows with AllocConsole() and WriteLn(), etc, all API-function based?

https://www.os2world.com/forum/index.php/topic,1859.msg18915.html#msg18915 (https://www.os2world.com/forum/index.php/topic,1859.msg18915.html#msg18915)

+1
Title: Re: XDF disk extractor
Post by: xynixme on November 02, 2018, 04:01:06 pm

Have you tried vfdisk.  Virtual floppy disk 6.0 and necessary control programs.

I got it to work.  But, I can't find the settings to set the drive letter.  It automatically assigns the next available letter.  I assume it's a device=...vfdisk.sys option.  I tried "A: 1" but that didn't do it.  It still goes to the next letter.  Does anyone know?

That's a rather old NetLabs RFC, FWIW.

Quote
RFC: the ability to assign a fixed drive letter, so it doesn't change depending on wether drive S: snd/or T: exists or not.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 03, 2018, 01:43:54 am
As it was said above, XDF format floppies work with an ISA floppy disk, only (driven
by ibm1flpy.add driver). XDF floppies work via a xdfloppy.flt filter driver, which works
on top of ibm1flpy.add. vfdisk.sys is a virtual floppy ramdisk driver. It is a pre-ADD-model
block device driver, so filters like xdfloppy.flt won't work with it. They work only with
ADD block drivers. So, xdfloppy.flt won't work with vfdisk.sys. Also, vfdisk.sys doesn't
support XDF itself.

vfdisk.sys has the vfctrl.exe command line utility to configure the emulated floppy disk
sizes. Also, there is a vfctrlpm.exe, which is a PM-based version of vfctrl.exe. The default
disk size can also be specified on a driver command line. The drive letter specified in the
driver command line, doesn't matter. The driver just uses the first available drive letter
after RESERVEDRIVELETTER.

To emulate XDF floppies, you can use the Super Virtual Disk (svdisk.sys). Though, it is
shareware, and makes a delay during the boot if it is unregistered. This is annoying. The
PM program used to configure it, also fails to start if it is unregistered, but the command
line program works fine. The nice feature of SVDISK is also the ability to use the HPFS-
formatted ramdisks. It has own disk formatting utility, which, unfortunately, works only
with SVDISK ramdisks. Also, I managed to format a virtual floppy disk as HPFS, then
did savedskf on it, and created a 1.44 MB HPFS floppy disk image. This disk image can
be then copied to a physical floppy disk with loaddskf.exe. So, this way it's possible to
create the HPFS-formatted floppies. The cool hack, BTW :)

There is also a diunpack.exe utility, which can extract the IBM *.dsk format disk images
(including the compressed ones). It works just like an archiver. So, it extracts a floppy image
to a subdirectory. But I doubt that diunpack.exe can handle XDF floppy images.

Mtools are used to access the FAT disk images, including floppies and hard disks. There
is also an USB flash-aware version written by Froloff, which is able to copy files from FAT/FAT32
flash sticks.

In mtools, you just specify a path to all images in its config file, and assigns drive letters to them.
The drive letters are internal (separate from OS/2 drive letters). In case of Froloff's USB-aware
version, you specify a USB MSD device Vendor ID and Product ID, instead of an image path.

Unfortunately, Froloff no more uses OS/2 and his site is down. But I could upload his mtools
to my ftp, if someone interested.

To use mtools, you need to put mtools.exe in the same directory together with mtools.conf,
on PATH. And create wrapper .cmd files like this:

Quote
/* */
Parse Arg argstr

'@set MTOOLSRC=./mtools.conf'
'@set COUNTRY=007,866'
'@msup mdir' argstr

-- this is for a "mdir" mtools subcommand. There is the "msup.exe" helper which
calls mtools with required parameters. As people said here above, a newer kLIBC-
based version uses kLIBC symlinks instead, but I did not tried it. Also, the disadvantage
of kLIBC symlinks is that they don't work in non-kLIBC-based programs, like CMD.EXE,
4OS/2 or FC/2. You still need wrapper .cmd files.

Though, as far as I can see, mtools won't help you to access the XDF floppy image contents.

PS: For accessing the disk images, including floppy images, CD iso images and VM images
of VBox/VirtualPC/QEMU/etc, you can use fat32.ifs with it's loop.add driver. But note that
it is in development, yet, and not everything is working at the moment. But accessing the
FAT/FAT32/exFAT/CDFS images works fine. But it does not support XDF floppy images as
well.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 03, 2018, 02:43:07 am
Here is Froloff's MTools version: ftp://osfree.org/upload/froloff/mtools-usb.zip
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 03, 2018, 03:55:55 am
Yea, I misread the info on Hobbes about the one version of mtools having XDF reading capability, it doesn't.
Title: Re: XDF disk extractor
Post by: Bogdan on November 03, 2018, 01:07:55 pm
As it was said above, XDF format floppies work with an ISA floppy disk, only (driven
by ibm1flpy.add driver). XDF floppies work via a xdfloppy.flt filter driver, which works
on top of ibm1flpy.add.
Almost, but ISA is a wrong term used here. In fact it will work with AT- and PS/2-compatible floppy controllers. I'm not sure about SCSI connections or SuperDisk devices. One of the reasons the CONFIG.SYS contains the following lines:
Code: [Select]
BASEDEV=IBM1FLPY.ADD
BASEDEV=IBM2FLPY.ADD
Righr, XDFLOPPY is just the filter driver. The XDF itself is implemented inside IBM1FLPY.ADD.

Quote
To emulate XDF floppies, you can use the Super Virtual Disk (svdisk.sys). Though, it is
shareware, and makes a delay during the boot if it is unregistered. This is annoying. The
PM program used to configure it, also fails to start if it is unregistered, but the command
line program works fine.
It's not expensive to register ;-)

Quote
The nice feature of SVDISK is also the ability to use the HPFS-
formatted ramdisks. It has own disk formatting utility, which, unfortunately, works only
with SVDISK ramdisks. Also, I managed to format a virtual floppy disk as HPFS, then
did savedskf on it, and created a 1.44 MB HPFS floppy disk image. This disk image can
be then copied to a physical floppy disk with loaddskf.exe. So, this way it's possible to
create the HPFS-formatted floppies. The cool hack, BTW :)
Is it possible to boot from them?

Further reading: http://www.os2museum.com/wp/the-xdf-diskette-format/ (http://www.os2museum.com/wp/the-xdf-diskette-format/)
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 03, 2018, 05:18:58 pm
I'm creating a utility called xdflibsf.  It should work in DOS or OS/2 and has simple syntax:

Code: [Select]
xdflibsf -b \input\path\ output.xdf    // Build an XDF file
xdflibsf -x input.xdf \output\path\    // Extract an XDF to disk

The \intput\path\ will serve as the root directory of the XDF disk.

It will initially only work with 3.5" formats.  It will extract files and write them.  I plan to eventually add a special utility to create a bootable XDF file with the indicated files on it.  A user will use xdflibsf to easiily build an XDF file, and then xdfcopy to physically put it onto a floppy disk.

In reading about XDF, it's taught me a lot about how the FDC works internally.

XDF is a most interesting disk format allowing 1,884,160 bytes per disk (80 tracks * 2 sides * (512+1024+2048+8192) bytes per sector for the non-uniform sector sizes).  And the original author was a genius in devising a way to interleave the tracks and read both sides of the disk track in only three revolutions even with the non-uniform sector sizes.  It means an effective throughput of ~22KB per three revolutions.  At 300-360 RPM, that equates to a theoretical maximum throughput of 39 seconds to read an entire disk, with that level being potentially achievable due to the genius interleaving layout.
Title: Re: XDF disk extractor
Post by: xynixme on November 03, 2018, 06:57:58 pm
Code: [Select]
xdflibsf -b \input\path\ output.xdf    // Build an XDF file
xdflibsf -x input.xdf \output\path\    // Extract an XDF to disk

Without expecting anything, you may want to consider arguments which are compatible with ZIP GUIs. Then it may be required to append one or more files, or to delete one or more files.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 03, 2018, 07:34:06 pm
Code: [Select]
xdflibsf -b \input\path\ output.xdf    // Build an XDF file
xdflibsf -x input.xdf \output\path\    // Extract an XDF to disk

Without expecting anything, you may want to consider arguments which are compatible with ZIP GUIs. Then it may be required to append one or more files, or to delete one or more files.

Sure.  What are the ones I should support?
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 03, 2018, 10:34:33 pm
2Bogdan Szmalcovnik:

Quote
Quote
BASEDEV=IBM1FLPY.ADD
BASEDEV=IBM2FLPY.ADD

Righr, XDFLOPPY is just the filter driver. The XDF itself is implemented inside IBM1FLPY.ADD.

Yes, IBM2FLPY.ADD is for PS/2 floppy drives. But if XDF is implemented inside ibm1flpy.add,
then why xdfloppy.flt is needed? I thought, all XDF suppport should be contained in one driver.

Quote
Quote
    To emulate XDF floppies, you can use the Super Virtual Disk (svdisk.sys). Though, it is
    shareware, and makes a delay during the boot if it is unregistered. This is annoying. The
    PM program used to configure it, also fails to start if it is unregistered, but the command
    line program works fine.

It's not expensive to register ;-)

Is it still sold? I thought, it's abandoned a long time ago...

Quote
Quote
    The nice feature of SVDISK is also the ability to use the HPFS-
    formatted ramdisks. It has own disk formatting utility, which, unfortunately, works only
    with SVDISK ramdisks. Also, I managed to format a virtual floppy disk as HPFS, then
    did savedskf on it, and created a 1.44 MB HPFS floppy disk image. This disk image can
    be then copied to a physical floppy disk with loaddskf.exe. So, this way it's possible to
    create the HPFS-formatted floppies. The cool hack, BTW :)

Is it possible to boot from them?

Why not? I think, it should work, though, I did not tried. The only thing you
need is to do sysinstx on it. Need to try...
Title: Re: XDF disk extractor
Post by: David Graser on November 03, 2018, 11:48:16 pm
2Bogdan Szmalcovnik:



Is it still sold? I thought, it's abandoned a long time ago...

[/quote]

It is listed at BMTMicro

https://www.bmtmicro.com/BMTCatalog/os2/svdisk.html

You may want to email them to see if it can still be bought and and a registration code sent.  Some of the programs they list are extremely old OS/2 programs.

I did buy ZTreeBold from them.
Title: Re: XDF disk extractor
Post by: Bogdan on November 04, 2018, 12:39:12 am
2Bogdan Szmalcovnik:

Quote
Quote
BASEDEV=IBM1FLPY.ADD
BASEDEV=IBM2FLPY.ADD

Righr, XDFLOPPY is just the filter driver. The XDF itself is implemented inside IBM1FLPY.ADD.

Yes, IBM2FLPY.ADD is for PS/2 floppy drives. But if XDF is implemented inside ibm1flpy.add,
then why xdfloppy.flt is needed? I thought, all XDF support should be contained in one driver.
Sorry, the source was from "The Guide to OS/2 Warp Device Drivers" (SG24-4627-00) page 143:

"The question is sometime raised regarding why we have both IBM1FLPY.ADD and IBM2FLPY.ADD instead of just using a single driver. The answer is that during XDF (extended Density Format) development, IBM needed to patch Micro Channel OEM (Other Equipment Manufacturer) ABIOS code but of course did not have the ABIOS source for OEM systems. So a change was made from using IBM2FLPY.ADD on OEM Micro Channel systems to using IBM1FLPY.ADD or in other words no longer using the ABIOS interface. This caused a change in the initialization to detect 3 types of systems, non-ABIOS, OEM ABIOS and IBM ABIOS. The floppy drivers were changed to load the appropriate floppy driver on the appropriate system. IBM development felt it was to cumbersome for Install to query Syslnit (part of the installation procedure) and to install the appropriate driver, so instead the intelligence was incorporated into the driver."
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 04, 2018, 02:34:13 am
2Bogdan: So, xdfloppy.flt is required to support XDF on OEM ABIOS machines with the
same ibm1flpy.add driver as with non-ABIOS machines? I.e., xdfloppy.flt handles the
differences between these two types of machines, so that, ibm1flpy.add is independent
from that? Or I misunderstand?
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 04, 2018, 05:03:53 am
It's working so far.  Here's the IBM PC DOS 7 disk 3 file contents extracted from the XDF format.  I have this format on a real floppy and was able to boot into my DOS 7 partition, run the XDF program, and then do a DIR A:\.

Here's my display:

Code: [Select]
XDFLibSF -- XDF Builder/Extractor Utility

 Volume in XDF is DISK      3
 Directory of g:\xdf\144us3.xdf

 SHELL1               48,117 11-17-1994 13:00:00 -----A-
 AV2                 183,919 11-17-1994 13:00:00 -----A-
 WBACKUP1            460,201 11-17-1994 13:00:00 -----A-
 BACKUP2             399,521 11-17-1994 13:00:00 -----A-
 STAC2               690,149 11-17-1994 13:00:00 -----A-
        5 file(s)        1,781,907 bytes
                            90,477 bytes free

Here's what DOS 7 displays:

Code: [Select]
A:\> DIR

 Volume in drive A is DISK      3
 Directory of A:\

SHELL1            48,117 11-17-94  1:00p
AV2              183,919 11-17-94  1:00p
WBACKUP1         460,201 11-17-94  1:00p
BACKUP2          399,521 11-17-94  1:00p
STAC2            690,149 11-17-94  1:00p
        5 file(s)        1,781,907 bytes
                            79,872 bytes free

A:\> _

My algorithm for calculating disk free space is incorrect.  It's not yet taking into account the sectors consumed by the root directory.  But, tomorrow's another day. :-)  Here's the source code (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/UTILS) so far.  I've only been developing it in Windows during debugging.  Will port to OS/2 when complete.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 04, 2018, 02:52:47 pm
Valery Sedletski, I can't figure out the algorithm to get the free diskspace.  I figured I would just count the number of free slots in the FAT12 table, but it's not working out.  This particular disk shows 265 free 512-byte slots in the FAT table, which results in about 60K more than it should have for the disk.

What is the algorithm?
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 04, 2018, 03:40:02 pm
2RickHodgin: FAT32 has a special field in FSInfo sector (the next sector after the boot sector), which
contains the count of free clusters. exFAT has similar field, IIRC, too. But FAT12/FAT16
has no such field and it calculates free space each time it mounts/accesses the medium.

The free clusters are just counted, from the FAT table. You just run a loop through all FAT
entries, from 2, to TotalClusters + 2. FAT entries 0 and 1 are reserved, so the 1st data cluster
is 2.

TotalClusters = (TotalSectors - StartOfData) / SectorsPerCluster;

TotalSectors and SectorsPerCluster are the fields from BPB.

StartOfData = ReservedSectors + SectorsPerFAT *
NumberOfFats + (RootDirEntries * sizeof(DIRENTRY)) / BytesPerSector +
((RootDirEntries * sizeof(DIRENTRY)) % BytesPerSector) ? 1 : 0;

Where sizeof(DIRENTRY) == 32. -- This is as I see the code from fat32.ifs (FAT case).
All the above variables are BPB fields. So, when you calculated the number of free
clusters, you should multiply it by SectorsPerCluster, and divide by BytesPerSector,
so you get the free space in bytes.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 04, 2018, 05:53:26 pm
2RickHodgin: FAT32 has a special field in FSInfo sector (the next sector after the boot sector), which
contains the count of free clusters. exFAT has similar field, IIRC, too. But FAT12/FAT16
has no such field and it calculates free space each time it mounts/accesses the medium.

The free clusters are just counted, from the FAT table. You just run a loop through all FAT
entries, from 2, to TotalClusters + 2. FAT entries 0 and 1 are reserved, so the 1st data cluster
is 2.

TotalClusters = (TotalSectors - StartOfData) / SectorsPerCluster;

TotalSectors and SectorsPerCluster are the fields from BPB.

StartOfData = ReservedSectors + SectorsPerFAT *
NumberOfFats + (RootDirEntries * sizeof(DIRENTRY)) / BytesPerSector +
((RootDirEntries * sizeof(DIRENTRY)) % BytesPerSector) ? 1 : 0;

Where sizeof(DIRENTRY) == 32. -- This is as I see the code from fat32.ifs (FAT case).
All the above variables are BPB fields. So, when you calculated the number of free
clusters, you should multiply it by SectorsPerCluster, and divide by BytesPerSector,
so you get the free space in bytes.

Hmmm.  That is what I'm doing, but it doesn't yield the same free space as what DOS 7 reports.  The number of free sectors seen in the FAT actually exceeds the disk capacity by about 60 KB.

Now, it's time to figure out what I'm doing wrong. :-)

UPDATE:  I figured out what I was doing wrong here, and it was an error in my logic.  But another problem remains (see the next post).
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 04, 2018, 08:08:00 pm
The free clusters are just counted, from the FAT table. You just run a loop through all FAT
entries, from 2, to TotalClusters + 2. FAT entries 0 and 1 are reserved, so the 1st data cluster
is 2.

TotalClusters = (TotalSectors - StartOfData) / SectorsPerCluster;
TotalSectors and SectorsPerCluster are the fields from BPB.

Note:  I use u8,u16,u32 for unsigned 8-bit, 16-bit, and 32-bit integers.  And s8,s16,s32 for signed 8-bit, 16-bit, and 32-bit integers.  I also use f32 and f64 for 32-bit and 64-bit floating point values.

I've worked and re-worked this algorithm several times.  I can't get it to match up with what DOS 7 reports.

Here's what I have right now.  I'm taking total_sector_count from the BPB's 16-bit value at offset 19.  The fat12 block is an unsigned char memory buffer that's loaded with the data beginning at sector 1 and running forward for the number of sectors specified at BPB's 16-bit value at offset 22 (sectors_per_fat).  These have been loaded from the XDF file image, and exist in memory.

My thinking is I take the total_sector_count provided in the BPB, subtract the 33 which are used by the FAT12 layout (boot + two FATs + root directory), and then take off the two reserved slots at the start of the FAT, and then proceed through the FAT counting until I reach my max offset, examining each 3-byte (2x 12 bit) entry I find for a 0 value (indicating it's unused / available / free) to get that count.  Then, to get total bytes, multiply that free sector count by 512 for bytes per free sector:

Code: [Select]
// Gets the free diskspace
s32 iiGetFreeDiskspace(u8* fat12, s32 total_sector_count)
{
    s32  lnThisSector, lnFreeSectors, lnMaxSector;
    u32  doubleSector;
    u8*  sectorPtr;

    // Initialize
    lnThisSector    = 0;
    lnFreeSectors   = 0;
    lnMaxSector     = total_sector_count - 33 - 2;

    // Iterate through the FAT
    for (sectorPtr = fat12 + 3; lnThisSector < lnMaxSector; lnThisSector += 2, sectorPtr += 3)
    {
        // Read this entry
        doubleSector = *(u32*)sectorPtr;

        // Check the two 12-bit values
        lnFreeSectors += ((doubleSector & 0x000fff) == 0);
        lnFreeSectors += ((doubleSector & 0xfff000) == 0);
    }

    // Indicate our count
    return(lnFreeSectors * 512);
}

This algorithm yields 159 sectors free, which is 81,408 bytes, a difference of 1,536 (3 sectors) from what DOS 7 is reporting.  So, where am I making my mistake?
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 04, 2018, 11:07:33 pm
I have the XDF extractor working.  Does anybody have an XDF image with sub-directories I can try?  The only ones I have are IBM PC DOS 7 XDF disks 2 thru 5, and they only have a few files in the root directory.

Where can I get some XDF images that have sub-directory contents?
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 04, 2018, 11:45:59 pm
I looked through my OS/2 diskettes and they all only have files in the root directory.  I found a few online and the same thing.

So, I think I'll break this project up into two projects.  I'll publish and release xdflibsf for an XDF file image to disk file extractor as a simple one-way utility.  It works on Windows right now.  I'll get the OS/2 version working next.  I'll need to know how to publish something on Hobbes.

xdflibsf source code:  http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/UTILS

Next, I'll begin work on an xdfmaker utility, which will be a file manager type system that mounts an XDF disk image and allows files to be added, edited, deleted, extracted, manage deleted files, volume labels, etc.

And, if I could get Arca Noae's support/help, I would be happy to write an xdfmount utility which would mount an XDF image as a RAM disk, allowing it to be changed/updated as needed, and then when it's unmounted it would write the changed contents back to the XDF disk image so it can be re-written to a real floppy disk if need be.  Also, an option to save it to another XDF filename.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 12:30:44 am
I have it compiling and running in OS/2.  It displays the directory contents properly, but some things are aligned different.  It also doesn't work with the paths the way it does in Windows.  It seems like Borland C++ and/or OS/2 process things differently than Microsoft Visual C++ and/or Windows does.  Am tracking them down...

UPDATE:  I have it working.  Here's sample output from disk 2 of IBM PC DOS 7.0 XDF image:

Code: [Select]
[D:\XDFLIBSF] xdflibsf -x e:\xdf\144us2.xdf d:\XDF\DISK2\
XDF/LibSF for OS/2 -- XDF Extractor Utility

 Volume in XDF is DISK      2
 Directory of e:\xdf\144us2.xdf

 DOS1                522,033 -----a- 11-17-1994 13:00:00, extracted
 DOS2                479,333 -----a- 11-17-1994 13:00:00, extracted
 SHELL2              190,695 -----a- 11-17-1994 13:00:00, extracted
 PEN1                 84,299 -----a- 11-17-1994 13:00:00, extracted
 WUNDEL               99,615 -----a- 11-17-1994 13:00:00, extracted
 BACKUP1             409,193 -----a- 11-17-1994 13:00:00, extracted
        6 file(s)        1,785,168 bytes
                            77,312 bytes free

[D:\XDFLIBSF] _

I'll publish on Hobbes as soon as I figure out how.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 01:15:06 am
2Rick Hodgin: Nice, so you got the free space calculation to work without my help?

Quote
And, if I could get Arca Noae's support/help, I would be happy to write an xdfmount utility which would mount an XDF image as a RAM disk, allowing it to be changed/updated as needed, and then when it's unmounted it would write the changed contents back to the XDF disk image so it can be re-written to a real floppy disk if need be.  Also, an option to save it to another XDF filename.

I think, there is a better option than to mount an XDF file as a ramdisk. In fact, I already added
the support for different disk images to fat32.ifs. FAT32.IFS now is able to mount the FAT/FAT32/exFAT
disk images at the subdirectory on another FAT/FAT32/exFAT file system, serving as a mount point. Also, with my
loop.add block device driver, I can mount any filesystem, contained in a file, to a drive letter. This include
not only FAT/FAT32/exFAT filesystems, but any other FS having an IFS driver for OS/2. Except for
FAT/FAT32/exFAT, it can mount the CD ISO images, floppy images, any "raw" disk images and different
VM images, like .vhd's from VirtualPC, .vdi's from VirtualBox, VMWare's .vmdk images, etc (see details
in fat32.inf file from FAT32.IFS). For supporting VM disk images, I ported a library from QEMU, qemuimg.dll.
This DLL is loaded by cachef32.exe daemon, and the daemon executes read/write operations by request of
the IFS or of loop.add driver. Currently, it works fine for mounting FAT/FAT32/exFAT/CDFS file systems. JFS
and HPFS inside disk images are mounted successfully too, with the help of loop.add, but there are currently
problems with I/O request via strat2/strat3 routines: when you start copying files from/to the HPFS or JFS
image, it starts copying, but when cache is flushed, it hangs. Not resolved it yet. But for FAT/FAT32/exFAT/CDFS,
which don't use start2 and caching, everything is working fine, without hangs.

So far, the qemuimg.dll lib supports the following disk image formats:

1) RAW disk images, including diskette and CD ISO images
2) Bochs VM images
3) Linux CLOOP (Cryptoloop) images, used by Knoppix LiveCD
4) Macintosh .dmg
5) VirtualPC .vhd
6) VMWare .vmdk
7) Parallels
8) QEMU vvfat, qcow, qcow2 formats
9) VirtualBox .vdi

Also, I planned to try adding support for compressed .dsk floppy images from IBM.
So, I'd suggest you to add the XDF support to this lib too. So, qemuimg.dll lib could
be reused by many different programs in theory, not only fat32.ifs/loop.add.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 01:27:58 am
2Rick Hodgin: Nice, so you got the free space calculation to work without my help?

I don't think so.  I think there's something happening I don't know about.  Either that or I've identified a flaw in DOS's expression of XDF free space on floppy disks.  I am counting the number of unused sectors, and multiplying by 512, and it yields a value different than DOS 7 is reporting.  So, either there's something they know that I don't, or I've exposed a bug/flaw in their stuff.  I'm guessing my knowledge is lacking. :-)

Quote
Quote
And, if I could get Arca Noae's support/help, I would be happy to write an xdfmount utility which would mount an XDF image as a RAM disk, allowing it to be changed/updated as needed, and then when it's unmounted it would write the changed contents back to the XDF disk image so it can be re-written to a real floppy disk if need be.  Also, an option to save it to another XDF filename.

I think, there is a better option than to mount an XDF file as a ramdisk. In fact, I already added
the support for different disk images to fat32.ifs. FAT32.IFS now is able to mount the FAT/FAT32/exFAT
disk images at the subdirectory on another FAT/FAT32/exFAT file system, serving as a mount point. Also, with my
loop.add block device driver, I can mount any filesystem, contained in a file, to a drive letter. This include
not only FAT/FAT32/exFAT filesystems, but any other FS having an IFS driver for OS/2. Except for
FAT/FAT32/exFAT, it can mount the CD ISO images, floppy images, any "raw" disk images and different
VM images, like .vhd's from VirtualPC, .vdi's from VirtualBox, VMWare's .vmdk images, etc (see details
in fat32.inf file from FAT32.IFS). For supporting VM disk images, I ported a library from QEMU, qemuimg.dll.
This DLL is loaded by cachef32.exe daemon, and the daemon executes read/write operations by request of
the IFS or of loop.add driver. Currently, it works fine for mounting FAT/FAT32/exFAT/CDFS file systems. JFS
and HPFS inside disk images are mounted successfully too, with the help of loop.add, but there are currently
problems with I/O request via strat2/strat3 routines: when you start copying files from/to the HPFS or JFS
image, it starts copying, but when cache is flushed, it hangs. Not resolved it yet. But for FAT/FAT32/exFAT/CDFS,
which don't use start2 and caching, everything is working fine, without hangs.

So far, the qemuimg.dll lib supports the following disk image formats:

1) RAW disk images, including diskette and CD ISO images
2) Bochs VM images
3) Linux CLOOP (Cryptoloop) images, used by Knoppix LiveCD
4) Macintosh .dmg
5) VirtualPC .vhd
6) VMWare .vmdk
7) Parallels
8) QEMU vvfat, qcow, qcow2 formats
9) VirtualBox .vdi

Also, I planned to try adding support for compressed .dsk floppy images from IBM.
So, I'd suggest you to add the XDF support to this lib too. So, qemuimg.dll lib could
be reused by many different programs in theory, not only fat32.ifs/loop.add.

You probably already have support.  The XDF file format is simply 23 sectors per track, and a normal floppy (2 heads, 80 tracks).  It is a literal FAT12 format with logical sector numbers provided for from the start of data through to the end (per the BPB's total sectors value).

If you extended any existing floppy driver to recognize and mount those formats for 1.2 MB, 1.44 MB and 2.88 MB, it would probably already work with your existing logic.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 01:32:32 am
PS: I have some XDF images, but only those available on OS/2 installation disks (Merlin, Warp3).
These are the same you have, so they doesn't contain subdirectories.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 01:35:29 am
PS: I have some XDF images, but only those available on OS/2 installation disks (Merlin, Warp3).
These are the same you have, so they doesn't contain subdirectories.

I appreciate your help, Valery.  It's been nice to have that little extra insurance to nudge me along. :-)
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 05, 2018, 01:38:08 am
It's easy to publish something on Hobbes. Download the README.TEMPLATE from /pub/incoming, adjust the lines that need adjusting, rename it to the same name as your package, with txt rather then zip as a suffix and use ftp to upload both the zip and txt files to /pub/incoming. Best to stick to lower case as well.

You might want to also look at OpenWatcom for a compiler, Borland is old, doesn't support certain things easily and I don't think it works as a cross compiler.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 01:51:32 am
2Rick Hodgin: Indeed, I tried two 1.8 MB diskette images from Merlin, I mounted them
successfully with fat32.ifs+loop.add. I was able to copy the diskette contents out of
the image! So, it is working, though, copying is slow, for some reason. Maybe, this is
because of sector interleave?

Also, as far as i understand, these XDF diskettes have 128 bytes per sector? So, it's
working with such sectors! (I also successfully tested 2048-byte sectors. FAT32 can
mount CD's with FAT/FAT32/exFAT file system. And its FORMAT/CHKDSK/SYS utilities
also work with CDRW/DVD+-RW/BD-RE disks in packet mode. So, it is tested by me
too and is working).

But strange why then windows utilities have problems with XDF images? Is it only
because sector size is hardcoded in most of them to 512 bytes?
Title: Re: XDF disk extractor
Post by: David Graser on November 05, 2018, 01:51:55 am

You might want to also look at OpenWatcom for a compiler, Borland is old, doesn't support certain things easily and I don't think it works as a cross compiler.

Latest beta can can be downloaded from

ftp://ftp.netlabs.org/incoming/open-watcom-c-os2-2.0-beta3.zip
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 01:53:09 am
2Rick Hodgin:

> I appreciate your help, Valery.  It's been nice to have that little extra insurance to nudge me along. :-)

Thanks! :)
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:01:23 am
2Rick Hodgin: Indeed, I tried two 1.8 MB diskette images from Merlin, I mounted them
successfully with fat32.ifs+loop.add. I was able to copy the diskette contents out of
the image! So, it is working, though, copying is slow, for some reason. Maybe, this is
because of sector interleave?

It shouldn't be slow if you're pulling from an XDF file.  The utility I wrote uses C Standard Library functions fopen(), fseek(), fread() and fwrite(), and I am able to extract the entire 1.8 MB in six files in about 1.5 seconds in OS/2 Warp 4.52 in VirtualBox.

Quote
Also, as far as i understand, these XDF diskettes have 128 bytes per sector? So, it's
working with such sectors! (I also successfully tested 2048-byte sectors. FAT32 can
mount CD's with FAT/FAT32/exFAT file system. And its FORMAT/CHKDSK/SYS utilities
also work with CDRW/DVD+-RW/BD-RE disks in packet mode. So, it is tested by me
too and is working).

But strange why then windows utilities have problems with XDF images? Is it only
because sector size is hardcoded in most of them to 512 bytes?

The XDF format "tricks" the FDC by indicating on the track record that it has 39 128-byte sectors.  But, the FDC does not rely upon that track record setting for each of the sectors within, and instead the OS sends sector size information for each sector.  So, it uses four separate sizes for the space around the track, resulting in one 512-byte sector, one 1024-byte sector, one 2048-byte sector, and one 8192-byte sector.

But, logically, the entire image inside the XDF is arranged as 23 sectors of 512 bytes.  Track 0 only uses 19 sectors, however, so the last 4 are simply ignored.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 02:01:36 am
Regarding the OpenWatcom 2.0: I just tried a daily snapshot of OW 2.0 from here:

https://efbe.musca.uberspace.de/

I just compiled fat32.ifs with it successfully. Previously, I tried another OW 2.0 from
GitHub, and had problems with it. The FS utility DLL won't build. It contains a 32-bit
source which implements CHKDSK/FORMAT/SYS/RECOVER routines. These routines
are 16-bits. So, the 16->32-bit wrappers created with 32-bit C compiler, doesn't
compile. The 32-bit compiler is broken. But the build from the former site is working
fine. The only bad thing I noticed is that it seems to compile significantly slower
than OW 1.9.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:04:38 am

You might want to also look at OpenWatcom for a compiler, Borland is old, doesn't support certain things easily and I don't think it works as a cross compiler.

Latest beta can can be downloaded from

ftp://ftp.netlabs.org/incoming/open-watcom-c-os2-2.0-beta3.zip

Thanks, David!  I have used Open Watcom before for my ES/1 development in IBM PC DOS 7.  Look forward to seeing it in OS/2.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 02:17:51 am
Interesting.

> The XDF format "tricks" the FDC by indicating on the track record that it has 39 128-byte sectors.  But, the FDC does not rely upon that track record setting for each of the sectors within, and instead the OS sends sector size information for each sector.  So, it uses four separate sizes for the space around the track, resulting in one 512-byte sector, one 1024-byte sector, one 2048-byte sector, and one 8192-byte sector.

So, this is only a trick for programming the FDC (at physical level).

> But, logically, the entire image inside the XDF is arranged as 23 sectors of 512 bytes.  Track 0 only uses 19 sectors, however, so the last 4 are simply ignored.

So, on logical level, XDF image is just an usual sequence of 23*80*2 sectors 512 bytes each == 1884160 bytes. So,
the problem of windows image programs is that it has 23 sectors, not 18?

Hm, but why then diunpack doesn't work with these images? It should.

Quote
L:\data\vm>diunpack WIN1.DSK -d aaa\


DIUNPACK.EXE Release 3.00 12-20-95
Copyright (C) IBM Corporation 1995
All rights reserved.


Reading disk image....


ERROR: Image file fat is corrupt
[/quite]
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:27:03 am
It's easy to publish something on Hobbes. Download the README.TEMPLATE from /pub/incoming, adjust the lines that need adjusting, rename it to the same name as your package, with txt rather then zip as a suffix and use ftp to upload both the zip and txt files to /pub/incoming. Best to stick to lower case as well.

Thank you, Dave.  Doing so now.  It will have two executables:

Code: [Select]
os2\xdflibsf.exe
win32\xdflibsf.exe

What other files should I add to it?  A readme.txt, etc?
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 02:29:04 am
Quote
L:\data\vm>vfctrl ?
VFDisk Control Program
  The possible commands are:
    Query VFDisks:  d:\os2\vfctrl.exe
    Help         :  d:\os2\vfctrl.exe ?
    Query VFDisk :  d:\os2\vfctrl.exe X:
    Set VFDisk   :  d:\os2\vfctrl.exe X: <nn>
       where <nn> is one of the following:
         0 EJECT
         1 1440 1440K 1440KB 1.44 1.44M 1.44MB
         2 1200 1200K 1200KB 1.2 1.2M 1.2MB
         3 720 720K 720KB
         4 2880 2880K 2880KB 2.88 2.88M 2.88MB
         5 360 360K 360KB
         6 1840 1840K 1840KB 1.84 1.84M 1.84MB

L:\data\vm>vfctrl x: 6

L:\data\vm>loaddskf WIN1.DSK x:

Unsupported disk type

So, vfdisk doesn't support XDF disks too, and disks of "6" format having
1840 bytes per sector, differ from XDF ones. But I think, it's possible to add
support of 23 sectors * 80 tracks * 2 sides disks to vfdisk, and it should
work.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:31:00 am
So, on logical level, XDF image is just an usual sequence of 23*80*2 sectors 512 bytes each == 1884160 bytes. So,
the problem of windows image programs is that it has 23 sectors, not 18?

Hm, but why then diunpack doesn't work with these images? It should.

Do not know.  But if you look at the file size, it's 80 * 2 * (512 + 1024 + 2048 + 8192).  They are literally a byte-per-byte transfer of the full data across the track in logical form.  It's only when they're written to a physical device that they have unusual data layouts.

Quote
Code: [Select]
L:\data\vm>diunpack WIN1.DSK -d aaa\

DIUNPACK.EXE Release 3.00 12-20-95
Copyright (C) IBM Corporation 1995
All rights reserved.

Reading disk image....

ERROR: Image file fat is corrupt

Is the .DSK file there a 1.44 MB disk image, or a 1.88 MB XDF image?  For IBM PC DOS 7.0 the first disk is a .DSK image, and it's a 1.44 MB image in standard format.  The disks 2 thru 5 are XDF format.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 02:44:25 am
Yes, the same in OS/2. The boot disks 0-2 are standard 1.44 images, and disks 3-...N are
1884160 bytes == 1.796875 MB XDF images. diunpack won't extract them.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:55:12 am
I'm made the XDF/LibSF utility available for direct download.  If you want to try it out please do.  Please report any bugs.

Direct download:  http://www.libsf.org/es2/xdflibsf.zip

Code: [Select]
os2\
os2\xdflibsf.exe
win32\
win32\xdflibsf.exe
readme.txt
xdflibsf_info.txt
Title: Re: XDF disk extractor
Post by: David Graser on November 05, 2018, 02:55:29 am

You might want to also look at OpenWatcom for a compiler, Borland is old, doesn't support certain things easily and I don't think it works as a cross compiler.

Latest beta can can be downloaded from

ftp://ftp.netlabs.org/incoming/open-watcom-c-os2-2.0-beta3.zip

Thanks, David!  I have used Open Watcom before for my ES/1 development in IBM PC DOS 7.  Look forward to seeing it in OS/2.

Rick, I found a site that may be of interest to you.

http://www.os2site.com/sw/upgrades/index.html

Although I am not a programmer, I am enjoying the sharing of information you and Valery are doing.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 03:07:19 am
Rick, I tried to extract this file: ftp://osfree.org/upload/win1.dsk with your
program and see that the setup.inf file corrupted. If I extract it with my
method, the same file contents are ok, it's a simple text file. But most
files seem to be extracted fine.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 03:11:19 am
Rick, I tried to extract this file: ftp://osfree.org/upload/win1.dsk with your
program and see that the setup.inf file corrupted. If I extract it with my
method, the same file contents are ok, it's a simple text file. But most
files seem to be extracted fine.

I'll take a look at it.  Was it the correct file size?  If so, I may not be honoring the number of reserved sectors properly and it could be skewed by reading the wrong sectors on the disk.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 03:12:47 am
Rick, I found a site that may be of interest to you.

http://www.os2site.com/sw/upgrades/index.html

Although I am not a programmer, I am enjoying the sharing of information you and Valery are doing.

Thank you, David.  I can't speak for Valery, but one of my most enjoyable things is to do things for others.  My goals are to empower people, and not hinder or hobble them behind EULAs and the like, but to give them source code, teach them how to use it, allow them to move in the ways they're best suited / desiring to do.

We are all stronger when we help each other achieve more than we can do on our own.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 03:15:17 am
Yes, the file size is 1884160 (you can download it and see). The reserved sectors are == 1.
It seems to be the same for all diskettes.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 03:25:10 am
Yes, the file size is 1884160 (you can download it and see). The reserved sectors are == 1.
It seems to be the same for all diskettes.

Can you post the first few lines of the correct setup.inf file?
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 03:29:45 am
2Rick:

Quote
[setup]
    help = setup.hlp

;   Place any programs here that should be run at the end of setup.
;   These apps will be run in order of their appearance here.
[run]

...

You can try searching these strings in the image with a hex viewer.
Title: Re: XDF disk extractor
Post by: David Graser on November 05, 2018, 03:39:31 am


So, vfdisk doesn't support XDF disks too, and disks of "6" format having
1840 bytes per sector, differ from XDF ones. But I think, it's possible to add
support of 23 sectors * 80 tracks * 2 sides disks to vfdisk, and it should
work.
[/quote]

Rick the source code is here if you want to take a look.

http://trac.netlabs.org/vfdisk
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 05, 2018, 03:53:47 am
It's easy to publish something on Hobbes. Download the README.TEMPLATE from /pub/incoming, adjust the lines that need adjusting, rename it to the same name as your package, with txt rather then zip as a suffix and use ftp to upload both the zip and txt files to /pub/incoming. Best to stick to lower case as well.

Thank you, Dave.  Doing so now.  It will have two executables:

Code: [Select]
os2\xdflibsf.exe
win32\xdflibsf.exe

What other files should I add to it?  A readme.txt, etc?

Readme and a license at least.
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 05, 2018, 04:02:38 am
About OpenWatcom, the version on Github is a fork. One of the developers got impatient with the review process etc and wanted to focus on 64bit and forked the code, he doesn't care about OS/2 as far as I know. Really they should have agreed to using different versioning.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 04:16:18 am
2Rick:

Quote
Hm, but why then diunpack doesn't work with these images? It should.

Quote
    L:\data\vm>diunpack WIN1.DSK -d aaa\


    DIUNPACK.EXE Release 3.00 12-20-95
    Copyright (C) IBM Corporation 1995
    All rights reserved.


    Reading disk image....


    ERROR: Image file fat is corrupt

This seems to be because, as you said, on track 0, only first 19 sectors are used. The
last 4 sectors are skipped. So, this place seems to be the hole in FAT table, this is why
diunpack says that FAT is corrupt.

I think, the hole should be handled specially. So, I wonder, how FAT32.IFS works
without the proper handling of that hole.

2Dave Yeo: Another fork at https://efbe.musca.uberspace.de/ seems to be better,
they seem to have OS/2 code not broken I tested it today. The beta3 on Netlabs' ftp
is that Steven Levine is working on. But it seems to be the same as Github fork, and
we get linker errors on some VBox code. The wl.exe and wrc.exe from yum repo seems
to be from Steven too.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 04:25:00 am
2Rick:

Quote
[setup]
    help = setup.hlp

;   Place any programs here that should be run at the end of setup.
;   These apps will be run in order of their appearance here.
[run]

...

You can try searching these strings in the image with a hex viewer.

I had to begin at physical sector number 35 + the logical sector number from the FAT, rather than 33.  Is that what the two reserved entries are for in the FAT?  They also take up two physical sectors on disk?
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 04:37:12 am
The updated file's been posted.  Valery, please try this one to see if it works for you.

Direct download:  http://www.libsf.org/es2/xdflibsf.zip

Code: [Select]
xdflibsf\
xdflibsf\os2\
xdflibsf\os2\xdflibsf.exe
xdflibsf\win32\
xdflibsf\win32\xdflibsf.exe
xdflibsf\readme.txt
xdflibsf\license.txt
readme.txt
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 04:38:30 am
Rick the source code is here if you want to take a look.

http://trac.netlabs.org/vfdisk

Thank you, David.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 04:49:37 am
FAT entries correspond to clusters, not sectors. Though, a cluster on a floppy is usualy one sector. But
floppies with bigger clusters are possible too. FAT entries 0 and 1 are reserved. Entry 0 starts with the same
byte as the media byte in BPB. The media byte is usually 0xf8 on fixed disks, and 0xf0 on 1.44 MB floppies.
360 KB, 1.2 MB floppies have special values for that byte. In fact, the very old FAT disks had no BPB, they
identified media formats with a media byte in the 1st byte of FAT. Later, BPB was introduced as a more
universal mechanism of defining the disk formats. But the 0 FAT entry remained the same for compatibility.
Entry 1 is also reserved. Since Win95, it is used for storing the "dirty" bit of the FAT volume (it is ignored
on floppies, but used on hard disks). Entries starting from 2 describe clusters of data area. The data area
on FAT12/FAT16 start after the root directory. The root directory fas fixed size, specified in the BPB, it cannot
grow. So, the first are reserved sectors. Then follow the FAT copies. The FATs are followed by the root directory.
After the root directory ends, the 1st cluster begins. So, FAT entry #2 correspond to 0 cluster, FAT entry #3 --
to 1 cluster, and so on. So, you should subtract 2 from the FAT entry number, and you'll get 1:1 correspondence
(FAT entry no. - 2) -> cluster number. So, clusters 0 and 1 not correspond to any sectors on disk, they should
be skipped.

On FAT32 and exFAT, the root directory must not immediately follow FATs, and its size is not fixed, it can grow.
Also, in difference from FAT12/FAT16, the root directory on FAT32/exFAT can be fragmented. Also, root directory
area is not reserved on FAT32/exFAT. It is located in data area, and has cluster number assigned. On FAT32, if
the root directory starts after the last FAT immediately, it occupies cluster #2 (and corresponds to FAT entry #2).
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 05:07:57 am
2Rick:

> The updated file's been posted.  Valery, please try this one to see if it works for you.

Now setup.inf extracted ok for me. Also, comparing file contents with the same files
in the image mounted with fat32.ifs, shows that the files are identical.

So, it now seems to work ok.
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 05, 2018, 05:14:38 am
2Valery, https://efbe.musca.uberspace.de/ (https://efbe.musca.uberspace.de/) is Franks build machine, he builds from the official sources and targets OS/2 and is where I was pointed to by the OW developers, mostly Frank, a couple of years ago. Steven also works on the official tree and periodically applies OS/2 fixes and has done some work to support high memory amongst other stuff IIRC. I have no idea about the source on Netlabs or where it originated but Franks server is the go to place for semi-official OS/2 builds based on the daily snapshots
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 05, 2018, 07:21:35 am
2Dave Yeo: Good to know. So, it is an official build indeed.

2Rick Hodgin:

Looking at the diskette I uploaded, ftp://osfree.org/upload/win1.dsk, I tried to check
it with CHKDSK and got:

Quote
[l:\data\vm\bochs]chkdsk j:
The current hard disk drive is: J:
The type of file system for the disk is FAT32.
The FAT32 file system program has been started.
FAT32: Unicode translate table for CP 866 loaded.
Error: Next cluster for 2 = 00000FF7
Error: Next cluster for 3 = 00000FF7
Error: Next cluster for 4 = 00000FF7
Error: Next cluster for 5 = 00000FF7
Error: Next cluster for 6 = 00000FF7
FAT32 version 0.10.r378 compiled on Oct 16 2018
The volume label is WIN 1.
The Volume Serial Number is E300-0000.
The type of file system for the disk is FAT12.

CHKDSK is checking fats :  0%
SYS1374: File Allocation Table (FAT) is bad on drive J:.
The copies of the FATs do not match
Please run CHKDSK under Windows to correct this problem


Errors found.  The /F parameter was not specified.  Corrections will
not be written to disk.

Looking at it in the hex viewer, I can see that the 2nd FAT copy is truncated.
It begins at 0x1800. Sectors #1 and #2 of the 2nd FAT begins with:

Quote
00001A00:  F9 FF FF 03-40 00 FF 2F-00 00 00 00-00 00 00 00  ∙  @  /

What is this? Sector #3 starts with

Quote
00001E00:  57 49 4E 20-31 20 20 20-20 20 20 08-00 00 00 00  WIN 1      
00001E10:  00 00 00 00-00 00 00 00-00 00 02 00-00 00 00 00            
00001E20:  50 41 43 4B-49 4E 47 20-4C 53 54 01-00 00 00 00  PACKING LST
00001E30:  00 00 00 00-00 00 00 00-00 00 02 00-00 08 00 00              

This looks like a directory containing a volume label "WIN1", followed by a file
directory entry for "packing.lst". Sectors #4-#7 contain the contents of that file.
And sectors #8-#10 contain zeroes. Then, staring at offset 0x2e00, starts a root
directory. It also starts from a volume label "WIN1". Note that the volume label
should be in the root directory, only. So, I wonder why sector #3 contains another
root directory. And the real root directory doesn't have the "packing.lst" file. So,
what is that packing.lst file and the second root root directory for? It appears that
second FAT is used to store some special data: a separate private root directory
with a file which lists the files contained in the real root directory! And these file and
directory are preceded with some more unknown data.

Also, this diskette has 484352 bytes of free space, according to fat32.ifs.
According to your utility, it has 485888 of free space. The difference is 1536 bytes,
which is 3 sectors/clusters. Can you check what PCDOS shows on this diskette?

And diunpack, like CHKDSK, sees that two FAT copies are not the same, this is
why it fails to extract files.

Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:11:45 pm
Also, this diskette has 484352 bytes of free space, according to fat32.ifs.
According to your utility, it has 485888 of free space. The difference is 1536 bytes,
which is 3 sectors/clusters. Can you check what PCDOS shows on this diskette?

I don't have an easy way right now to do that.  My desktop machine blew out a capacitor (!!) this weekend.  Scared the whole family.  Popped.  Caught fire for a few seconds.  Really freaky.  Probably the dangers of using a 10+ year old desktop machine.  But, I'll try to get it setup so I can check.  I am able to borrow a desktop machine from my work and will set it up there.

Also, I think my algorithm is wrong, because it also reports 1536 bytes differently than the IBM PC DOS 7.0 disk 3 XDF floppy does in DOS 7.  I don't know why I'm off though.  I begin at FAT + 2 entries, and then proceed through to the end counting 0 entries.  Seems like a straight-forward algorithm. :-)

I'm going to re-visit my whole mounting logic and see if I can't make it generic enough to work with other things.  I'll create an xdfmount utility that is basically like a file viewer, so files can be manipulated when mounted by that utility, so it doesn't have to actually have a driver to mount as a hard drive on the machine.

I'll later expand it to libsfdsk for a generic disk mounter.  This will actually be how I create the FAT12/16/32 driver to use with ES/1 methinks. :-)  I'll get the logic working well in a solid environment, then translate it to physical disks.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 02:16:41 pm
What is this? Sector #3 starts with

Code: [Select]
00001E00:  57 49 4E 20-31 20 20 20-20 20 20 08-00 00 00 00  WIN 1      
00001E10:  00 00 00 00-00 00 00 00-00 00 02 00-00 00 00 00            
00001E20:  50 41 43 4B-49 4E 47 20-4C 53 54 01-00 00 00 00  PACKING LST
00001E30:  00 00 00 00-00 00 00 00-00 00 02 00-00 08 00 00              

This looks like a directory containing a volume label "WIN1", followed by a file
directory entry for "packing.lst". Sectors #4-#7 contain the contents of that file.
And sectors #8-#10 contain zeroes. Then, staring at offset 0x2e00, starts a root
directory. It also starts from a volume label "WIN1". Note that the volume label
should be in the root directory, only. So, I wonder why sector #3 contains another
root directory. And the real root directory doesn't have the "packing.lst" file. So,
what is that packing.lst file and the second root root directory for? It appears that
second FAT is used to store some special data: a separate private root directory
with a file which lists the files contained in the real root directory! And these file and
directory are preceded with some more unknown data.

I'm not sure.  It's almost as if it's a proprietary tweak to the normal FAT file system to include something the loader might check to make sure the disk has been created using a special tool and not just copied.

The "08" there on the first line after the WIN 1 plus spaces is a volume label indicator.  It sure looks like a special directory entry.
Title: Re: XDF disk extractor
Post by: Gregg Young on November 05, 2018, 04:55:34 pm
2Valery, https://efbe.musca.uberspace.de/ (https://efbe.musca.uberspace.de/) is Franks build machine, he builds from the official sources and targets OS/2 and is where I was pointed to by the OW developers, mostly Frank, a couple of years ago. Steven also works on the official tree and periodically applies OS/2 fixes and has done some work to support high memory amongst other stuff IIRC. I have no idea about the source on Netlabs or where it originated but Franks server is the go to place for semi-official OS/2 builds based on the daily snapshots

I built the netlabs betas from the official sources as of the date they were built. Beta 3 includes adding support for the WINDOW keyword in wrc.exe, Those changes were all committed so if Frank's is a daily from the official sources it contains the same fixes. The code in the repository for wl and wrc is unchanged since the beta 3. 

The RPM packages are from the same build as the beta 3.
Title: Re: XDF disk extractor
Post by: David Graser on November 05, 2018, 05:15:43 pm
Also, this diskette has 484352 bytes of free space, according to fat32.ifs.
According to your utility, it has 485888 of free space. The difference is 1536 bytes,
which is 3 sectors/clusters. Can you check what PCDOS shows on this diskette?

I don't have an easy way right now to do that.  My desktop machine blew out a capacitor (!!) this weekend.  Scared the whole family.  Popped.  Caught fire for a few seconds.  Really freaky.  Probably the dangers of using a 10+ year old desktop machine.  But, I'll try to get it setup so I can check.  I am able to borrow a desktop machine from my work and will set it up there.


Rick, contact me at dwgras at swbell dot net

I have an old Toshiba Satellite notebook  M65-S9092 with Windows XP and eCS 2.1.  Send me your address in email and I will send it to you if the shipping is not exorbitant. I don't have the eCS installation disk because I threw them away when I moved to ArcaOS.  I haven't checked on the shipping price yet.

The battery is dead so I only use the power supply.  The right side of the screen has a few lines but is still good.  The screen is a 17 inch.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 05:18:25 pm
I've uploaded the xdflibsf package to Hobbes /pub/incoming/.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 05, 2018, 05:45:26 pm
Rick, contact me at dwgras at swbell dot net

I have an old Toshiba Satellite notebook  M65-S9092 with Windows XP and eCS 2.1.  Send me your address in email and I will send it to you if the shipping is not exorbitant. I don't have the eCS installation disk because I threw them away when I moved to ArcaOS.  I haven't checked on the shipping price yet.

The battery is dead so I only use the power supply.  The right side of the screen has a few lines but is still good.  The screen is a 17 inch.

Thank you, David.  Much appreciated.  If shipping's too much I can PayPal you.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 06, 2018, 02:04:21 am
Valery, I'm looking at the BPB for the XDF disks, and they contain a total of 37 sectors:

Code: [Select]
root directory entries = 224
224 * 32 bytes per entry = 7168 bytes
7168 / 512 = 14 sectors

 1 = boot sector         ; 1
11 = sectors for fat1    ; 12
11 = sectors for fat2    ; 23
14 = root dir sectors    ; 37

According to this logic, data should begin at logical sector number 38.  However, when I am looking at the hex layout, I find the data begins at logical sector number 35.

Does that seem right?  Why would it begin at 35 and not 38 given the BPB data?
Title: Re: XDF disk extractor
Post by: Dave Yeo on November 06, 2018, 02:56:29 am
I'd suggest rereading http://www.os2museum.com/wp/the-xdf-diskette-format/ (http://www.os2museum.com/wp/the-xdf-diskette-format/) in particular how there is a disk image hidden in the second FAT that gets loaded if no driver and how the missing 3 sectors are really missing but marked as bad. At least if I'm following along correctly. Michal is pretty good with this stuff.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 06, 2018, 03:49:53 am
I'd suggest rereading http://www.os2museum.com/wp/the-xdf-diskette-format/ (http://www.os2museum.com/wp/the-xdf-diskette-format/) in particular how there is a disk image hidden in the second FAT that gets loaded if no driver and how the missing 3 sectors are really missing but marked as bad. At least if I'm following along correctly. Michal is pretty good with this stuff.

Will do.  I have dyslexia.  Large amounts of reading are difficult for me.  But, in the interest of doing it right, I'll copy-and-paste into my text-to-speech reader. :-)
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 06, 2018, 11:00:21 am
2Dave Yeo: Thanks for the link. MichalN blog is a good reading, indeed.

2Rick Hodgin: The root directory end at offset 0x4a00 (sector 37 (if sectors are counted from 0)). But
I see the "hidden root directory" repeating again here (a "WIN1" volume label and a "packing.lst" file again).
Why it's again here? We need to read the article in MichalN blog, and see, I think.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 06, 2018, 02:36:03 pm
I'd suggest rereading http://www.os2museum.com/wp/the-xdf-diskette-format/ (http://www.os2museum.com/wp/the-xdf-diskette-format/) in particular how there is a disk image hidden in the second FAT that gets loaded if no driver and how the missing 3 sectors are really missing but marked as bad. At least if I'm following along correctly. Michal is pretty good with this stuff.

It looks like the important parts here:
Code: [Select]
When the XDF TSR was loaded, a 3½” HD XDF floppy looked like a
medium with 80 cylinders and 23 sectors per track.  When the XDF
TSR wasn’t loaded, the same medium looked like a teensy disk with
just a few sectors total, usually containing some text file
showing the XDF disk’s contents or a readme file.

The first cylinder of a XDF floppy used standard 512-byte sectors,
only with 19 sectors per track rather than the standard 18.  The
format used 9 sectors per FAT (for the 3½” HD XDF variant, by far
the one most widely used).  It abused the fact that the second
FAT is normally never read by DOS (as long as the first FAT copy
is readable).  The second FAT copy on a XDF disk instead contained
the small 8-sector backwards compatible image.  The XDF sectors
used high sector IDs and were thus not seen by standard DOS/BIOS.
Track 0 (cylinder 0, head 0) was laid out as follows: 1 boot
sector, 11 FAT sectors, 7 sectors of a backward compatible
micro-disk.  That is 19 sectors total.

The second track (cylinder 0, head 1) started with the root
directory (7 sectors) and was followed by the data area.  Now
there’s an obvious problem: The XDF disk claimed to have 23
sectors per track, but physically there were only 19.  What
happened with the missing 4 sectors?  Easy:  XDF marked the
corresponding clusters in the FAT as bad.

It is important to note that XDF images always contained full
XDF-sized tracks, even for cylinder 0.  Some sectors in a XDF
image were thus not written to a disk, but the XDF image was
uniform.
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 06, 2018, 02:51:16 pm
2Rick Hodgin: The root directory end at offset 0x4a00 (sector 37 (if sectors are counted from 0)). But
I see the "hidden root directory" repeating again here (a "WIN1" volume label and a "packing.lst" file again).
Why it's again here? We need to read the article in MichalN blog, and see, I think.

This portion appears to be in conflict:
Code: [Select]
The first cylinder of a XDF floppy used standard 512-byte sectors,
only with 19 sectors per track rather than the standard 18.  The
==>format used 9 sectors per FAT<=== (for the 3½” HD XDF variant, by far
the one most widely used).  It abused the fact that the second
FAT is normally never read by DOS (as long as the first FAT copy
is readable).  The second FAT copy on a XDF disk instead contained
the small 8-sector backwards compatible image.  The XDF sectors
used high sector IDs and were thus not seen by standard DOS/BIOS.
Track 0 (cylinder 0, head 0) was laid out as follows: 1 boot
sector, ==>11 FAT sectors<==, 7 sectors of a backward compatible
micro-disk.  That is 19 sectors total.

Hmmm....

I'm going to have to examine the disk using a good hex editor, one that can follow links.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 06, 2018, 03:38:51 pm
2Rick: Yes, it's 11 sectors per FAT. Not sure, where 9 sectors were used. 1 reserved sectors + two
11-sector FATs. 1+2*11 = 23 sectors == 1 usual track. Though, DOS/BIOS only sees the
hidden image which is 19 sectors long. Hm.

I didn't read the article to the end, yet, though

PS: Yes, 1.44 MB DS/HD indeed uses two 9-sector FATs. XDF format uses 11 sectors per FAT,
so here Michal is right.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 06, 2018, 03:49:21 pm
2Gregg Young: Oh, nice. So, you added the "WINDOW" keyword to wrc.exe? (Or it's Steven's fixes?)
IIRC, the version from OW 1.9 didn't supported adding resources to OS/2 binaries during the linking,
though, it did support that for windows binaries. But supported adding resources to existing OS/2
binaries (after they are linked). So, it's good to know if newer versions of wrc from OW 2.0 support
adding resources to OS/2 binaries during linking?
Title: Re: XDF disk extractor
Post by: Gregg Young on November 06, 2018, 08:26:15 pm
2Gregg Young: Oh, nice. So, you added the "WINDOW" keyword to wrc.exe? (Or it's Steven's fixes?)
IIRC, the version from OW 1.9 didn't supported adding resources to OS/2 binaries during the linking,
though, it did support that for windows binaries. But supported adding resources to existing OS/2
binaries (after they are linked). So, it's good to know if newer versions of wrc from OW 2.0 support
adding resources to OS/2 binaries during linking?

2Valery Sedletski I added the "WINDOW" support. Steven did the high memory stuff for the linker. I also fixed some issues with wipfc.exe which is in the netlabs version but should also be in Frank's snapshot version since all the code was committed.

You said that the snapshot 2.0 linker worked for vbox. Did you use wlink? I didn't see wl in the package. I built wl from the latest code if you would like to try it I can send it to you. Contact me at ygk at qwest dot net.
Title: Re: XDF disk extractor
Post by: Valery Sedletski on November 07, 2018, 01:29:33 am
2Gregg Young: Is not wl.exe just a renamed wlink.exe? The only difference, it seems, is that wl.exe is built statically. wlink.exe is linked to wlinkd.dll. And wrc.exe from Watcom is linked with wrcd.dll. Wrc.exe from yum repository is built statically too. I just took wlink.exe from OW 2.0 (Frank's build) and renamed it to wl.exe, and relpaced the previous version of wl.exe with it. So yes, I built VBox successfully with these versions of wlink.exe and wrc.exe from Frank's site. The only parts built with OW 1.9 are BIOS and Video BIOS ROMs. They fail to link with wl.exe from OW 2.0. The error from linker is:

Quote
Error! E2011: file L:\src\vbox\prev\out\os2.x86\release\obj\VBoxPcBios32\VBoxPcBios32.lib(L:\src\vbox\prev\src\VBox\Devices\PC\BIOS\pci32.c): invalid object file attribute
Title: Re: XDF disk extractor
Post by: RickCHodgin on November 22, 2018, 01:04:08 pm
I don't have an easy way right now to do that.  My desktop machine blew out a capacitor (!!) this weekend.  Scared the whole family.  Popped.  Caught fire for a few seconds.  Really freaky.  Probably the dangers of using a 10+ year old desktop machine.  But, I'll try to get it setup so I can check.  I am able to borrow a desktop machine from my work and will set it up there.

As of last night, I have my developer environment restored.  Someone at work gave me a 2004 desktop machibe with a 1.5 GHz Athlon XP. :-)  It's working wonderfully.  I have OS/2 4.52 installed, 106 kernel, IBM PC DOS 7, all my old folders and developer tools.

Need to get a green monochrome monitor now.
Title: Re: XDF disk extractor
Post by: RickCHodgin on December 09, 2018, 07:58:53 pm
The XDF disk extractor has been posted on Hobbes (xdflibsf.zip):
http://hobbes.nmsu.edu/h-browse.php?dir=/pub/os2/util/disk

Source code:
http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2

I also realized something today.  The inner-most track on a standard 3.5" floppy disk is about 6.5" in circumference, going out to the outer-most track being about 20.5" in circumference, resulting in about 1096 inches of continuous double-sided data.  A standard 3.5" 1.44MB floppy disk has 12,500 bytes per track per side unformatted.  Formatting to 1.44MB results in 18*512 = 9216 bytes per track per side, a slack of about 182(!!) bytes per sector.  XDF gained its larger space savings by reclaiming 14 of those 182 byte blocks by only using four sectors (one 8KB, one 2KB, one 1KB, and one 512B), or about 3650 bytes per inch.

If, however, the floppy drive itself were modified and used a spiral track like a CD/DVD, and employed variable RPM and kept the sector length consistent throughout the disk to that seen in the inner-most track, and switched to 8KB sectors, it would result in 1096 inches of 3650 bytes per inch with formatting overhead ... it would be about 3.75MB per disk formatted on the same 1.44MB floppies we used today.  The only variable is the speed it rotates keeping the sector length the same as that found on the inner tracks, but throughout the entire disk, and using 470 8KB sectors to reclaim the extra wasted space by sector markers.

I wonder why nobody ever created a drive like that?  Even if they used a wider format and only got 3MB per disk, it would still be better than even the 2.88MB disks they had, and all still using the same density media as the 1.44MB floppies.  It would've been a little slower to access, but I don't think people would've minded because it would've held that much more, requiring 1/2 to 1/3rd fewer disk changes during installation.  Imagine, OS/2 Warp 4 from IBM could've booted off one floppy disk rather than three.

OS/2 could've shipped on 1/3rd to 1/2 the number of floppies for machines with floppy drive support for a spiral track mode and variable RPM.
Title: Re: XDF disk extractor
Post by: ivan on December 10, 2018, 01:40:20 am
I think you will find the reason they went for discrete tracks on floppies is the ease of finding files.  The start of a file is located as a sector of a track and the positioning motor moves the head assembly to that track.  Since a file can be split between different sectors and tracks using a spiral track would take a long time for the head to traverse back and forth to read all the file segments.  It only works for a CD/DVD because they are write once read many times - you don't delete files on a CD/DVD without having to write the  whole thing again.  That is also why hard disks use discrete tracks and sectors.
Title: Re: XDF disk extractor
Post by: RickCHodgin on December 10, 2018, 03:27:47 am
I think you will find the reason they went for discrete tracks on floppies is the ease of finding files.  The start of a file is located as a sector of a track and the positioning motor moves the head assembly to that track.  Since a file can be split between different sectors and tracks using a spiral track would take a long time for the head to traverse back and forth to read all the file segments...

A 3.75MB format though ... it would've been a nice option even if slower.  It could've been an option available for formatting.

Plus, I think drives that have this feature could move to sectors fast with some logic indicating which sector belongs where in the spiral as it would be fixed.  File systems could also still begin on logical boundaries, they would just be translated for physical writing.

I'm just surprised.  3.75MB on a standard 1.44MB disk... that's desirable.  IBM added support for 1.84MB.
Title: Re: XDF disk extractor
Post by: RickCHodgin on December 31, 2018, 03:18:32 pm
I have observed where this file exists here:
https://ecsoft2.org/xdflibsf-os2

It shows a DSK file sub-directories.  When looking for XDF-compressed files I could not find any that had sub-directories, and therefore couldn't get the code there tested / working:

Code: [Select]
///////////
// Note:
// Nov.04.2018 -- RCH -- I could not find an XDF file that had sub-directories, so I couldn't test this.
// // Load the sub-directory
// subdir = (SDirEntry*)iiLoadDirectory(fh, entry->first_cluster);
// if (!subdir)
// {
// // Error reading the directory
// printf("Error entering directory %s\n", filename_11_3);
//
// } else {
// // Update the deeper path
// lnPathLen = strlen(path);
// iiBuildFilename(filename_11_3, filename, extension, true);
// strcpy((s8*)path + lnPathLen, (s8*)filename_11_3);
// strcpy((s8*)path + strlen((s8*)path), "\\");
//
// // Display the sub-directory
// iiDisplayDirectory(fh, path, subdir, lnTotalFiles, lnTotalSize, tlExtract, argv);
//
// // Reset the path
// path[lnPathLen] = 0;
//
// // Clean house
// free(subdir);
// }
//////

Source code (line 795):
http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/UTILS/xdflibsf/xdflibsf.cpp

If anyone has an existing publicly released XDF-compressed disk with sub-directories could you ZIP it up and post it somewhere so I can test out that functionality?
Title: Re: XDF disk extractor
Post by: RickCHodgin on December 31, 2018, 09:18:03 pm
I have observed where this file exists here:
https://ecsoft2.org/xdflibsf-os2
...

If anyone has an existing publicly released XDF-compressed disk with sub-directories could you ZIP it up and post it somewhere so I can test out that functionality?

I got an email today from eCSoft/2 Administration.  They sent me the attached file on that link so I can update the source code.  I plan to work on that tomorrow evening, and post the update next weekend.

UPDATE:  I have not had a chance to work on this yet.  I plan to do it immediately after I complete my current project / task, which should be this week.  I hope to get this utility updated by Friday, and have it posted this weekend.  If anyone wants to help me, the source code is out there.