DISKIO - storage device detection logic

[Dariusz Piatkowski]

Alright...the one well known DISKIO problem/challenge stems due to the fact that any removable storage devices, these being primarily connected through USB, cause DISKIO to error when no actual media is mounted.

AFAIU at the start of execution DISKIO simply asks DosPhysicalDisk API for a number of partitionable disks using this API call:

Code: [Select]

DosPhysicalDisk (INFO_COUNT_PARTITIONABLE_DISKS, &nDisks, sizeof(nDisks), 0, 0)

Sure enough, ALL devices defined as storage media, whether these are currently mounted and accessible, show up in that list.

Further on, DosPhysicalDisk is called again but this time in order to obtain the device handle for DosDevIOCtl32 calls later on:

Code: [Select]

DosPhysicalDisk (INFO_GETIOCTLHANDLE, &usPhysical, sizeof(usPhysical)

OK, so far so good, only one caveat here: "...The handle returned for the specified partitionable disk can only be used with the DosDevIOCtl function for the Category 09h Physical Disk Control IOCtl Commands...".

Now DISKIO proceeds to query the device parameters by calling DosDevIOCtl32:

Code: [Select]

DosDevIOCtl32 (&dpb, sizeof(dpb), &bCmd, sizeof(bCmd), DSK_GETDEVICEPARAMS, hfHandle)

Here is the thing: I think it ONLY gets the parameters but doesn't actually do any checking to see if the media can actually be accessed. It assumes that since the storage device was found it can therefore be tested.

What I am not sure about however is how to interpret the meaning of the logic that decides whether to retrieve the DEVICEPARAMETERBLOCK or BIOSPARAMETERBLOCK record structures.

In other words, the IF statement check "if (hfHandle & PHYSICAL)" below, where we have the following defines:

#define PHYSICAL     0x1000
#define CDROM        0x2000

...does that mean that hfHandle will be BIT anded with PHYSICAL and if they MATCH the device we are querying must therefore be a mounted storage media?

I see we distinguish between PHYSICAL & CDROM, perhaps USB attached should have a different define as well?

Code: [Select]

   // now check if we have a valid physical file handle
  if (hfHandle & PHYSICAL)
   {
      // get the info for DEVICEPARAMETERBLOCK
      if (DosDevIOCtl32 (&dpb, sizeof(dpb), &bCmd, sizeof(bCmd), DSK_GETDEVICEPARAMS, hfHandle))
      {
         DosPhysicalDisk (INFO_FREEIOCTLHANDLE, NULL, 0, &usPhysical, sizeof(usPhysical));
         return (-1);
      }

      *lSectors = dpb.cSectorsPerTrack;
      *lTracks  = dpb.cCylinders;
      *lSides   = dpb.cHeads;
      *lSector  = DSK_DEFAULT_SECTOR_SIZE;

      #ifdef DEBUG
         printf("\n DEBUG => found the disk and storing in DEVICEPARAMETERBLOCK...\n");
      #endif /* DEBUG */
   }
   else
   {
      // get the info for BIOSPARAMETERBLOCK
      if (DosDevIOCtl32 (&bpb, sizeof(bpb), &bCmd, sizeof(bCmd), DSK_GETDEVICEPARAMS, hfHandle))
      {
         DosClose (hfHandle);
         return (-1);
      }

      *lSectors = bpb.usSectorsPerTrack;
      *lTracks  = bpb.cCylinders;
      *lSides   = bpb.cHeads;
      *lSector  = bpb.usBytesPerSector;

      #ifdef DEBUG
         printf("\n DEBUG => found the disk and storing in BIOSPARAMETERBLOCK...\n");
      #endif /* DEBUG */
   }

Anyways...a LOT of detail there, and so I'm going to run some debug sessions to understand this a tad better. In the meantime, any hints would of course be greatly appreciated!!!

[Dave Yeo]

At the worse, I notice the empty USB devices are always 96MB here. Perhaps the smallest possible actual partition? Anyways, if no one comes up with a better method, fall back to testing if CHS all contain ones or such that works out to 96MB.
As the Installation Volume Manager ignores those phantom drives, perhaps there's a better way.

[Dariusz Piatkowski]

Hey Dave!

At the worse, I notice the empty USB devices are always 96MB here. Perhaps the smallest possible actual partition? Anyways, if no one comes up with a better method, fall back to testing if CHS all contain ones or such that works out to 96MB.
As the Installation Volume Manager ignores those phantom drives, perhaps there's a better way.

Yeah, so even with that approach we seem to lack consistency b/c those empty devices here show up as 27MB:

Code: [Select]

DEBUG => using DosPhysicalDisk for $7:

 DEBUG => found the disk and storing in DEVICEPARAMETERBLOCK...

 DEBUG => hfHandle is : 4099
 DEBUG => lSectors is : 32
 DEBUG => lTracks  is : 512
 DEBUG => lSides   is : 12
 DEBUG => lSector  is : 512

Hard disk 7: 12 sides, 512 cylinders, 9 sectors per track = 27 MB

Initially what I thought I could do is to use DosDevGetIOCtl but call:

1) Category: IOCTL_DISK (08h) with Function: DSK_BLOCKREMOVABLE (20h), Description: Block Removable), which apparently detects whether the media is removable

OR

2) Category: IOCTL_DISK (08h) with Function: DSK_GETDEVICEPARAMS (63h), Description: Query Device Parameters, which shows Device Attributes:

Code: [Select]

Device Attributes
A bit field that returns flag information about the specified drive:

Bit 0 Removable Media flag. This bit is set to 1 if the media cannot be removed. It is set to 0 if the media is removable.

Bit 1 Changeline flag. This bit is set to 1, if the device support determines that the media was removed since the last I/O operation. To query whether the media has changed, call the physical device driver Strategy Command "1h - MEDIA CHECK". (Refer to "Physical Device Driver Strategy Commands" in the Physical Device Driver Reference for more information.)

If this bit is set to 0, then the physical device driver can return the value 0, Unsure if media has changed, from the Media Check function.

Bit 2 Greater than 16MB Support flag. If this bit is set to 1, the physical device driver supports physical addresses greater than 16MB.

...but that is starting to seem like a dark dark tunnel I'm heading into! lol

I suppose there is one last element to look at there, that being the 'Extended BPB for Devices - The term Extended BPB refers to the BIOS Parameter Block', which has the following record structure:

Code: [Select]

typedef struct _BIOSPARAMETERBLOCK {
  USHORT     usBytesPerSector;                 /*  Number of bytes per sector. */
  BYTE       bSectorsPerCluster;               /*  Number of sectors per cluster. */
  USHORT     usReservedSectors;                /*  Number of reserved sectors. */
  BYTE       cFATs;                            /*  Number of FATs. */
  USHORT     cRootEntries;                     /*  Number of root directory entries. */
  USHORT     cSectors;                         /*  Number of sectors. */
  BYTE       bMedia;                           /*  Media descriptor. */
  USHORT     usSectorsPerFAT;                  /*  Number of secctors per FAT. */
  USHORT     usSectorsPerTrack;                /*  Number of sectors per track. */
  USHORT     cHeads;                           /*  Number of heads. */
  ULONG      cHiddenSectors;                   /*  Number of hidden sectors. */
  ULONG      cLargeSectors;                    /*  Number of large sectors. */
  BYTE       abReserved[6];  /*  Reserved. */
  USHORT     cCylinders;                       /*  Number of cylinders defined for the physical device. */
  BYTE       bDeviceType;                      /*  Physical layout of the specified device. */
  USHORT     fsDeviceAttr;                     /*  A bit field that returns flag information about the specified drive. */
} BIOSPARAMETERBLOCK;

typedef BIOSPARAMETERBLOCK *PBIOSPARAMETERBLOCK;

Seems like this could provide better/more valuable information to build the logic with?

Outside of this approach, shouldn't I be able to simply call the matching USB API? I took a look at the published USBDevToolkit, but other than the header file there is absolutely no other information available...soooo....?

The final approach is to assume that the storage device is in fact good but perhaps add-in an extra attempt to actually read something off of it with maybe DosQueryFslnfo (apparently the old DosQFslnfo), and if that fails flag the device as non-accessible (i.e. not mounted) and skip!

[Sandra Asja Eickel]

Hello Dave and Dariusz,

I've never heard of or seen these non-existent but reserved drives with anything but the size of an Iomega ZIP Drive 100 medium, so those 96 MB are the standard. Perhaps it's best to ignore anything smaller than 100 MB unless a switch to test also floppy drives and really old CF and SD cards is given (or the test of a specific drive letter is requested). (Re-)Writable optical media and network drives with different access methods might also get strange results.

Greetings,
Sandra-Asja

[KO Myung-Hun]

Hi/2.

Parameter Packet Format of DSK_GETDEVICEPARMS(63H) of Category 08H has command information member. If it is set to 1, then it tries to read from a media currently in a drive.

DSK_GETDEVICEPARAMS (63h) - Command Information

    Command Information
        A byte with bit 0 defined as follows:

        Return the recommended BPB for the drive, which is the BPB for the physical device, unless it is a formatted fixed media. Then, it is the BPB that was on the media when the system was booted.
        Return the BPB for the media currently in the drive. This always reads the BPB off the current media in the drive. An error is returned if the media is unformatted.

        All other bits are reserved, and must be set to 0.

I think, DosCevIOCtl() will return an error if a media is not connected.

[Rich Walsh]

In the 'fsDeviceAttr' field, Bit 3 (0x08) will be on if this is PRM.

[Dariusz Piatkowski]

Alright, a bit of an update.

I spent considerable amount of time over the holidays trying to understand the DISKIO 'device type' detection logic. This is important because the remaining code execution pathway is largely based on what type of a device is being processed.

In short, there are three types recognized, which in turn control the definition of DosDevIOCtrl CATEGORY call.

This section declares two of them:

Code: [Select]

#define PHYSICAL     0x1000   //4096
#define CDROM        0x2000   //8192

Later on a dynamic CATEGORY assignment is made through the following macro:

Code: [Select]

CATEGORY(x)  (((x) & CDROM) ? IOCTL_CDROMDISK : ((x) & PHYSICAL) ? IOCTL_PHYSICALDISK :IOCTL_DISK)

So basically:
if device=CDROM, then CATEGORY = IOCTL_CDROMDISK
if device=PHYSICAL, then CATEGORY = IOCTL_PHYSICALDISK
else CATEGORY = IOCTL_DISK

Indeed, in my debugging I have found the following to be substantitated:

1) HD        - 4099, which means DosDevIOCtl is given CATEGORY = IOCTL_PHYSICALDISK (09h)
2) CDROM - 8195, which means DosDevIOCtl is given CATEGORY = IOCTL_CDROMDISK (80h)
3) USB      - 4099, which means DosDevIOCtl is given CATEGORY = IOCTL_PHYSICALDISK (09h)

...alas, and here is where our problems arises: both HD and USB storage devices are treated like the same category device, that being IOCTL_PHYSICALDISK.

At the moment my stipulation is that the use of "...else CATEGORY = IOCTL_DISK..." was intended to cover stuff like USB based devices...but that's just a guess, and maybe that was even meant to support throughput testing on remote storage devices? (think UNC \\server\mount_point type of stuff)

So to get this under control what I would like to do first is to correctly recognize a USB storage device as such and differentiate it from both the CDROM as well as regular HD.

So where does that device handle come from (which is then logically AND'ed in that MACRO)? Well, that comes from a call to DosOpen, or DosPhysicalDisk (in which case only the IOCTL_PHYSICALDISK (09h) can execute).

...in turn, this begs the question on my part: should either DosOpen or DosPhysicalDisk not have the ability to properly recognize a USB based storage device as being different than a regular HD, and therefore return a handle which leads me down the path of CATEGORY = IOCTL_DISK???

Seems like there is something missing in this logic and I can't quite put my finger on this.

Once I understand this I think I will be in a better postion to determine which approach is most suited to determine if a USB based storage device actually has mounted media in it, that could in fact be tested.

EDIT => spelling...yikes! *-(

[Dave Yeo]

Well, I notice the drives object has no problem detecting a removable drive, usually USB now a days but previously things like zip disks. No idea how it works.

[Lars]

I seem to remember that IOCTL_PHYSICALDISK was for accessing all sectors of a physical disk whereas IOCTL_DISK was for accessimg a drive / a volume (or partition) of a disk.
So that has nothing to do with device type.

[Dariusz Piatkowski]

Hello Lars,

I seem to remember that IOCTL_PHYSICALDISK was for accessing all sectors of a physical disk whereas IOCTL_DISK was for accessimg a drive / a volume (or partition) of a disk.
So that has nothing to do with device type.

OK, that's good to know as I am currently looking at using the media/removable check for each device to identify whether that device should actually be tested. Basically: if the media is 'mounted', we are good, otherwise we need to skip it.

Here is the WIP logic (ignore the obvious missing declarations, etc, this is just general logic with some specific API stuff tossed in as anchors):

Code: [Select]

void device_map (int iDisks, int iCDROMs)
{
   int      iCnt;
   APIRET   rc = NO_ERROR;

   // we have a TOTAL number of HD-type and CDROM-type devices now, so let's
   // take a pass through them all and check to see if they have mounted media
   // in them, if YES, that means we can test, otherwise, we should NOT!
   for (iCnt = 0, iCnt < iDisks, iCnt+)
   {
      // what device am I processing?
      sprintf(szName, "$%d:", iDisk);

      // let's get a handle for this device
      if (DosOpen (szDrv,                                      /* device name */
                   &hfHandle,                     /* address of device handle */
                   &ulAction,              /* address of DosOpen action taken */
                   0,                               /* logical size of device */
                   FILE_NORMAL,                                 /* attributes */
                   FILE_OPEN,              /* action to take if device exists */
                   OPEN_FLAGS_DASD | OPEN_FLAGS_FAIL_ON_ERROR |
                   OPEN_ACCESS_READWRITE | OPEN_SHARE_DENYREADWRITE,  /* mode */
                   0)                           /* device extended attributes */
      {
         // call the DosDevIOCtl API here
         if (DosDevIOCtl (HANDLE (hfDevice),                    /* device handle */
                          IOCTL_DISK,               /* category of request - 08h */
                          DSK_BLOCKREMOVABLE,  /* function being requested - 20h */
                          pParms,                 /* Input/Output parameter list */
                          ulParms,        /* size of Input/Output parameter list */
                          &ulParmLengthInOut,  /* Input: size of parameters list */
                                          /* Output: size of parameters returned */
                          pData,                       /* Input/Output data area */
                          ulData,              /* size of Input/Output data area */
                          &ulDataLengthInOut); /* Input: size of input data area */
                                                /* Output: size of data returned */
         
         // now parse the result and update the global device structure record
         // WIP
      }

      // display the device status information here
      // WIP

   }

} // end of device_map

Subsequently that "global device structure" record would be used by the remainder of the DISKIO logic to check whether the particular device should, or should not be benchmarked.

Therefore, what your remark tells me is that I should be fine to use that IOCTL_DISK (08h) & DSK_BLOCKREMOVABLE (20h) combo to test for presence of actual media (where a partition object is present) as opposed to just the presence of a storage device that may not in fact have any media currently loaded (thus the problem we are seeing with USB devices) - I think anyways...for now...

[Andi B.]

Not sure what you're really want to know but DataSeeker uses DosQueryFSAttach() to distinguish between different drive types. I think I copied this from 4os2 code back then but not sure anymore. Anyway http://trac.netlabs.org/dataseeker/browser/trunk/DriveInfo.c contains the code which finds out if the drive is a disk, floppy, network, CD/DVD, or .... Also if removable or not. Looks like -

    apirc = DosQueryFSAttach(sTemp, 0, FSAIL_QUERYNAME, p2, &cbBuffer);
    //TRACE1(" apirc=%4d ", apirc);
    if ( apirc == NO_ERROR )
      {
      //TRACE1("%-8s ", p2->szName + p2->cbName + 1);
      if ( p2->cbFSDName > sizeof(DriveInfo[uDrvNo].fsName)-1 ) p2->cbFSDName = sizeof(DriveInfo[uDrvNo].fsName)-1 ;
      memcpy(DriveInfo[uDrvNo].fsName, p2->szName + p2->cbName + 1, p2->cbFSDName);
      DriveInfo[uDrvNo].fsName[p2->cbFSDName] = '\0';

      switch ( p2->iType )
        {
        case FSAT_CHARDEV:   // Resident character device
          DriveInfo[uDrvNo].type = DT_FLOPPY;
          break;

        case FSAT_PSEUDODEV: // Pusedu-character device
          if ( IsMediaRemoveable(uDrvNo) )
            DriveInfo[uDrvNo].type = DT_REMOVABLE;
          else
            DriveInfo[uDrvNo].type = DT_RAMDRIVE;
          break;

        case FSAT_LOCALDRV:  // Local drive
          if ( !strncmp (DriveInfo[uDrvNo].fsName, "CDFS", 4) )
            DriveInfo[uDrvNo].type = DT_CDROM;
          else
            if ( IsMediaRemoveable(uDrvNo) )
            DriveInfo[uDrvNo].type = DT_FLOPPY;
          else
            DriveInfo[uDrvNo].type = DT_HARDDISK;
          break;

        case FSAT_REMOTEDRV: // Remote drive attached to FSD
          if ( !strncmp (DriveInfo[uDrvNo].fsName, "RAMFS", 5) )
            DriveInfo[uDrvNo].type = DT_RAMDRIVE;
          else if ( !strncmp (DriveInfo[uDrvNo].fsName, "CDWFS", 5) )
            DriveInfo[uDrvNo].type = DT_RSJ;
          else
            DriveInfo[uDrvNo].type = DT_NETWORK;
          break;
        }

[Dariusz Piatkowski]

Well, that few hours of work took me completely NOWHERE!!! 

OK, that's good to know as I am currently looking at using the media/removable check for each device to identify whether that device should actually be tested. Basically: if the media is 'mounted', we are good, otherwise we need to skip it.
....
Therefore, what your remark tells me is that I should be fine to use that IOCTL_DISK (08h) & DSK_BLOCKREMOVABLE (20h) combo to test for presence of actual media (where a partition object is present) as opposed to just the presence of a storage device that may not in fact have any media currently loaded (thus the problem we are seeing with USB devices) - I think anyways...for now...

Yeah, I completed that function and learned that the DosOpen API won't deal with anything other than either a fully qualified filename, or in the case of a drive it must be a proper drive letter/number.

Soooo...my hope, as slim as it was, that the numbered DEVICE name could in fact be used by DosOpen was incorrect. Given this I lack a way to connect the devices that the INFO_COUNT_PARTITIONABLE_DISKS parameter gives me when calling DosPhysicalDisk to actual volume/partition objects means that I have no way to check whether a USB device currently has media mounted in it and therefore coudl be tested by DISKIO.

Lars did point out in another thread (circa 2021 => https://www.os2world.com/forum/index.php?topic=2792.0) that I may need to use DosPhysicalDisk, but the problem with that approach is in the fact that the returned device handle can only be used with DosDevIOCtl CATEGORY=09h functions and none of those functions return information about the status of media on the device itself, and I never did figure out what is that "...special "filename" to use [with] DosOpen..." either.

Hmmm...

I tell ya it almost seems like core of DISKIO logic should be re-written to approach this from partition/volume perspective, which would give us visiblity to FS stuff (and thus media presence). That subsequently could be used to determine whether an actual DEVICE (which the partition/volume belong to) should be tested. Right now DIKSIO's logic is the inverse of that.

Maybe I just need to re-focus on handling the exception that arises when an unmounted-media USB device is being benchmarked so that this exception can be correctly handled...LOL?

[Dariusz Piatkowski]

Hey Andi!

Not sure what you're really want to know but DataSeeker uses DosQueryFSAttach() to distinguish between different drive types. I think I copied this from 4os2 code back then but not sure anymore. Anyway http://trac.netlabs.org/dataseeker/browser/trunk/DriveInfo.c contains the code which finds out if the drive is a disk, floppy, network, CD/DVD, or .... Also if removable or not...

Aha...good stuff for sure!

This has helped me a lot in understanding how to correctly call the other DosDevIOCtl category functions and I was in fact able complete the logic for the device mapping function. Alas (see my previous thread update) that's still not enough to tell me whether a simple device # has media mounted on it and could in fact be benchmarked.

FYI - there is a very real chance here I'm missing something obvious, so don't by shy in giving me a quick smack...LOL