Access to HPFS partitions using DOS drivers

┌─────────┐                               ┌─────────┐        │   FAT   │──────────────────────────────»>│  HPFS   │ │=========│                               │=========│        └─────────┘<«──────────────────────────────└─────────┘           iHPFS                                   HPFS Access Marcus Better                       Andreas Kinzler
 * This document is based on the work done by:
 * Marcus Better 
 * Andreas Kinzler 

Disclaimer
Use the included information at your own risk. Remember: it's your disk, you're on your own!
 * Blame only yourself if anything goes wrong.
 * Don't blame me.
 * Please understand that I didn't do much tinkering to write this up.

Why an HPFS driver?
People who work with OS/2 know that HPFS is a far better file system, basically for three reasons: When a partition is formatted using FAT what happens is that an array of pointers, called the File Allocation Table [FAT] is created. Each of the entries in the FAT array points to a File Allocation Unit, or cluster as CHKDSK calls it, which is the minimum size of disk storage that the FAT file system will assign to a file.
 * 1) HPFS disk partitions fit more data than equally sized FAT partitions.
 * 2) HPFS disk access is faster than FAT access.
 * 3) HPFS partitions are more robust.

The main problem with FAT is its 16-bit restriction, because the FAT array can have no more than 64k entries. This means that if a disk is very large, the minimum size for the file allocation unit grows also very large.

A decade ago disks where not really big. A very fat disk will have 100 Megs of storage, which means that its cluster size would be quite small: only 2k. However, with current disk sizes going over the one Gig barrier, cluster sizes of 16k and 32k are now common.

Why is a 32k cluster size bad? Well, if a file is to be stored in the disk, then it will occupy at least one cluster. Most files are small, which means that in the long run the vast majority of stored files will occupy big clusters, wasting a lot of space. For example, if your AUTOEXEC.bat file is 2,000 bytes longs (which is quite big), then you will be be wasting 32k-2k=30k to store it! This shocking shortcoming of the FAT file system is called External File Fragmentation.

Furthermore, when the operating system access a FAT disks it must read a whole cluster regardless of the file size. For the above example, this means that even though only 2k worth of data is stored in your AUTOEXEC.bat, the file system will nevertheless read in 32k: 16 times more resources will be used than what it's required!

This explains why the FAT file system will die soon. HPFS, and other file systems, have no 16-bit limits and use clusters that are only 512 bytes long. That's one of the main reason why people have turned to OS/2. Besides this, FAT has other technical problems.

The following table shows the cut points where the cluster duplicates its size as the partition gets bigger. The percentage in the third column is an estimate of the efficiency of the file system, this is, it shows how much of the storage is wasted to big clusters due to external fragmentation (I extracted this data from the article by Jeff Prosise in PC Magazine). The meaning of the numbers in the above table is the following. If you have a partition bigger than 512 Megabytes, then when you format using FAT you will get 16K clusters, which means that your file efficiency will drop down to 83%. This means that due to FAT limitations you will be wasting 17% of your disk. For example, for your new 1.0 Giga disk FAT uses 16k clusters, which means that you put to waste 170 Megs! If you happen to have a 4 Giga disk then half of it will be wasted by FAT. This figures presume an average file size of 3K: your mileage will vary depending on your disk usage.

This explains why some people argue that OS/2 deletes their files when they move them to an HPFS partition: as HPFS is nearly 100% efficient, they think that not all the data got copied! What really happened is that all the disk space lost to external fragmentation didn't get copied, and the HPFS file system simple reclaimed that wasted space. FAT is a waste!

Now that I convinced you to use HPFS on partitions bigger than 128 Megs, let me tell you how can you access that data from MS-DOS.

Finding a copy of the drivers
The first thing you need to do is to get one of the available drivers to access your HPFS partitions from DOS. I found the following in the Hobbes OS/2 archives (their also available at Simtel): hpfsa02b.zip 165,254 Aug-95 HPFS driver for DOS, read & write, 8k mem ihpfs121.zip 14,295 Sep-95 iHPFS v1.21, DOS installable HPFS driver amos320.zip  41,402 Jan-95 TSR for mounting HPFS drives from DOS hpfsds10.zip 106,367 Jan-95 HPFSDOS 1.00 - HPFS from DOS (read only) hpfsr16e.zip 20,615   1993 Read HPFS-partitions under plain DOS To get these files, ftp to hobbes.nmsu.edu or to any of its mirrors and change directory to /dos. In there you will find the files, or their newer versions.

After getting the zipped files, unzip them into a directory. I set up all of them in my C:\BIN\HPFS4DOS subdirectory. C: is the drive from where I boot MS-DOS, and it's formatted under FAT.

Using an HPFS driver for DOS
The drivers I have tried are iHPFS and HPFS_access (by Marcus Better and Andreas Kinzler respectively). Both drivers load as TSR programs (Terminate and Stay Resident), and assign to each HPFS partition a drive letter. After this, each HPFS partition looks to DOS as a regular (network) drive, and files in the HPFS partitions can be accessed from any DOS program.

iHPFS is freeware in the sense that you may copy it for free and upload it everywhere (as long as you include the docs and don't alter anything). However, iHPFS will not let you write into an HPFS partition. HPFS_access can write into HPFS partitions, but after writting 16 Megabytes or 20 minutes, whatever comes first, it will set to READ-ONLY the HPFS partition. This, I think, encourages users to register their copies.

A neat feature in both drivers is that they can be unloaded from memory. I have been playing around with both of them, and I can swap one for the other. To do this, I wrote a batch file called REhpfs.bat, which I also include in this document.

I have found that when I install HPFS_access from my AUTOEXEC.bat file afterwards it will refuse to unload (it complains about interrupt vector $2 being already super-seeded). But if I install iHPFS first then I can swap one for the other as many times as I want. However, every time I swap iHPFS out it will leave a small TSR stub.

I contacted Marcus Better about this, and he sent me the following explanation:

"iHPFS uses some software interrupts. In DOS, this is done by hooking onto a chain of interrupt service routines. When iHPFS is uninstalled, it needs to remove itself from this chain. If, in the meantime, another TSR is installed that uses the same interrupt, just restoring the chain to the state it was in prior to the installation of iHPFS may damage the chain. The only proper way to remove a TSR without doing any damage is to leave the interrupt chain as it is, and just deactivate the service routine, so that it passes all interrupts to the next TSR in the chain. This requires that a small piece of the program's code is left in memory, and that is why the stub of 320 bytes or so remains after iHPFS is uninstalled. Except for passing on interrupts, this stub is completely inactive. If the TSR is written so that to uninstall it instead tries to restore the interrupt vector to its original state, then when another TSR program is installed after the driver it won't be able to uninstall without damaging the other TSR, and so the driver (quite correctly) must refuse to uninstall. This is what happens with iHPFS, but other TSRs like MSCDEX should also cause the same behaviour."

To me, what this says is that DOS sucks, and that there's no way to do things right. You can always reboot if you want to clean things up, or if you don't like the memory map that MEM /C gets you from the DOS command prompt.

Mixing and matching drive letters
As it was probably the case for most OS/2 users, most of my software runs in the DOS platform. And as it also happened to you, I installed OS/2 when I got my 1.2 Gig hard drive. Previously, I had only a 322 Meg drive, formatted with FAT using 8k clusters (and wasting almost 10% of my storage capacity!). My setup was the following: DRIVE #1 Letter: C:    322 Megs FAT DOS v6.2 When I got my newer drive, I decided to use my older drive as the second drive. I was careful to get a same-brand drive, which probably saved me quite some hours of pain, as often times different brand drives won't talk to each other. My configuration would look like this: DRIVE #1      DRIVE #2 Letter: C:    Letter: D:     1.2 Gigs       322 Megs FAT           FAT 30% waste     8% waste After I installed the newer drive, I found that I could not format it to be bigger than 528 Megs! My machine is not that new, which means that its BIOS has a restriction that will not let it see disk cylinders beyond 1,024. There's a fix for this, developed by Ontrack. (The Ontrack driver to access big disks came in the newer disk, but I'm using one that I downloaded from the net).

What the Ontrack driver does is to install itself in the MBR, or Master Boot Record, so that when the computer boots the first thing that will load is the Ontrack driver. This driver patches the BIOS and does away with the 1,024 cylinder limitation: after that DOS can read bigger disks (the same goes for any other operating system that uses the BIOS to access disks). With this Ontrack software I was able to format my Hard Drive #1 to its 1.2 Gig full capacity.

Latter I learned that the newer disk drivers for OS/2 can handle big disks with no problem, but you must use the newer versions of the OS/2 drivers (you can get these at Hobbes too): IBM1S506 add   28,328  15-Jan-95  21:21:26 OS2DASD dmd    33,578  04-Jan-95  20:52:32 IBMIDECD flt   20,490  10-Jan-95  20:47:00 What all these means is that if you will never use DOS, then you don't need to have Ontrack patch you BIOS because the IBM1S506.add and OS2DASD.dmd will handle big disk from OS/2 directly. However, if you are like me, then you will want to be able to boot from DOS from time to time, and for this you will probably install Boot Manager [BM] to launch both DOS and OS/2.

Boot Manager is a small program that lets you select the operating system that will boot in your computer. For this, it tries to install itself into the MBR to intercept the boot process. If you already installed the Ontrack driver, then BM will be the second in chain, and the third will be your operating system.

The reason I didn't get rid of Ontrack from my 1.2 Gig disk is that it let me install BM at the END of my disk, instead of forcing me to put it at the beginning. I never liked BM to be in the first partition, because some operating systems (like Linux, they tell me) mess it up if it's at the beginning of the disk (see OS/2 Warp Unleashed pp 14).

After some thinking, I decided that my new 1.2 Gig drive would have three partitions. The first one would be drive C:, the DOS bootable partition using 127 Megs, formatted in FAT with a cluster size of 2K. The second one would be the OS/2 partition, with a total of 561 Megs and the last one I reserved to install in 511 Megs a third operating systems (like Win95): this I formatted under FAT with 8k clusters. The second drive became drive letter D:, and I installed OS/2 to boot from drive E: in the first disk drive. I also have a CD-ROM (G:) and use a RamDisk (H:). My setup looks as follows:  ┌──────────────────────────────────────────────────────────┐       │ Name     Status        Access      FS Type        MBytes │ ├──────────────────────────────────────────────────────────┤ Disk #1 │ DOS     Bootable    C: Primary    FAT-->2K         127  │ 1.2 Gig │ OS/2    Bootable    E: Logical    HPFS             561  │ │         None        F: Logical    FAT-->8K         511  │ │         Startable    : Primary    BOOT MANAGER       2  │ ├──────────────────────────────────────────────────────────┤ Disk #2 │         None        D: Primary    FAT-->8K         321  │ ├──────────────────────────────────────────────────────────┤ CR-Rom │          None        G: Primary    CD-Rom           640  │ ├──────────────────────────────────────────────────────────┤ RamDisk │         None        H: Primary    FAT                3  │ └──────────────────────────────────────────────────────────┘  My friends tried to convince me to leave a big 561+511 HPFS partition instead of using drives E:+F:, but I want to thinker with other operating systems so I'm keeping my F: FAT partition. I can always reformat it, and there are programs to make my E: partition bigger without losing the data in it (look into FIPS, by Arno Schaefer , or buy PartitionMagic).

After I had OS/2 up and running I tried to install the iHPFS driver to access my E: partition from DOS. There I found two problems. First, upon boot DOS will not recognize my E: partition, and will name my 511 Meg F: partition as drive E:. And second, when I finally managed to have DOS recognize my HPFS partition, the drive letters will come up backwards: OS/2 drive E: will be drive F: in DOS, and OS/2 drive F: will be E: in DOS.

To solve my first problem I installed in my DOS CONFIG.sys a dummy driver, called DUMBDRV.sys, which simply reserves a drive letter.

Marcus Better, the author of iHPFS, is also the person who wrote DUMBDRV.sys, and he distributes it as freeware in the same package where iHPFS can be found. After DUMBDRV.sys reserves a drive letter it must be released before it can be used. This is what the program KILLDRV.exe does. To sum it put, what I did to access my OS/2 partition was to include the following lines in my DOS startup files: Added to CONFIG.sys ===================================    DEVICE=C:\BIN\HPFS4DOS\DUMBDRV.sys   ==> Reserve letter F:

Added to AUTOEXEC.bat: ===================================    C:\BIN\HPFS4DOS\KILLDRV F:           ==> Release Drive F:     C:\BIN\HPFS4DOS\IHPFS /L F:1 /C=512  ==> Install iHPFS as drive F: At this point in time, the HPFS drive is F:, and the FAT drive is E:, exactly backwards as I use them in OS/2. To swap them back to their right names I wanted to use the ASSIGN.com command. However, I discovered that beginning with version 6.0, Microsoft removed the ASSIGN.com from MS-DOS. Looking into my DOS manual I found that there is something called the "Supplemental Disk", which has the missing commands. After some fishing in the Internet, here is where I found ASSIGN for each of the current MS-DOS versions. Site: ftp.microsoft.com      Directory: /Softlib/MSLFILES =======================      ============================ FileName      Size     Date      Description ---   --- DOS6SUPP.EXE 537,136  Apr-1993  MS-DOS 6.0   Supplemental Files DOS62SP.EXE 763,182  Dec-1993  MS-DOS 6.2   Supplemental Disk Util SUP621.EXE  755,334  Jun-1994  MS-DOS 6.21  Supplemental Disk SUP622.EXE  781,125  Jun-1994  MS-DOS 6.22  Supplemental Disk MW6SUP.HQX  824,319  Sep-1994  Supplemental File Converters The file /Softlib/index.txt @ ftp.microsoft.com contains the description of all the service packs and software upgrades offered freely by Microsoft through the Internet. The above file descriptions were taken from this descriptor file. Use binary download to get these files with ftp.

Don't forget to expand the ASSIGN.co_ file to obtain ASSIGN.com. For this, you must use the MicroSoft provided EXPAND.exe command which come with our MS-DOS diskettes. Be careful: the OS/2 Expand command cannot unpack ASSIGN.co_. You can also find EXPAND in the Supplemental Disk that you can download from ftp.microsoft.com.

After all this, these are the lines that I added to my CONFIG.sys and AUTOEXEC.bat files: Added to CONFIG.sys: =======================================    DEVICEHIGH=C:\BIN\HPFS4DOS\DUMBDRV.sys   ==> Reserve F:

Added to AUTOEXEC.bat: =======================================    C:\BIN\HPFS4DOS\KILLDRV F:               ==> Release F:     C:\BIN\HPFS4DOS\IHPFS /L F:1 /C=512      ==> iHPFS is F:     C:\BIN\HPFS4DOS\ASSIGN E=F F=E           ==> Swap E: <==> F: As I mentioned before, I maintain all my HPFS4DOS in my C:\BIN subdirectory, and all these commands and drivers are invoked from there.

CD-ROM and RamDisk
The couple [DUMBDRV.sys, KILLDRV.exe] can be used to install a RamDisk in a higher drive letter than the one for the CD-ROM. The problem is that to access the CD-ROM from DOS it is necessary to load MSCDEX.exe in the AUTOEXEC.exe file, which is processed after CONFIG.sys.

The trick it to invoke DUMBDRV.sys in CONFIG.sys before creating the RamDisk, to force the RamDisk to use a higher drive letter. Later on, when AUTOEXEC.exe is being processed, the drive letter grabbed by DUMBDRV.sys can be released for MSCDEX.exe to use, as it's shown in the following code: Added to CONFIG.sys: =========================================    DEVICEHIGH=C:\BIN\HPFS4DOS\DUMBDRV.sys    ==> Reserve F:     DEVICEHIGH=C:\BIN\HPFS4DOS\DUMBDRV.sys    ==> Reserve G:

DEVICE=C:\BIN\SB16\CCD.SYS /D:MSCD001    ==> CD-Rom driver DEVICE=C:\DOS\RAMDRIVE.SYS 3072 256 48 /E ==> RamDrive H:

Added to AUTOEXEC.bat: =========================================    C:\BIN\HPFS4DOS\KILLDRV G:                ==> Release G:     C:\DOS\MSCDEX.EXE /D:MSCD001 /M:4 /L:G    ==> CD-Rom is G:

C:\BIN\HPFS4DOS\KILLDRV F:               ==> Release F:     C:\BIN\HPFS4DOS\IHPFS /L F:1 /C=512       ==> F: is HPFS C:\BIN\HPFS4DOS\ASSIGN E=F F=E           ==> Swap E: <==> F:

Further explanations
The following text comes from the documentation in the AMOS3.doc file that accompanies the AMOS.exe HPFS driver, written by Allan Mertner . AMOS v3 has capabilites similar to iHPFS and HPFS_access. I transcribe his explanation here for your convenience.
 * In some cases, the drive letters AMOS v3 assigns to your partitions differs from the ones you usually have under OS/2. This is sad, but there is not much I can do about it - and there is a workaround.

The problem for AMOS v3 is, of course, that it cannot use a drive letter which has already been used by DOS. Let's have an example:

Joe has a hard disk which is partitioned like this: C: FAT    Dos Boot drive D: HPFS   OS/2 Boot drive E: HPFS   Lots of data In addition to this, Joe also has a CD-ROM assigned to drive letter F:.

When Joe boots DOS, DOS assigns the following drive letters: C: FAT    Dos Boot Drive D: CDROM C: FAT    Dos Boot drive D: CDROM E: HPFS   OS/2 Boot drive F: HPFS   Lots of data This is the easy case! Joe could either use the /L: parameter for MSCDEX to load the CD-ROM as drive F:, or load AMOS v3 before loading MSCDEX - this would give him his normal drive lettering.
 * because DOS cannot see the HPFS drives. When Joe now loads AMOS v3, the first available drive letter is E:.  AMOS v3 cannot reassign drive letters used by DOS, so the final drive lettering becomes

Now Joe adds another hard disk, containing one single partition, which he has formatted using the FAT file system. His drive letters look like this: OS/2 DOS C:  C:   FAT    Dos Boot Drive D:  F:   HPFS   OS/2 Boot Drive E:  G:   HPFS   Lots of data F:  D:   FAT    Joe's new disk G:  E:   CD-ROM

Everything is wrong! And loading the drivers in a different way will not change the fact, that Joe's new disk is occupying the D-Drive. This is a lesson learned: Try to avoid placing FAT drives AFTER HPFS drives. Always, always, always keep FAT drives before the HPFS drives. Since this is not always possible in real life, fortunately there is a solution that works in most cases: ASSIGN.

The DOS ASSIGN utility can change the meaning of drive letters, and can actually solve Joe's problems. All he has to do is write ASSIGN D=F,F=D,E=G,G=E This effectively swaps the meaning of drive letters D-F, and E-G. If YOU find a program where this workaround does not work, please let me know...

Switching HPFS drivers
I wrote the REhpfs.bat command to change from iHPFS and HPFS_access. To keep track of things, I create a file in the C:\ root directory that tells which of the two drivers is currently in use: C:\IHPFS       ==> File exists when iHPFS is loaded C:\HPFSacc.ess ==> File exists when HPFS_access is loaded In my AUTOEXEC.bat file I always load iHPFS, and I can later swap it out to use HPFS_access. If I go the other way around then HPFS_access won't unload. Taking all this into account, these are the relevant contents of my CONFIG.sys and AUTOEXEC.bat file: Added to CONFIG.sys: =========================================    DEVICEHIGH=C:\BIN\HPFS4DOS\DUMBDRV.sys    ==> Reserve F:     DEVICEHIGH=C:\BIN\HPFS4DOS\DUMBDRV.sys    ==> Reserve G:

DEVICE=C:\BIN\SB16\CCD.SYS /D:MSCD001    ==> CD-Rom driver DEVICE=C:\DOS\RAMDRIVE.SYS 3072 256 48 /E ==> RamDrive H:

Added to AUTOEXEC.bat: =========================================    C:\BIN\HPFS4DOS\KILLDRV G:                ==> Release G:     C:\DOS\MSCDEX.EXE /D:MSCD001 /M:4 /L:G    ==> CD-Rom is G:

C:\BIN\HPFS4DOS\KILLDRV F:               ==> Release F:     C:\BIN\HPFS4DOS\IHPFS /L F:1 /C=512       ==> F: is iHPFS C:\BIN\HPFS4DOS\ASSIGN E=F F=E           ==> Swap E: <==> F:

IF EXIST C:\HPFS_acc.ess del C:\HPFS_acc.ess ==> Delete other ECHO Using iHPFS>C:\iHPFS. ==> Create C:\iHPFS.

REhpfs.bat

 * REhpfs.bat ==> Reload a new driver to access HPFS partitions from DOS
 * - /A Reloads Andreas Kinzler's HPFS_access
 * - /I Reloads Marcus Better's iHPFS
 * - swaps back E: and F: drive letters
 * - Reloads current driver if no arguments are issued.

@echo off


 * Synchronize current ASSIGN.com version with MS-DOS's

VER | FIND /I "6.20" >NUL IF ERRORLEVEL 1 echo REhpfs.bat: Works ONLY with MS-DOS v6.20 IF ERRORLEVEL 1 goto _out


 * Change back drive letters to their original names

CD E:\ CD F:\ SMARTDRV /c /r /q C:\BIN\HPFS4DOS\ASSIGN E=E F=F
 * C:\BIN\HPFS4DOS\ASSIGN


 * Remove the HPFS driver

IF EXIST C:\iHPFS. C:\BIN\HPFS4DOS\IHPFS  /u IF EXIST C:\HPFS_acc.ess C:\BIN\HPFS4DOS\WIN_ENG


 * When HPFS_access is installed at boot time, it CANNOT be uninstalled
 * - That's the reason NOT to load it before any other HPFS driver

MEM /C | FIND /I "HPFS" | FIND "8,048" >nul IF NOT ERRORLEVEL 1 echo ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ IF NOT ERRORLEVEL 1 echo █ CANNOT uninstall HPFS_access because it was loaded at boot █ IF NOT ERRORLEVEL 1 echo ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀


 * This code looks for the following line in the MEM /c listing:
 * Name          Total       =   Conventional   +   Upper Memory
 * MSDOS      17,245   (17K)     17,245   (17K)          0    (0K)
 * HPFS        8,048    (8K)      8,048    (8K)          0    (0K)
 * MSDOS      17,245   (17K)     17,245   (17K)          0    (0K)
 * HPFS        8,048    (8K)      8,048    (8K)          0    (0K)


 * Reloads HPFS

IF (%1)==(/a) goto _HPFS_access IF (%1)==(/A) goto _HPFS_access IF (%1)==(/i) goto _iHPFS IF (%1)==(/I) goto _iHPFS IF EXIST C:\iHPFS. goto _iHPFS IF EXIST C:\HPFS_acc.ess goto _HPFS_access goto _iHPFS


 * _HPFS_access

IF EXIST C:\iHPFS. del C:\iHPFS. ECHO Using HPFS_access>C:\HPFS_acc.ess C:\BIN\HPFS4DOS\WIN_ENG F=1 1024b goto _swap


 * _iHPFS

IF EXIST C:\HPFS_acc.ess del C:\HPFS_acc.ess ECHO Using iHPFS>C:\iHPFS. C:\BIN\HPFS4DOS\IHPFS /L F:1 /C=512 goto _swap


 * _swap
 * Swap again drive letters

C:\BIN\HPFS4DOS\ASSIGN E=F F=E


 * _out
 * REhpfs.bat ==> End of file