Large Hard Drives in OS/2

The standard PC BIOS is limited to 1024 cylinders. In combination with limits on the number of sectors and number of heads, this limits the size of most IDE hard drives to precisely 504MB. (Note that "MB" is defined as 1024^2, or 1,048,576, bytes, while most drive manufacturers use it to mean "millions of bytes," e.g., 1,000,000 bytes, which often leads to confusion.) SCSI controllers normally impose a limit of 1GB, but many controllers allow this to be changed to increase the limit to 2GB or higher. The ATA-2 (aka "EIDE") standard also allows a way around this limit, but where SCSI requires only a controller that supports this option, with ATA-2 the controller and sometimes the motherboard's BIOS must work together and support this feature. A few recent BIOSes also support "LBA [Logical Block Addressing] mode," in which the sector-head-cylinder scheme is abandoned entirely. (SCSI hard drives actually use such a mode, and SCSI controllers convert this to the sector-head-cylinder coding the BIOS and many PC programs and OSes expect.) LBA mode does not, however, help to get around the 1024 cylinder limit, since LBA mode is not used during the booting of an OS. Many BIOSes link LBA mode and geometry re-mapping in a single BIOS option, however, which leads many to believe (incorrectly) that LBA mode is used to get around the 1024-cylinder limit.

One side note here: The geometry reported by the drive, and often printed on the drive, is usually a convenient fiction having more to do with these BIOS limits than with the actual physical construction of the drive. Most modern 1/2- and 1/3-height drives have just one or two platters, each with two heads, and the number of sectors per cylinder varies, with more sectors on outer tracks than inner ones. Because the BIOS can't handle a variable number of sectors per track, and because it can handle more heads than most drives have, the drive lies to the controller, which passes on the same lie to let the BIOS handle larger drives than it could if the true geometry were reported. ATA-2 controllers, if properly configured, will convincingly tell another lie to the OS. In SCSI, the drive talks to the controller (telling the truth, mostly) in LBA codes and the controller lies to the OS. ATA-2 "LBA mode" makes ATA-2 work like SCSI in this respect.

The upshot of the 1024-cylinder limit is that, in order to use an IDE drive larger than 504MB, or a similarly-sized ATA-2 drive connected to a system without large drive support in the BIOS and/or controller, either part of the drive will be unused or a special driver of some sort will be necessary. Older drivers tended to map 1GB drives into two parts, each with 1024 or fewer cylinders; newer drivers tend to extend the number of heads that the BIOS will accept and re-map the drive's geometry so that double (or quadruple or more) the number of heads is reported to the BIOS, and half (or a quarter or less) the number of cylinders. Unfortunately, these solutions sometimes confuse matters under OS/2, since OS/2 doesn't use the BIOS for hard drive access except during the boot process, and so OS/2 and DOS won't agree about the drive geometry or what's on the drive. OS/2 simply reads the drive as it reports itself -- that is, with too many cylinders for the BIOS to handle. Note, however, that recent FixPacks to OS/2 Warp include code to let OS/2 recognize many of the special DOS driver programs, and to map the drive's geometry appropriately.

In theory, these problems shouldn't occur with ATA-2 systems, since the geometry re-mapping should be handled by the controllers; but in some cases, the various components may not handle each others lies very well, and it all comes tumbling down, resulting in the sorts of problems you get with IDE systems. If you're in this boat, just treat your system as an IDE system, and read on.

There are four possible solutions to the 1024-cylinder problem under OS/2:


 * 1) Get the file ontrac.zip from any of a variety of OS/2 ftp or web sites. This contains a fix that allows OS/2 to use the DOS-style geometry, but I don't know how well it works. A similar fix is supposedly available for Maxtor's extender software, and perhaps for others.
 * 2) Recent OS/2 FixPacks contain code that allows OS/2 to handle at least some of these drive re-mapping schemes automatically. If you already have OS/2 installed and are trying to add a new drive, try these FixPacks. IBM also has a utility available that will create OS/2 install floppies using the latest drivers, but I don't recall the name of this utility. Look for it on software.watson.ibm.com.
 * 3) Re-partition and re-format the drive under OS/2. Any partitions you place entirely under the 1024 cylinder limit will be useable by DOS or to boot OS/2 or any other OS or to store OS/2's Boot Manager. The remainder of the disk will be useable only as an OS/2 HPFS data partition. You may also need to change your CMOS to reflect the number of heads and sectors printed in the drive's manual or on the drive itself, but only 1023 or 1024 cylinders. You may also need to use FDISK /MBR from DOS or FDISK /NEWMBR from OS/2 to eliminate the DOS extender software (it normally resides in the Master Boot Record of your first physical disk, and these commands rewrite this area).
 * 4) If you've got an ATA-2 drive and an IDE controller, a new ATA-2 controller may help, but only if the drive, controller, and maybe your motherboard's BIOS can cooperate (not all will). If you're considering a PCI EIDE controller, I strongly recommend you read Pat Duffy's PCI EIDE controller summary (which is posted to the comp.os.os2.misc newsgroup most Sundays, as there are pitfalls with many of these controllers

I favor options 3 and 4 because they don't rely upon an inherently kludgy patch to fix a BIOS limitation and because large drives benefit from HPFS to begin with. (Using a FixPack doesn't preclude these options.) More specifically, the FAT file system used by DOS allocates space for files in blocks which vary in size depending upon the size of the drive. The allocation block size varies as follows:

(There's some fudging around the low end of this scale, on partitions smaller than 32MB, if I recall correctly.) Thus, most large drives, if formatted for a single partition (which is how most arrive and how most users seem to leave them), end up using a 16K or larger allocation block size. Because files vary semi-randomly in size, the average file will end up wasting half of the allocation block size -- 8K or more in this case. This can add up to literally hundreds of megabytes of wasted disk space on a drive that's reasonably large.

HPFS, by contrast, uses an allocation block size of 0.5K no matter what the drive's size, so wasted drive space comes out to a few megabytes at most. (There's also a 0.5K overhead per file in HPFS for control structures.) HPFS is also more efficient and robust than FAT.

To give a concrete example, I have a 1724MB HPFS partition on my system which contains 14,827 files (it is, as of yet, mostly empty, with only 593MB in use by those files, for an average file size of just under 41K; this partition could probably hold over 40,000 files at this rate). This results in slack of about 3,700K plus 7,400K HPFS overhead -- roughly 11MB total "wasted" space (though whether the 7,400K HPFS structures should be considered "waste" is debatable at best). Under FAT, this would jump to a whopping 237,000K (just under 232MB).

If anything, the above paints a rosy picture of FAT under OS/2, since OS/2 tends to store a lot of Extended Attributes (EAs), each of which will consume at least one allocation block under FAT, although most files' EAs are only a few bytes in size. Under HPFS, there's room for some EAs in the base structures for each file, and additional EA storage comes in the HPFS 512-byte chunks, so EA overhead is much lower under HPFS. In addition, the vast majority of files on most hard disks are rather small in size, with a few truly huge files (like OS/2's swap file, which on my system starts at 30MB) dragging up the average. Because of this, the assumption of 50% of the allocation block size being wasted per file is a bit optimistic when the allocation block size gets very large.

If you need to use FAT for some reason, I'd recommend partitioning a large drive into smaller chunks -- 255MB at the most, or preferably less than 128MB -- to keep the allocation block size down.

One more tip: On a system with multiple physical disks, I recommend restricting primary partitions to the first physical disk whenever possible. The OS/2 manual has an excellent discussion of this in the section on "Setting Up a Hard Disk" (starting on page 297 in my copy of the Warp manual). In brief, putting a primary on the second physical disk will result in logical drive letters on the first physical disk changing when adding or removing physical drives. This is usually undesirable. Further, OS/2 and the BIOS may map drive letters for primary partitions differently when a primary exists on a second IDE controller or on a SCSI controller when an IDE controller is also present. In extreme cases, these problems have resulted in unbootable systems and/or an inability to install OS/2. Some FDISK programs don't let you install a logical partition unless a primary is also present, but OS/2's FDISK will allow this, as will some others, such as Linux's fdisk.

More detailed, but less OS/2-specific, information on large hard drives can be found on http://thef-nym.sci.kun.nl/~pieterh/storage.html. This is a very well-written FAQ covering lots of issues related to IDE and related technologies.