IDE Info - Danis

I found this snippet off the eCS mailing list so interesting I thought I would include it here as well.

I have also included other snippits from Dani as well.

On Fri, 03 Nov 2000 18:16:13 -0400, iallen wrote: >Was it on SCSI? Don't forget that in the past with EIDE (ATA or whatever >you want to label it) the 1024 cylinder mark has been a moving target. >One time around 512MB, then for some (other?) reason there was a limit >around 4GB, and now at around 8.3GB we find the present one in most cases. >The changes may give the idea that the system was getting over the 1024 >cylinder problem, but they were not in the case of EIDE at least. > >If something was getting around it previously, such as on (some?) SCSI >systems, then there must have been a switch from CHS translation to >obtaining an IPL just using sector numbers for addressing. Perhaps now >the EIDE systems are now _also_ able to use sector addressing only, and >the 1024 cylinder problem is bypassed that way. There is a definite >change, but perhaps you did not catch the statement I made earlier that it >was EIDE drives I referred to. I've seen claims that people with SCSI >systems did not have a limit, but I've yet to see them explain how. If >INT13 has the limit and is in use, what did it switch to instead? Has SCSI >had the BIOS extensions for some time perhaps?

Let me step in to put some light onto this subject...

We are talking about a scenario, where the BIOS is involved in the communication between the application and the disk. Thus, the application doesn't talk to the disk but rather an abstract interface that the BIOS provides: the INT13 API. This is true for both ATA and SCSI systems.

Now, what most people don't know (and usually don't have to care about), is the fact, that up to *two* different CHS geometry translations may be in effect during this process! Let's look at them in detail.


 * 1) The interface between the application and the BIOS

The 'old' INT13 API provides an CHS-style interface only:

10 bits for the cylinder number (C), max. range 0..1023 8 bits for the head number     (H), max. range 0..254 6 bits for the sector number   (S), max. range 1..63

The upper limit of 254 for H is because of programming errors in some MSDOS flavours, the lower limit of 1 for S is for historical reasons. CHS combined gives a maximum addressable range of up to 1645056 sectors (7.84GiB). This limit is imposed by the API spec only, I didn't talk about any kind of hardware up to now!

The BIOS chooses a "virtual" geometry based on the above limits and some selectable flavours in the BIOS settings (normal/large/LBA). Both partners using this interface have to agree on this geometry. This "virtual" geometry is used for partitioning and formatting the disk!

The 'new' (enhanced) INT13 API provides an LBA-style interface only with a 64 bit LBA value. This gives a maximum addressable range of 8388608 PeB.


 * 2) The interface between the BIOS and the disk

In case of an ATA system, the ATA spec offers two addressing schemes from day one - an CHS addressing scheme and a LBA one. The SCSI spec knows about the LBA addressing scheme only.

The ATA CHS addressing is as follows:

16 bits for the cylinder number (C), max. range 0..65535 4 bits for the head number     (H), max. range 0..15 8 bits for the sector number   (S), max. range 1..255

This gives a maximum addressable sector range in CHS mode of 267386880 sectors (127.5GiB) but the spec allows this to be limited to (16383/16/63) or 16514064 sectors (7.8GiB). Most devices don't support CHS addressing beyond this limit.

The drive proposes a "best" CHS translation geometry for this communication level, but the BIOS may decide to override that within the above limits.

The ATA LBA addressing uses a 28 bit LBA value which offers a maximum addressable range of 268435456 sectors (128GiB). There are proposals to lift this limit to up to 64 bit in the forthcoming ATA specs. For details have a look at the ANSI T13 committee web site.

In case of SCSI, there were different upper limits of the addressable LBA range in the past (6/10/12 byte commands). For details have a look at the ANSI T10 committee web site.

As you can see, with the LBA addressing scheme there never was a true limit on the addressable range in both ATA and SCSI systems, the only limit is the 'old' BIOS CHS interface.

The OS/2 BootManager that comes with WSeB (and MCP/eCS as well), and the parts involved in later OS/2 boot stages (partition sector loader, OS2BOOT, OS2LDR, ...), can take advantage of the 'new' INT13 BIOS API to overcome the dreaded 1024 cylinder limit, and boot OS/2 beyond this old barrier.

Ciao, Dani

Daniela Engert, systems engineer at MEDAV GmbH

Excellent explanation! I would just add that there is a little catch to all this translation stuff: it must be ensured that once certain translation is used to partition and format a HD, it must stay the same. If it's changed, the computer won't find data on the HD where it expects to and even a complete data loss may occur as a result.

Everything should be OK as long as no one touches the BIOS settings (LBA/Large etc.) and doesn't actually install the HD in another system with different settings.

A little secret:

Both IBM1S506 and DaniS506 don't mind! When a disk drive is found, the partitions are scanned. If there is a partition with a valid BPB found (or at least a consistent MBR in case of DaniS506), the driver uses this CHS info instead of the one reported from the BIOS. This way you can shuffle disks between different machines without hassles.

Ciao, Dani

>I believe, and correct me if I am wrong. But as long as your operating systems >are located in the first 8 gig you are okay.

These first 8 GiB (actually, 7.8 GiB) are within the 1024 boot limit if the disk is partitioned/formatted using the common "LBA" geometry mapping (255 heads/63 sectors) most BIOSes offer today. If a different maping is used, the bootable area shrinks - sometimes down to 504 MeB in case of the most inappropriate mapping for large disks: the "normal" mapping (15 or 16 heads/63 sectors).

With the advent of WSeB/MCP/eCS this 1024 cylinder limit is no longer an issue!

"June 25, 2000 - I'm sure there are a few of you out there wrestling with windoze 2000 on an otherwise perfectly good OS/2 machine. There has been a solution to the problem with windoze eating IBM's Boot Manager for sometime, but it required having a file from an earlier pre-release version. Here is an as always elegant solution from Daniela Engert as posted on the OS/2 Hardware list:

There is a much better solution:

using your favourite sector editor, change in BootManager's first sector

1) sectors/cluster, number of FATs, number of root dir entries, sectors/FAT  to zero   2) reserved sectors to 32 3) format signature to BOOTMGR

With these changes Win2k will no longer touch the BootManager in any way."

"Voice Newsletter 08/2000 - OS/2 Tips" http://www.os2voice.org/VNL/past_issues/VNL0800H/vnewsft.htm :

"July 23, 2000 - Danilea Engert asks "Maybe you are interested in patching FP13 FDISK so that it writes out a modified BootManager which will not be affected by Win2K":

For FP13 FDISK use this script file (the Bootmanager sector image is at  file offset 13404h)

--- FDISK.PAT -- FILE FDISK.COM VER 13411 04 CHA 13411 00 VER 13412 01 CHA 13412 20 VER 13414 02 CHA 13414 00 VER 13416 02 CHA 13416 00 VER 1341A 0C CHA 1341A 00 VER 1343A 46415420202020 CHA 1343A 424F4F544D4752

--- FDISK.PAT --

and run "PATCH FDISK.PAT /A" from the x:\OS2 directory. Then run FDISK and recreate the Bootmanager.

Other FDISK versions may have the sector image at another offset.

Ciao,

Dani"

>Under eCS my USB Zip drive causes 2 drive icons to show >up. One icon is a removable drive icon and the other >is a floppy icon. The floppy icon is the bad one. >This is somewhat annoying. Comments?

I've repeated the following at various places in the past; thus another time:

BASEDEV=USBMSD.ADD /FIXED_DISK /FLOPPIES:0 /DEVICES:1

Hard Drive Geometry and OS/2 (This article appeared in Dec 2000 VOICE netzine)

By Daniela Engert December 2000

Referenced web sites: T10 Technical Committee (I/O Interfaces):http://www.t10.org Technical Committee T13 AT Attachmenthttp://www.t13.org

Let me step in to shed some light onto the subject of how ATA (IDE) and SCSI sub-system drive geometry is seen from within OS/2.

We are talking about a scenario, where the BIOS is involved in the communication between the application and the disk. Thus, the application doesn't talk to the disk but rather an abstract interface that the BIOS provides: the INT13 API. This is true for both ATA and SCSI systems.

Now, what most people don't know (and usually don't have to care about), is the fact, that up to *two* different CHS (Cylinder, Head, Sector) geometry translations may be in effect during this process! Let's look at them in detail.


 * 1) The interface between the application and the BIOS.

The 'old' INT13 API provides a CHS-style interface only: 10 bits for the cylinder number (C), max. range 0..1023 8 bits for the head number (H), max. range 0..254 6 bits for the sector number (S), max. range 1..63

The upper limit of 254 for Heads, is because of programming errors in some MSDOS flavours, the lower limit of 1 for Sectors is for historical reasons. CHS combined gives a maximum addressable range of up to 1,645,056 Sectors (7.84GiB). This limit is imposed by the API spec only, I haven't talked about any kind of hardware up to now!

The system BIOS chooses a "virtual" geometry based on the above limits and some selectable flavours in the BIOS settings (normal/large/LBA). Both partners using this interface have to agree on this geometry. This "virtual" geometry is used for partitioning and formatting the disk!

The 'new' (enhanced) INT13 API provides an LBA-style interface only with a 64 bit LBA value. This gives a maximum addressable range of 8,388,608 PeB.

SCSI host adapters usually provide an INT13 API as well. They have to do so if they offer the boot capability. Thus SCSI attached disks suffer from the same BIOS limitation as ATA disks do. To be compatible these SCSI BIOS implementations need to fake up "virtual" geometries as well. The most common standard here is the so called "Adaptec" standard which uses different CHS mapping for disks with less than 1 GiB, less than 2GiB, and more than 2GiB (IIRC) - sometimes this mapping is selectable in the SCSI BIOS. LSI (formerly known as NCR or Symbios) based SCSI host adapters behave differently: if you attach a partitioned/formatted disk to such a controller, its BIOS will use the CHS geometry found in the MBR instead of calculating one by itself based on the disk size.

The same rules about matching "virtual" geometries reported by the BIOS and used during disk partitioning/formatting apply here as well!


 * 2) The interface between the BIOS and the disk.

In the case of an ATA system, the ATA specification offers two addressing schemes from day one - a CHS addressing scheme and an LBA one. The SCSI specification knows about the LBA addressing scheme only.

The ATA CHS addressing is as follows: 16 bits for the cylinder number (C), max. range 0..65535 4 bits for the head number (H), max. range 0..15 8 bits for the sector number (S), max. range 1..255

This gives a maximum addressable sector range in CHS mode of 267,386,880 sectors (127.5GiB) but the spec allows this to be limited to (16,383/16/63) or 16,514,064 sectors (7.8GiB). Most devices don't support CHS addressing beyond this limit.

The drive proposes a "best" CHS translation geometry for this communication level, but the BIOS may decide to override that within the above limits.

The ATA LBA addressing uses a 28 bit LBA value which offers a maximum addressable range of 268,435,456 sectors (128GiB). There are proposals to lift this limit to up to 64 bit in the forthcoming ATA specs. For details have a look at the ANSI T13 committee web site.

In the case of SCSI, there were different upper limits of the addressable LBA range in the past (6/10/12 byte commands). For details have a look at the ANSI T10 committee web site.

As you can see, with the LBA addressing scheme there never was a true limit on the addressable range in both ATA and SCSI systems, the only limit is the 'old' BIOS CHS interface.

The OS/2 BootManager that comes with WSeB (and MCP/eCS as well), and the parts involved in later OS/2 boot stages (partition sector loader, OS2BOOT, OS2LDR, ...), can take advantage of the 'new' INT13 BIOS API to overcome the dreaded 1024 cylinder limit, and boot OS/2 beyond this old barrier.

A little secret:

Neither OS/2 IDE driver (IBM1S506 and DaniS506) minds if one changes the BIOS settings (LBA/Large etc.) or moves the drive to a new system with different settings! When a disk drive is found, the partitions are scanned. If there is a partition with a valid BPB found (or at least a consistent MBR in the case of DaniS506), the driver uses this CHS info instead of the one reported from the BIOS. This way you can shuffle disks between different machines without hassles.

Glossary of terms used in this article:

ATA: (AT Attachment) The specification for IDE drives.

Int 13: A base operating system interrupt, dating from DOS, used to activate disk functions, such as seek, read, write and format.

LBA: (Logical Block Addressing) A method used to support IDE hard disks larger than 504MB (528,482,304 bytes) on PCs. LBA provides the necessary address conversion in the BIOS to support drives up to 8GB. BIOSs after mid 1994, which are sometimes called "Enhanced BIOSs," generally provide LBA conversion.

MBR: A disk drives Master Boot Record.

BPB: Boot Parameter Block, part of the first sector of IBM/MS compatible filesystems, describes the basic organization of the partition (amongst others, the CHS geometry used while formatting the partition).

PeB: Petabyte, 1 PeB = 2^50 byte = 1024 TeB = 1024 * 1024 GiB

Daniela Engert, systems engineer at MEDAV GmbH. Daniela is the author of the DANIS506.ADD and DANIATAPI.FLT replacement drivers, as well as a number of other freeware OS/2 items.

>Unfortunately we were unable to upgrade our old server to support harddrives > >20 MB. TYAN Tech support said that the Intel HX chipset can only go to 20GB, so >there's an even more fundamental limitation than the typical old BIOS limit of 8 >GB (if true).

This is crap! Even the oldest IDE controller hardware has *no* limitation on the size of attached disk drives. With the same register set you can address any sector provided both the host driver and the disk firmware agree on the interpreation of the register contents. This is how the new ATA-6 spec overcomes the current 128GiB addressing limit found with software/firmware that conforms to up to ATA-5 only.

Hi Michal!

On Wed, 28 Feb 2001 10:52:51 -0100, Michal Necasek wrote:

>>I agree.. It is very nice to see someone not constrained by >>contamination and licensing rules extend the code to where >>it needs to be.. I tried in vain to get specs from VIA and >>other vendors for years, but they refused to supply it to 'IBM'..

> But did Dani really get all the specs? I think for some >chipsets she just used some kind of PCI probe to see what >the BIOS (or Windows drivers or whatever) is doing...

Yes that's true (at least it was in early stages of chip support). Even if there are no official specs, there are other sources to gain some insight from. The following list shows the current support status of the various IDE chips. As you can see, some kind of documentation is available for almost all chips.

Ciao, Dani

Vendor 			      ATA	ATAPI	  ATA33 ATA66 ATA100 Device			    PIO  DMA  PIO  DMA docs Revision			   32bit     32bit south bridge id	 south bridge revision

0x8086 Intel 0x1230 PIIX			     x    x	x   x	    -	  -	-    x     < 02			      x    -	x    -	   -	  -	-    x       0x84C4 < 04			     x    -	x    -	   -	  -	-    x   0x7010 PIIX3 		      x    x	x    x	   -	  -	-    x   0x7111 PIIX4 		      x    x	x    x	   x	  -	-    x   0x7199 PIIX4 		      x    x	x    x	   x	  -	-    x   0x84CA PIIX4 		      x    x	x    x	   x	  -	-    x   0x2411 ICH			      x    x	x    x	   x	  x	-    x   0x7601 ICH			      x    x	x    x	   x	  x	-    x   0x2421 ICH0			      x    x	x   x	    x	  -	-    x   0x244B ICH2			      x    x	x   x	    x	  x	x    x   0x244A ICH2			      x    x	x   x	    x	  x	x    x

known bugs: PIIX3: some chips 'forget' to assert the IRQ sometimes. These chips are not detectable in advance.

0x1106 VIA 0x0571 571 0x0586 < 0x20 586		      x    x	x    x	   -	  -	-    - >=0x20 586A/B 	      x    x	x    x	 x	  -	-    x       0x0596 < 0x10 596/A		      x    x	x   x	    x	  -	-    x	 < 0x10  596B		      x    x	x    x	   x	  x	-    x       0x0686 < 0x10 686		      x    x	x    x	  x	  -	-    - < 0x40 686A		      x    x	x    x	   x	  x	-    x	 >=0x40  686B		      x    x	x    x	   x	  x	x    - >= 0x8231		     x    x	x    x	   x	  x	x    -

known bugs: - all: no host side cable type detection. - all: the busmaster 'active' bit doesn't match the actual busmaster state. - 596B: don't touch the busmaster registers too early after interrupt

0x10B9 ALi 0x5229 M5229 < 0x20			     x    x	x    -	 -	  -	-   (x) < 0xC1	 1533, 1543E, 1543F  x    x	x    -	 x	  -	-   (x) < 0xC2	 1543C		     x    x	x    xR  x	  -	-   (x) 0xC3/ 0x12	 1543C-E	     x    x	x    xR (x)	  -	-   (x) < 0xC4	 1535, 1553, 1543C-Bx x   x	x    - x	  x	-   (x) >=0xC4	 1535, 1553	     x    x	x   x	    x	  x	x    -

known bugs: - 1535 and better: varying methods of host side cable type detection. - up to 1543C: busmaster engine 'active' status bit is nonfunctional in UltraDMA modes. - up to 1543C: can't do ATAPI DMA writes. - 1543C-E:	 UltraDMA CRC checker fails with WDC disks. - 1543C-Bx:	 must not stop busmaster reads with 0x08.

0x1039 SiS 0x5513 5513 < 0xD0			     x    x	x    x	   -	  -	-    x     >=0xD0			      x    x	x    x	   x	  -	-    x       >= 0x0530		      x    x	x    x	   x	  x	-   (x) >= 0x0635		     x    x	x    x	   x	  x	x    -

- older SiS: don't touch the busmaster registers too early after interrupt

0x1095 CMD 0x0640 CMD 640		     -    -	-    -	   -	  -	-    x     00   refuse! 0x0643 CMD 643 < 03			     x    x	x    x	   -	  -	-    x     >=03			      x    x	x    x	   x	  -	-    x   0x0646 CMD 646 < 03			     x    x	x    x	   -	  -	-    x     >=03			      x    x	x    x	   x	  -	-    x   0x0648 CMD 648		      x    x	x    x	   x	  x	-    x   0x0649 CMD 649		      x    x	x    x	   x	  x	x   (x)

known bugs: - 640: the enable bit of the secondary channel is erratic. You need to check both settings '0' and '1' for a populated channel. - 640: revision 0 doesn't work reliably. - up to 646: both channels share internal resources. Serialization is required.

0x105A Promise 0x4D33 Ultra/FastTrack33	     x    x	-    -	   x	  -	-    - 0x4D38 Ultra/FastTrack66	     x    x	-   (x)    x	  x	-    - 0x4D30 Ultra/FastTrack100	     x    x	-   (x)    x	  x	x    - 0x0D30 Ultra/FastTrack100	     x    x	-   (x)    x	  x	x    -

known bugs: - all: don't issue superfluous PIO transfer mode setups. - all: if any device is initialized to UltraDMA, you need to reset the channel if you want to select MultiWord DMA instead. - Ultra66/100: ATAPI DMA should work according to Windows drivers, but I can see channel lockups only.

0x1078 Cyrix 0x0102 CX5530		     x    x	x    x	 x	  -	-    x

known bugs: - all: busmaster transfers need to be cache line aligned instead of word aligned.

0x1103 HighPoint 0x0004 HPT 36x/37x <=01 HPT 366		     x    x	x    x	   x	  x	-    x       02 HPT 368		      x    x	x    x	   x	  x	-    - >=03 HPT 370		     x    x	x    x	   x	  x	x    x

known bugs: - HPT366: random failures with several disks. - HPT366: random PCI bus lock ups in case of too long bursts. - HPT366: IBM DTLA series drives must be setup 'inventively' to work reliably at Ultra DMA mode 4 speed.

0x1022 AMD 0x7401 AMD 751		     x    x	x    x	   x	  -	-    - 0x7409 AMD 756		     x    x	x    x	   x	  x	-    x   0x7411 AMD 766		      x    x	x    x	   x	  x	x    -

known bugs: - 756: no host side cable type detection. - 756: SingleWord DMA doesn't work on early chip revisions.

0x1191 AEC/Artop 0x0005 AEC 6210		     x    x	? ?	  x	  -	-    - 0x0006 AEC 6260		     x    x	? ?	  x	  x	-    - 0x0007 AEC 6260		     x    x	? ?	  x	  x	-    -

known bugs: 6210 (possibly 6260): task file registers are inaccessible until busmaster engine is stopped. possibly all: both channels share internal resources. Serialization is required.

0x1055 SMSC 0x9130 SLC90E66		     ? x	? ?	  x	  x	-    x

0x1166 ServerWorks 0x0211 OSB4			     x    x	? ?	   x	  -	-    x

known bugs: at least some chip revisions can't do Ultra DMA mode 2.

0x1045 Opti 0xC621    n/a		      x    -	? -	  -	  -	-    x   0xC558     Viper		      x    x	? ?	  -	  -	-    x   0xD568			      x    x	? ?	 -	  -	-    x     < 0xC700 Viper		      x    x	? ?	  -	  -	-    x     >=0xC700 FireStar/Vendetta? x   x	x    x	  x	  -	-    x   0xD721     Vendetta? x   x	x    x	  x	  -	-    x   0xD768     Vendetta		      x    x	x    x	    x	  -	-    x

known bugs: C621: both channels share internal resources. Serialization is required. FireStar: Ultra DMA works reliably only at mode 0.

Daniela Engert, systems engineer at MEDAV GmbH Grfenberger Str. 34, 91080 Uttenreuth, Germany Phone ++49-9131-583-348, Fax ++49-9131-583-11