• Welcome to OS2World OLD-STATIC-BACKUP Forum.
 

News:

This is an old OS2World backup forum for reference only. IT IS READ ONLY!!!

If you need help with OS/2 - eComStation visit http://www.os2world.com/forum

Main Menu

OS2 & eCS 64 bit kernel? Do we need one? When? How?

Started by RobertM, 2008.08.24, 03:47:54

Previous topic - Next topic

RobertM

#15
Quote from: obiwan on 2008.09.02, 06:07:36
It might be useful to research what exactly constitutes 64-bit support. The technical implications of a 64-bit kernel are different whether you are talking about Windows or Linux, and OS/2 is neither. Much of this discussion to me seems very abstract, without a clear explanation of exactly what about the OS/2 kernel prevents a 64-bit binary from executing.

Absolutely nothing. If I understand correctly, JFS has to handle 64bit data structures. That's one step. The rest (instructions) could be handled via a thunking layer of sorts. OS/2 doesnt seem to impose a massive penalty in doing so between 16bit (device drivers) and 32 bit (higher end services). If a similar method could be written, then other than the larger memory support, a kernel replacement wouldnt be needed... and supposedly the kernel can handle more (memory) anyway - through certain rarely used APIs.

Quote from: obiwan on 2008.09.02, 06:07:36
That is what we are talking about, right? Ability to run 64-bit applications?

No, not necessarily. Though the other benefits would not necessarily require a true 64bit kernel either. Memory management is still hampered by (1) 32bit flat memory model (ie: 4GB addressable - with newer system components eating a larger and larger portion of that address space - such as newer video cards), (2) a scalable memory model, that by being able to access a 64bit address space would not be limited in shared and private memory arenas to the extent the current implementation is.

This would allow bigger file system caches, as well as transparent use of larger caches in things such as MySQL; as well as no more resource bottlenecks with Firefox (or trying to balance the virtualaddresslimit, cache sizes and such to allow FF and other apps to run without eventually running out of resources).

Quote from: obiwan on 2008.09.02, 06:07:36
We're not talking about rewriting the OS/2 kernel so it does everything the kernel does using 64-bit instructions instead of 16 and 32, right? That seems completely pointless.

Indeed. As it is, the 32bit kernel, IIRC, still maintains a bunch of 16bit assembler code for certain operations.

In doing further reading and digging online, it seems like the kernel can support up to 64GB - which only some tools and very few programs can use (Theseus for one).



Quote from: obiwan on 2008.09.02, 06:07:36
Else, are we talking about introducing a completely new kernel with a different abi, and a different name? Then what would we be introducing it to? That confuses me, because that would be a different OS. Not that there is anything wrong with using a different OS, if you want to. Just, if we're talking about a different OS, what does that have to do with OS/2? And why then talk about creating a new one, when others exist?

No, it wouldnt necessarily be. OS/2 doesnt use the same kernel as it did in the 1.x days. Some holdover code maybe (or more accurately, supposedly rewritten in the newer kernel with the same functionality included). But it was still an OS/2 kernel in OS/2.

Quote from: obiwan on 2008.09.02, 06:07:36
As far as I know, no-one has done any testcase to see what happens when you try to call a 64-bit instruction on OS/2.

Depends on what you deem a 64bit instruction. Software or hardware? Hardware wise... not a clue. Software wise, JFS.

As it is, the kernel should theoretically be able to support the Intel PAE mode - now, how useful that would be, I dont know (ie: can programs take advantage of all the extra memory the OS would then see?)

Quote from: obiwan on 2008.09.02, 06:07:36
It guess it might trap, but why? How is a 64-bit instruction passed to the processor? Does the kernel really prevent it? Could the existing kernel be slightly modified to allow the instruction to pass? If so, that can probably be accomplished.

Hmm... do such CPUs run in 64bit mode with virtual (or no) 32bit support -or- 32bit mode with no 64bit support? Not playing devil's advocate, I dont know. Is it like running earlier Intel CPUs in real or protected mode, and/or having to switch between the two? Or are both CPU instruction sets available at the same time?

Quote from: obiwan on 2008.09.02, 06:07:36
I really don't know the answers to these questions, and honestly I'm only mildly interested. It just seems to me that if anyone is seriously interested in executing 64-bit code on OS/2, the place to start is to read and experiment with exactly what prevents it, if anything. Not just jump to the conclusion that the kernel must be completely re-engineered for it, and make grandiose plans for this to come about somehow. Just because other OS's now have 64-bit versions of their kernels.

Maybe you all know something I don't know about this, and the reason you aren't saying it is because you assume everybody knows. However, since I don't know, and this piece is clearly missing from every thread, it would be useful to state it, with references. Otherwise I just don't get this.

It would indeed... but the people who would need to answer some of this are those who work - or worked - on the kernel.

Hopefully one of them will. I've made one of our kernel discussion threads known to Scott Garfinkle. Hopefully he will decide to provide some insight at some point.

Robert


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


mobybrick

I believe that when a CPU switches into x64 mode, it can continue to process 32-bit instructions.

The problem is that in x64 mode, 16-bit instructions no longer work. And given that the current OS/2 kernel still contains some 16-bit code, then that is a big problem.

On the horizon are disks >2Tb and systems with >4Gb RAM. Both of these are the limits of the current OS/2 kernel with its main API and/or ADD limit. These will need to be addressed, somehow, soon. Why soon? It'll take some time to develop and debug any workarounds IMHO (even if these are possible) and will need some time to test.

Just my 2c worth.

Regards,
Moby

RobertM

Quote from: mobybrick on 2008.11.30, 15:40:22
I believe that when a CPU switches into x64 mode, it can continue to process 32-bit instructions.

The problem is that in x64 mode, 16-bit instructions no longer work. And given that the current OS/2 kernel still contains some 16-bit code, then that is a big problem.

Not to mention device drivers and file systems... I guess that would mean the eradication of 16bit code, or a 16bit emulation mode or thunking layer to handle those calls...

Quote from: mobybrick on 2008.11.30, 15:40:22
On the horizon are disks >2Tb and systems with >4Gb RAM. Both of these are the limits of the current OS/2 kernel with its main API and/or ADD limit. These will need to be addressed, somehow, soon. Why soon? It'll take some time to develop and debug any workarounds IMHO (even if these are possible) and will need some time to test.

Just my 2c worth.

Regards,
Moby

Thanks Moby - that cleared up a lot. :)

Rob


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


mobybrick

Apparently there is a 65536 cylinder limit in the kernel. Which means we need an update to the Kernel now - not soon - but now - 32 bits or not. An apparent workaround in the latest Dani driver does not appear to work. It also explains why LVM barfs out with weird errors on my 750Gb disc :(

Regards,
Moby

Blonde Guy

The ability to handle disks larger than 500 GB is not a kernel limitation, but a limitation of the OS2DASD driver. Even 16-bit code could handle an arbitrary sized disk.

The kernel replacement is a much bigger job that the DASD driver replacement. Eventually we need the kernel replaced, but not because of larger disk drives.

I'm not aware of any replacement of OS2DASD that enables us to boot from larger than 500 GB drives.
Expert Consulting for OS/2 and eComStation

mobybrick

Quote from: Blonde Guy on 2008.12.04, 02:57:28
The ability to handle disks larger than 500 GB is not a kernel limitation, but a limitation of the OS2DASD driver. Even 16-bit code could handle an arbitrary sized disk.

Yes, I'd thought that after thinking about it - the Kernel has no perception of cyl/head/track - that, after all, is the whole point of the IFS with the DMD sitting on top of it.

Quote from: Blonde Guy on 2008.12.04, 02:57:28
The kernel replacement is a much bigger job that the DASD driver replacement. Eventually we need the kernel replaced, but not because of larger disk drives.

I'm not aware of any replacement of OS2DASD that enables us to boot from larger than 500 GB drives.

There is the LVM15105 package that contains a newer OS2DASD.DMD (and LVM engine) - its newer than the one that ships with the latest eCS RC5 - it supposedly enables >512Gb boot support.

I'm trying it at the moment - hopefully it will correct some LVM errors I see with any 750Gb disk.

Regards,
Moby.

mobybrick

Quote from: Blonde Guy on 2008.12.04, 02:57:28
The kernel replacement is a much bigger job that the DASD driver replacement. Eventually we need the kernel replaced, but not because of larger disk drives.

Although there is also supposedly an architectural 2Tb ADD limit (according to the docs with DANIS506) - this sounds more like a kernel issue. 2 Tb disks aren't that far away I'm afraid - this will need fixing soon IMHO...

RobertM

Hello all,

Regarding some of the questions myself and others brought up regarding the current kernel, Scott Garfinkle wrote me back (after reading some of the threads) and said:

Short answer: OS/2 has no visibility to PAE so anything more than 4gb is ignored. As for the addressability of 4gb plus devices, that's mapped by the BIOS. So there's 4gb there, some of which is clearly not generally-available RAM.


Thanks Scott...

Rob



|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


RobertM

#23
Quote from: mobybrick on 2008.12.04, 13:08:04
Quote from: Blonde Guy on 2008.12.04, 02:57:28
The kernel replacement is a much bigger job that the DASD driver replacement. Eventually we need the kernel replaced, but not because of larger disk drives.

Although there is also supposedly an architectural 2Tb ADD limit (according to the docs with DANIS506) - this sounds more like a kernel issue. 2 Tb disks aren't that far away I'm afraid - this will need fixing soon IMHO...

I believe this is a limitation in the DASD device drivers. They were updated to 48bit LBA - but limited to 2TB.
The current DASD drivers support both the older 28bit (128GB) and 48bit (128PB) addressing modes (but limited to 2TB?!?!).

In addition, as a note to those trying this. A boot drive is limited to 512GB - period (assuming the docs are correct for the DASD drivers). Thus, do not waste more than a 500GB hard drive on your boot drive. Any drives greater than 512GB should be added as secondary data drives.



The  current version  of OS2DASD.DMD  removes  this limit  and allows  total
  capacity  of the  hard drive  up to  2 Tb.  This support  has  the following
  options:

  1.  If the  drive is used as  a boot drive, only  first 512 Gb  of the drive
      space will be accessable to the system.

  2.  If drive is used only as a  data storage drive the total capacity (up to
      2 Tb) of drive will be accessable to the system.

      The two new parameters are available to select these options:

  o /BOOTABLE --- applied to  the unpartitioned large drive (during the bootup
      prior to the first partitioning  of the drive) and makes drive bootable,
      but limiting its available capacity to 512 Gb.


  o /FULLSIZE --- applied to  the unpartitioned large drive (during the bootup
      prior to the first partitioning of the drive) and allow use of the total
      capacity of  the drive (up to 2  Tb), but prevents the  drive from being
      made bootable.




I suspect that all the new variants of the IBMDASD driver have those requirements.

I also suspect (but could be wrong - maybe there are other issues?) that those people who have been having trouble with using greater than 512GB drives to full capacity, are having such trouble because the drive was set as bootable - thus limiting it's recognized size to 512GB. They thus simply need to install a different (<512GB) boot drive, and use their >512GB drive for data partitions only.

Also, these drivers all seem to have other updating prerequisites for them to work properly... LVM (required), IBM1S506 (suggested elsewhere), kernel (suggested elsewhere), etc. Failing to update those (and any others I missed) may also be the cause of certain problems. The tricky thing seems to be reading the prerequisites for each package to ensure every necessary component is updated.

HTH,
Robert


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


RobertM

#24
From the DASD readme:

  3.9  Very Large (128/137 Gb limit) Hard Drives Considerations (48 bit LBA)
  ____________________________________________________________________________

  Legacy LBA  (also know as 28 bit  LBA) maximum drive capacity  is limited to
  137 GB (268,435,455 sectors by 512  bytes each).  The 48 Bit LBA support was
  introduced  to break this  barrier and  provide support  for hard  disk with
  capacities larger than 137 GB.

  This version of  the IBM1S506.ADD driver has the 48 bit  LBA support and the
  driver will  automaticaly detect drives with  the capability of  48 bit LBA.
  (BUT) This 48 bit LBA support will allow the  use of hard disk drives with up to 2
  TB (Terabytes) (1 TB = 1024 GB) capacity.




Emphasis and (BUT) added. Which brings us back to the possibility that (a) it's a kernel limit? or (b) it's just an incomplete implementation of the 48 bit LBA specs...



|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


mikeoconwarp

Hi Robert,

Almost certain that all the >512GB disks are SATA not PATA - note the IBM1S506.ADD only supports INTEL SATA controllers, and not in RAID or AHCI modes, and even then only up to 1.5Gbps. The updated format.com etc., are included in the LVM-Update-14105.


RobertM

Quote from: mikeoconwarp on 2008.12.05, 05:59:03
Hi Robert,

Almost certain that all the >512GB disks are SATA not PATA - note the IBM1S506.ADD only supports INTEL SATA controllers, and not in RAID or AHCI modes, and even then only up to 1.5Gbps. The updated format.com etc., are included in the LVM-Update-14105.

Hi Mike,

No, they still make >512GB in PATA. Here's one... http://www.newegg.com/Product/Product.aspx?Item=N82E16822148133 - ya woulda thought they would have done away with PATA at those sizes by now... but I guess PATA isnt dead quite yet.

I primarily use only the DaniS506.add drivers anyway (which support more chipsets)... but neither version supports RAID on SATA because there is no full RAID on SATA - most of it is in the software and drivers for the particular OS instead of chipset based like various SCSI RAID.

None of that has ever really been an issue for me though (the RAID part) because I only use RAID on my big servers, and thus to conserve CPU cycles, I would only use SCSI with true hardware RAID even if both (SATA & SCSI) were available.

On a related note, someplace out there, there is a software RAID driver for OS/2... I've got it someplace, may dig it up and check it out.

Robert


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


mikeoconwarp

Hi Robert,

I knew I should have Googled on it!  :-[

FWIW on the eCS-Technical group at Yahoo, earlier today Massimo from Italy, after a long search/testing posted the following:

Hi all,

i'm glad to inform everyone that the LSI Logic MegaRAID SATA 150-6
Controller (both PCI and PCI-X) works perfectly under eCS with
the MRAID.ADD drivers bundled into eCS 1.2.x

http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sata/megaraid_sata_1506/index.html

Finally i've disk redundancy (raid 1 - mirroring) on my production server.

massimo s.

Mike

mobybrick

So we are actually in need of core code updates now, even before 2Tb drives hit us :(

The 512Gb boot drive limit is a significant problem as a large number of new systems have disks this size. Worse, some systems only support/have room for one disk drive (e.g. HTPCs) .

Funny, I always read the doscs as 512Mb boot drive to mean, er, boot *volume* but it does appear boot disk - this explains why I have LVM issues with my new 750Gb disk system- even tho the boot volume is just 40Gb. LVM sees all of the disk - or it appears to - but the Boot Manager menu always displays 'LVM-->'

Was OS2DASD.DMD ever made available on the DDK?

I wonder if IBM would pass on the code for little bits to Serenity, or if Serenity/Mensys could hire IBM to produce these updates? It looks as if we're going to begin to need this, unless another solution can be found?

Quote from: RobertM on 2008.12.05, 05:35:14
From the DASD readme:

  3.9  Very Large (128/137 Gb limit) Hard Drives Considerations (48 bit LBA)
  ____________________________________________________________________________

  Legacy LBA  (also know as 28 bit  LBA) maximum drive capacity  is limited to
  137 GB (268,435,455 sectors by 512  bytes each).  The 48 Bit LBA support was
  introduced  to break this  barrier and  provide support  for hard  disk with
  capacities larger than 137 GB.

  This version of  the IBM1S506.ADD driver has the 48 bit  LBA support and the
  driver will  automaticaly detect drives with  the capability of  48 bit LBA.
  (BUT) This 48 bit LBA support will allow the  use of hard disk drives with up to 2
  TB (Terabytes) (1 TB = 1024 GB) capacity.




Emphasis and (BUT) added. Which brings us back to the possibility that (a) it's a kernel limit? or (b) it's just an incomplete implementation of the 48 bit LBA specs...



RobertM

One thing I did note in my digging is that Daniela offered IBM her code for her project(s) (ie: at least the DaniS506 driver - maybe more), and they finally took her up on the offer and grabbed the v1.66 (I think that version) code. Whether or not they took other code I dont know.

Thus, also, whether or not that impacts future revisions to the OS2DASD driver I dunno either.

I read elsewhere a couple things (both, as with many things OS/2, conflict).

One such thing states that during the OS/2 Loader & early OS/2 kernel stages, that a mini "file system" "driver" (part of one of them) is used for disk access (hence some file naming and placement restrictions for files used during OS start). If my understanding is correct (of such process), that may be why there is a limitation in the boot drive size (and thus it is something that would have to be addressed in the loader and kernel).

The other related thing states that the loader and kernel hand off the file I/O requests to the actual file system drivers. But I have been thinking (and could be wrong, as I have less than a layman's understanding of the process involved) that the limitation still may be required by the loader and kernel in order for them to properly access each other and the file system drivers to start such process - thus again meaning the limitation is in the kernel and/or loader.

Either way, it seems that both conflicting explanations on how the boot process and OS load are handled would indicate the possibility that something in the kernel and/or loader is still the cause of 512GB drive limit for boot drives - I am presuming, if such is the case, they would need to be modified to properly handle larger geometries in the same fashion that the DASD and IDE/SCSI drivers do.


But again, that's all wild speculation...




More interesting to me is the limitation of 2TB for physical disk size noted in the same statement indicating 48 bit LBA support.




As for HTPCs, there is at least a solution that works easily enough - adding a second "hard drive" - something like a CompactFlash card on an IDE or SATA adapter that can be mounted in the case due to it's small size... and thankfully, since OS/2 doesnt use much space, it would be rather inexpensive, and allow (something I like to do) "partitioning" data onto a separate physical drive, with the OS on the other.

Not an elegant solution - but it at least makes installing OS/2 and eCS on an HTPC (or similar device) something achievable even with the small form factor and lack of additional drive bays.




Hopefully both limitation will soon be removed...

Robert


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|