Hybrid "64-Bit" eComStation

[djcaetano]

Hi there.
Yesterday I was thinking about 64 bits and eComStation. The biggest limitation eComStation has when working on 64 bit systems is the 4GiB barrier. I know it is not a simple limit to overcome, but I have been thinking of some measures to allow the use of extra memory, although it will be a limited one.
Several users are running eComStation inside a virtual machine and are ok with most of its limitations. On the other hand, I think running an OS inside another full blown OS is somewhat annoying for daily use (one have to cope with problems and annoyances of both OSs). This way, I was thinking if it is not possible to make one of those "OS/2 5.0" projects, like OS Free, make its kernel 64 bits and configure its environment to support 32 bit systems (virtualization with almost full access to guest OS) and run eComStation inside of it. What is the point of doing that?
Well, at first, this will enable the underlying OS (OS Free, for instance) to access the entire memory. The memory unnused by eComStation could be used as a virtual drive (emulation), which could be used as a normal harddisk by eComStation. This virtual drive space could be used in two main ways:

a) Store swapper.dat
b) Store temporary application files (like the "undo" files of Ceres Sound Studio!)

This would provide a usefull extra memory space. I know it is not the same and the lower and upper shared/private arena problems would remain the same, but at least some things could use it. At least, AFAIK, swapped data is not limited to 4GiB (Virtual Address Space should go up to 64TiB).
Would that be possible (considering the available community/serenity resources)?

Regards,
Daniel Caetano

PS: As an extra, this approach would provide a use for these "new OS/2 kernel projects", allowing them to improve faster. In time, some original OS/2 services could be moved to the new kernel and privileges could be removed from the guest OS... allowing a slow transition between systems.

[Doug Bissett]

a) Store swapper.dat

I haven't seen any indication that swapper.dat ever gets used, when I have 1.5 GB, or more, installed. It may be possible to get it to be used, but I don't think I have enough programs installed to be able to do that, unless I run more than one virtual machine.

In fact, I have been using RAMFS to create a memory drive for the %TEMP% stuff, when I have more than 2 GB installed. I think that 2 GB would be the minimum to be able to do that, but I have had no indication that it causes a problem.

It might take a bit of fancy programming, but I think that modifying RAMFS to use memory above the (normally) accessible area, might be possible, and it wouldn't involve playing with the kernel.

[Dave Yeo]

I use a RAMFS volume with 1 GB, seems to work fine. Also with 1 GB, my swap hardly ever grows to over 300 MB so 1.5 GB should be ideal. I did have Thunderbird run away one day and the system crashed with a swapfile full error. Seems that it was actually a 2 GB limit on the swapfile.

[Neil Waldhauer]

One of the strong points of running OS/2 is to continue to run the 16-bit parts and old programs. But with the x86-64 architecture, the real mode and virtual mode are not available to programs running in long (64-bit) mode.

Getting OS/2 to run in long mode would be a long struggle. We'd lose a lot of what makes OS/2 what it is.

It would probably be easier just to port the parts of OS/2 that you like to Linux.

[djcaetano]

Hi Neil,
There is no way to keep V86 and access memory above 4GB? Is it possible to enable the use of PAE on eComStation?

[Andy Willis]

Froloff (nick on netlabs IRC) had written a device driver to use more memory.  I don't see a need here for more than 3G of memory, I run out of shared memory long before I start to use my RAM up but if the additional memory (such as a W510 here with 16G) could use the rest for a RAM drive would be handy for doing builds (a 13G RAM drive would be really nice).  If there were a way to increase shared memory would be really useful but I think exponentially more work.
Andy

[Roderick Klein]

Or you can see if the DLL's can be loaded into high memory instead of trying to increase the shared memory of OS/2.
Thats what we are currently experimenting on and analyzing with memory dumps why this fail.

Regards,

Roderick Klein
Mensys B.V.

[conradc1234]

Just out of curiousity - was there a 64bit version of eComStation very early on in the development cycle? I have a cd here labelled as such but it  won't boot (quite old at this stage)....
Thanks,
Conrad.

[ivan]

Conrad, I think you will find that was for the IBM PowerPC - a very rare thing indeed.

[Roderick Klein]

Hi there.
Yesterday I was thinking about 64 bits and eComStation. The biggest limitation eComStation has when working on 64 bit systems is the 4GiB barrier. I know it is not a simple limit to overcome, but I have been thinking of some measures to allow the use of extra memory, although it will be a limited one.
Several users are running eComStation inside a virtual machine and are ok with most of its limitations. On the other hand, I think running an OS inside another full blown OS is somewhat annoying for daily use (one have to cope with problems and annoyances of both OSs). This way, I was thinking if it is not possible to make one of those "OS/2 5.0" projects, like OS Free, make its kernel 64 bits and configure its environment to support 32 bit systems (virtualization with almost full access to guest OS) and run eComStation inside of it. What is the point of doing that?
Well, at first, this will enable the underlying OS (OS Free, for instance) to access the entire memory. The memory unnused by eComStation could be used as a virtual drive (emulation), which could be used as a normal harddisk by eComStation. This virtual drive space could be used in two main ways:

a) Store swapper.dat
b) Store temporary application files (like the "undo" files of Ceres Sound Studio!)

This would provide a usefull extra memory space. I know it is not the same and the lower and upper shared/private arena problems would remain the same, but at least some things could use it. At least, AFAIK, swapped data is not limited to 4GiB (Virtual Address Space should go up to 64TiB).
Would that be possible (considering the available community/serenity resources)?

Regards,
Daniel Caetano

PS: As an extra, this approach would provide a use for these "new OS/2 kernel projects", allowing them to improve faster. In time, some original OS/2 services could be moved to the new kernel and privileges could be removed from the guest OS... allowing a slow transition between systems.

The real thing that will give it more breathing space s that we are working on fixing the loading of DLL's into high memory.
But thats not finished currently. Still in R&D stages.

Roderick Klein
Mensys B.V.

[Andreas Schnellbacher]

Hi there.
Yesterday I was thinking about 64 bits and eComStation. The biggest limitation eComStation has when working on 64 bit systems is the 4GiB barrier.

The real thing that will give it more breathing space s that we are working on fixing the loading of DLL's into high memory.
But thats not finished currently. Still in R&D stages.

Roderick, this thread is about 64 bit. Sooner or later 3 GB won't suffice anymore. I'm sure, the time isn't now, for non-server arrangements. E.g. DB apps (are there still any running with OS/2?) need more RAM for higher performance.

Daniel, the answer from Ivan was the only serious one, so far. No, we won't experience updates to the system to allow more than about 3 GB. IBM support is dead. We will rather manage to get the desktop ported to an OSS system. But also this is far away from reality.

[eirik]

Dear all,

backwards compatibility is one issue, adapting to new hardware and computing is another.  Backwards compatibility means, as Neil remarks, the possibility of running older applications.  For many eCS customers this is an important issue.  But as older applications gradually are phased out, surviving on backwards compatibility is a dangerous and non-sustainable long-term strategy.  This does not imply eCS developers should disregard backwards compatibility, but that it should not carry as much weight as it appears to do today.

The future lies in meeting new computing needs on modern hardware.  ACPI is important here.  In the longer term, however, being able to use more than 4 GB of memory will be essential.  The current work that Roderick describes on being able to load more DLLs into high memory is a step in the right direction, but can hardly be called a long-term solution.  With the current speed in hardware development, in just a few years we will be looking at PCs with 10-20 GB in memory with the capability of processing large amounts of data fast.

In addition, connectivity is key.  For laptops (and in the future also stationary PCs)  this entails wireless access, an area where I unfortunately have seen little progress the last few years.  Getting the MultiMac project rolling again as GenMac has its clear limitations (and I know as I have tried hard to get wireless working on my T410), is important to attract new users and hold on to many current users.

User friendliness, flexibility to design the work environment, and low vulnerability towards viruses (high security) are the best selling points of eCS.  Unfortunately, this is not worth much in the longer term if we are unable to meet future computing needs.

Eirik

[RobertMauro]

a) Store swapper.dat

I haven't seen any indication that swapper.dat ever gets used, when I have 1.5 GB, or more, installed. It may be possible to get it to be used, but I don't think I have enough programs installed to be able to do that, unless I run more than one virtual machine.

I have. But I have a few setups that will tax an entire 4GB real memory pool. At 1 and 1.5GB of actual RAM, I've grown my swap file a bit. BUT, I think this is dependent on how the memory pool is divided (such as VIRTUALADDRESSLIMIT settings) and used (such as what's using the HM pool).

Dear all,

backwards compatibility is one issue, adapting to new hardware and computing is another.  Backwards compatibility means, as Neil remarks, the possibility of running older applications. 

Actually, it is far worse than just the app layer. In 32 bit mode, a CPU can run 32 bit and 16 bit instructions. In 64 bit mode, it can run 64 bit and 32 bit instructions. Many of OS/2's lower level services still do 16 bit calls through an interesting but oddly very fast thunking layer (if people remember such implementations in earlier versions of Windows, they'll remember the big performance penalty). That ALSO includes numerous device interfaces.

Just out of curiousity - was there a 64bit version of eComStation very early on in the development cycle? I have a cd here labelled as such but it  won't boot (quite old at this stage)....
Thanks,
Conrad.

I would love an ISO of that to play with...


The only option that seems even somewhat viable is to take the "Windows 32 bit server/Certain 32 bit client route" - namely using the buggy PAE stuff on certain CPUs. But that's often problematic.

The other option would be to fully utilize OS/2's virtual addressing mechanisms, which allows multiple 4GB pages. There may be a decent performance penalty involved, I suspect. It would also require a bunch of code changes that may not be possible. Theoretically, per this article, OS/2's "new" kernel can address up to 64 TERAbytes of virtual memory. The questions that begs though, are, (1) is the article accurate and (2) how much additional work (that IBM either never started or never finished) would be required to take advantage of that capability?

[Dave Yeo]

The other option would be to fully utilize OS/2's virtual addressing mechanisms, which allows multiple 4GB pages. There may be a decent performance penalty involved, I suspect. It would also require a bunch of code changes that may not be possible. Theoretically, per this article, OS/2's "new" kernel can address up to 64 TERAbytes of virtual memory. The questions that begs though, are, (1) is the article accurate and (2) how much additional work (that IBM either never started or never finished) would be required to take advantage of that capability?

While in theory the i386 and kernel can address 64 TB of virtual memory, that would be something like 16 billion processes using the full 4GB address space. You have to subtract the kernel address space as well as the PCI space reducing it to 48 TBs. The overhead of keeping track of that would be huge (tables of tables of tables) and the system is still limited to (usually) 3.5 GB real ram and I've discovered that if swapper.dat grows over 2 GB the system crashes with a swap file error so theory and practice are far apart. Using PAE might be able to extend that by quite a bit but everything has to written with PAE in mind including the kernel and device drivers (using a device driver to access PAE might be a workaround). That is the problem with Windows and PAE, one badly behaved driver and the whole system becomes unstable.
I'm not an expert and could be wrong about the above but it seems to me we're stuck in 32 bit land with it being a huge hassle to use the 3.5 GB of address limit and to have 2 programs using 3.5 GB is probably not doable.

[RobertMauro]

The other option would be to fully utilize OS/2's virtual addressing mechanisms, which allows multiple 4GB pages. There may be a decent performance penalty involved, I suspect. It would also require a bunch of code changes that may not be possible. Theoretically, per this article, OS/2's "new" kernel can address up to 64 TERAbytes of virtual memory. The questions that begs though, are, (1) is the article accurate and (2) how much additional work (that IBM either never started or never finished) would be required to take advantage of that capability?

While in theory the i386 and kernel can address 64 TB of virtual memory, that would be something like 16 billion processes using the full 4GB address space. You have to subtract the kernel address space as well as the PCI space reducing it to 48 TBs. The overhead of keeping track of that would be huge (tables of tables of tables) and the system is still limited to (usually) 3.5 GB real ram and I've discovered that if swapper.dat grows over 2 GB the system crashes with a swap file error so theory and practice are far apart. Using PAE might be able to extend that by quite a bit but everything has to written with PAE in mind including the kernel and device drivers (using a device driver to access PAE might be a workaround). That is the problem with Windows and PAE, one badly behaved driver and the whole system becomes unstable.
I'm not an expert and could be wrong about the above but it seems to me we're stuck in 32 bit land with it being a huge hassle to use the 3.5 GB of address limit and to have 2 programs using 3.5 GB is probably not doable.

Hi Dave,

I think there may be a few other things to it than that:

Swapper:
- The swapper limitations are due to the way it is created, written and maintained - it is not using any disk I/O functions that support greater than the original 2GB file size - hence, the system will crash as soon as it tries to allocate above the 2GB barrier. Some experimentation and tracing by others after moving it to a JFS partition helped diagnose this. The particular parts of the OS responsible don't use the new functions.

Memory pages and such:
Page allocations ("16 billion processes") don't need to be done in such a fashion. Keep in mind that the overhead to do such for one 4GB space is already allocated within that space. The only thing required would be a pointer to that address space, which (within that address space) would contain pointers for that space. In total, that's 16,000 pointers.

And of course, that space, as you noted, gets decreased by bus mapping - but it ALSO gets decreased by "fixed" memory mapping, such as the OS area and parts of the shared mapping. In total, all that really needs to happen is the "HMA" could be paged, each "memory set" maintaining its own mappings within its memory area. That allows 4GB pages to hold the HMA and its own pointers.

Other issues:
This would require a number of other changes, to allow seeing and virtually utilizing the 64TB address space.

As for the swap file, I disable it on all systems with 4GB - it's just a waste of disk space and allocation time.