Author Topic: OS2 & eCS kernel  (Read 22316 times)

demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #30 on: 2011.04.25, 03:41:25 »
Quote
* Creating compatibility arenas as needed for certain apps
Either as expanded or separate arenas, or arenas within the high memory arena, or as virtual arenas that can be allocated/loaded/activated as needed when a context switch is done to "activate" the code using that arena (in a similar fashion to how the kernel currently does it when switching between processes - but taking into account the larger amount of memory, instead of limiting such things to the current arena sizes).

I think this can be solved by the subsystem DLLs.

Quote
* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.

Mainly, I don't think this will be much of an issue if the apps are given much more than just 512MB of memory. Even on a 32-bit kernel, I think that a 2GB/2GB, 3GB/1GB, or even 3.5GB/.5GB app/kernel split would be much better. I think that the arena idea came from the fact that compatibility with DOS programs & DLLs and resource sharing between 16- & 32-bit codebases were goals. I really don't see a reason to continue the arena paradigm, but if someone else has some enlightenment for me in this area, I'd love to hear it.

Quote
* Revising/rewriting address allocation for bus/bus devices/APIC mappings
This one is pretty self explanatory - needs to work in a 64 bit memory implementation, thus, will probably need to be moved, as it should no longer be "mapped downwards" from the top of the 4GB 32 bit mapping space (otherwise, there will be a nice memory hole in the 64 bit memory space and some nice kludges needed to address that).

Keep in mind that the addresses for certain devices & buses are defined by certain specs. For instance, PCI configuration spaces are defined by the PCI specs. The addresses of device registers are usually stored in the BAR registers of that device. In many aspects, our hands are tied when it comes to certain mappings. However, you're correct, this might be cover-able with mappings. We won't really know what's possible until the code is being written.

Quote
It also needs to take into account an increased mapping range -  it needs to be expanded to something more reasonable, or needs to be open ended (due to the far larger available addressing space). If I understand this situation properly, currently, eCS and Warp users are already running into problems with this, though often they don't understand why... problems such as 512MB video cards registering only 128MB in OS/2 due to frame buffer mappings in the system arena - as well as the hole in the actual addressable space created by various motherboards to allocate the APIC mappings in the top 512MB.

It seems, to me, that all of this really is problem of the kernel not being able to properly plug onboard device memory into it's overall virtual memory map. I think that this can be addressed appropriately.

Now, with all of that out of the way, I'm sure that it's obvious that a 32-bit kernel is a goal that could be reached far sooner than a 64-bit one. A 64-bit one is a great goal to have, but it's something that I would consider not reachable until 5-10 years from now. Especially when you consider that we'd might as well move to an EFI boot system for a 64-bit system, as it really would be a waste of time implementing a system that boots from bios for a 64-bit OS/2 replacement that probably wouldn't be ready until far after most bios based motherboards are sitting at the dump. Also, remember that a 64-bit kernel implies the imminent development 64-bit replacements for the rest of the OS.

Your thoughts, please.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: OS2 & eCS kernel
« Reply #31 on: 2011.04.25, 05:25:01 »


Quote
* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.

Mainly, I don't think this will be much of an issue if the apps are given much more than just 512MB of memory. Even on a 32-bit kernel, I think that a 2GB/2GB, 3GB/1GB, or even 3.5GB/.5GB app/kernel split would be much better. I think that the arena idea came from the fact that compatibility with DOS programs & DLLs and resource sharing between 16- & 32-bit codebases were goals. I really don't see a reason to continue the arena paradigm, but if someone else has some enlightenment for me in this area, I'd love to hear it.

I am NOT sure what mechanisms are used, but I do know REXX does various resource sharing. That would be VERY dangerous to break,  because of (a) how the OS uses REXX for certain things, (b) how various WPS add-ons use REXX and (c) because of numerous apps with REXX hooks or that utilize REXX code.

That aside, with a 64 bit memory range, using the 32 bit limitations for anything but compatibility would kinda defeat one of the purposes. As a for instance, the JFS and HPFS386 caches are limited by the sizes of those arenas. That's highly detrimental on things like database and web servers - on other operating systems, I can drop much larger amounts of memory to those daemons. As a for instance, considering the size of the Star Trek Phase 2 forum (and it's amount of traffic), it would be very nice if I could use higher memory amounts for stuff like the temp tables (that get created on both complex requests or on searches). Sadly, the amount of memory I can allocate is limited due to the arena size limitations created by the 32 bit addressable space. Especially since memory is also being allocated to disk caches, (database) table caches, thread management (in my case, just shy of 2000) and so on.
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: OS2 & eCS kernel
« Reply #32 on: 2011.04.25, 05:30:21 »
Quote
One big thing I've realized, though it's not mentioned much, is that consideration must be given for the large companies with big OS/2 and/or growing eCS installations - many of which have been running the same custom software since time immemorial. That's where the issues with HPFS386 and 16 bit (or hybrid 32/16 bit) apps comes in - and the importance of dealing with them - not to mention who knows how many other apps there may be out there that may be hybrids.

True, however, I'm sure that these large companies have also ran into these same problems that are listed on this thread, so I'd imagine that they'd be in the market for something more modern (& still compatible with their current OS/2 investments).

Which brings us back to an updated eCS... there's a certain large company dealing with such things right now, and the solution is to port the smaller generic apps to Linux, and keep the OS/2 investments in the others.

Quote
* Re-implementing a suitable swapping mechanism
With the advent of a 64 bit kernel - and thus more memory being accessible, numerous other Linux ports or native apps can become available that will be able to take advantage of the expanded memory space.

Swapping's something that's not always easy to achieve depending on the role that's being filled by the OS. Desktop OSs usually have different requirements than server OSs. This implies that all policies effecting this mechanism need to be dynamically changeable.

Hence the problem. Server OS or desktop OS; Warp does a far better job at this than Windows - whatever mechanism it uses.

Quote
While I have very very little experience with how Linux does this, Windows is absolutely pathetic at swapping - often swapping in use code to disk while plenty of memory is available, while OS/2 is very very decent at swapping the right code to disk, and only swapping active code/data to disk in the event of physical memory exhaustion.

Sounds like there needs to be a better aging policy implementation.

Got me... I wouldn't want to touch the Windows code that handles it with a ten foot pole. As it is, they've written workarounds and kludges to try to alleviate those issues, instead of trying to fix them.




CONTINUED IN NEXT POST

|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #33 on: 2011.04.25, 05:50:34 »


Quote
* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.

Mainly, I don't think this will be much of an issue if the apps are given much more than just 512MB of memory. Even on a 32-bit kernel, I think that a 2GB/2GB, 3GB/1GB, or even 3.5GB/.5GB app/kernel split would be much better. I think that the arena idea came from the fact that compatibility with DOS programs & DLLs and resource sharing between 16- & 32-bit codebases were goals. I really don't see a reason to continue the arena paradigm, but if someone else has some enlightenment for me in this area, I'd love to hear it.

I am NOT sure what mechanisms are used, but I do know REXX does various resource sharing. That would be VERY dangerous to break,  because of (a) how the OS uses REXX for certain things, (b) how various WPS add-ons use REXX and (c) because of numerous apps with REXX hooks or that utilize REXX code.

That aside, with a 64 bit memory range, using the 32 bit limitations for anything but compatibility would kinda defeat one of the purposes. As a for instance, the JFS and HPFS386 caches are limited by the sizes of those arenas. That's highly detrimental on things like database and web servers - on other operating systems, I can drop much larger amounts of memory to those daemons. As a for instance, considering the size of the Star Trek Phase 2 forum (and it's amount of traffic), it would be very nice if I could use higher memory amounts for stuff like the temp tables (that get created on both complex requests or on searches). Sadly, the amount of memory I can allocate is limited due to the arena size limitations created by the 32 bit addressable space. Especially since memory is also being allocated to disk caches, (database) table caches, thread management (in my case, just shy of 2000) and so on.

I'm sure that REXX could stand to me ported & extended to 64-bit, since we wouldn't want to confine it to living in a 32-bit compatibility box.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: OS2 & eCS kernel
« Reply #34 on: 2011.04.25, 05:53:23 »

I'm sure that REXX could stand to me ported & extended to 64-bit, since we wouldn't want to confine it to living in a 32-bit compatibility box.

That, especially considering it's way of assigning memory/variable objects, would be wonderful. Not to mention the benefits of OREXX being compiled with a real compiler that doesn't impact performance or cause massive memory leaks.
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #35 on: 2011.04.25, 06:18:52 »
I guess that means I need to procure an AMD64 box. I think I wanna persue this, whether I'm accepted as the claimer of this bounty or not.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: OS2 & eCS kernel
« Reply #36 on: 2011.04.25, 06:50:35 »

I'm sure that REXX could stand to me ported & extended to 64-bit, since we wouldn't want to confine it to living in a 32-bit compatibility box.

That, especially considering it's way of assigning memory/variable objects, would be wonderful. Not to mention the benefits of OREXX being compiled with a real compiler that doesn't impact performance or cause massive memory leaks.

Ooops, side note on this. Though REXX has been released from IBM's clutches, it does not include the OS/2 specific code or APIs, and will also lack various OS hooks.
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #37 on: 2011.04.25, 08:02:13 »
Ooops, side note on this. Though REXX has been released from IBM's clutches, it does not include the OS/2 specific code or APIs, and will also lack various OS hooks.

I'm sure that new bits can be written to weave REXX into the OS/2 replacement. Besides, there also has to be a new presentation manager & workplace shell. So, these parts would've had to have been rewritten anyway.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #38 on: 2011.04.25, 18:59:20 »
Is there anything else that needs to be addressed? Is it fair to work on a 32-bit kernel, with an eye to the future for a 64-bit one? Afterall, OW isn't ready for 64-bit targets yet & I wouldn't want to base our future on the GCC toolchain.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #39 on: 2011.05.07, 23:49:01 »
Is there anything else that needs to be discussed before I start working on a rough write up?
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: OS2 & eCS kernel
« Reply #40 on: 2011.05.08, 00:42:43 »
Is there anything else that needs to be discussed before I start working on a rough write up?

I'd suggest talking to Roderick Klein to see if he can give you some direction, ideas or such. rwklein on the forums.
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #41 on: 2011.05.11, 17:15:00 »
Is there anything else that needs to be discussed before I start working on a rough write up?

I'd suggest talking to Roderick Klein to see if he can give you some direction, ideas or such. rwklein on the forums.

I'm in constant contact with him. Just trying to get as much info as possible before development actually starts. I'm trying to find a middle ground between what's wanted, what's possible, & what's impractical.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!

miturbide

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1154
    • View Profile
    • OS2World
Re: OS2 & eCS kernel
« Reply #42 on: 2011.05.11, 21:23:09 »
If I switch to survival mode the goal should be:
- A complete OS2KRNL file replacement that works exactly as the current kernel at the Warp 4.52. Under a OSI open source license that will open the doors to keep improving/bug fixing the kernel. No third party copyrights, no strange patents that will limit us in the future. 64bits, new architecture, should not be a must. Something that you can change the OS2KRNL file and system keeps running like nothing happened. But that is only my way to see it.  (since I'm complete dumb on how OS/2 internal works)

Sure, sound easy, but as it is not an easy goal to reach.

Like I told before, from the bounty perspective, if Demetrius want to work on this kind of project, you can offer the goal that can be reached and we can always talk to the sponsors to re-define the bounty.
Demetrius, even if you are not interested on working on this bounty I appreciate your analysis on this forum. I think your discussion is very useful for anybody interested on this topic.

I think the goals of this bounty should be well defined, but if we put a to extensive "santa's list" on it, will scare any developer to work on it against the money we can raise.
Martín Itúrbide
OS2World.com NewsMaster
Open Source Advocate

Skype - martiniturbide
Google Talk - martiniturbide@gmail.com

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: OS2 & eCS kernel
« Reply #43 on: 2011.05.11, 22:14:09 »
Martin,

Love your idea for it. It is a great starting point. Alas, OS2KRNL replacement will also probably need an OS2LDR replacement. Demetrious has his work cut out for him.

Another interesting thing, IIRC, OS2KRNL is currently part assembly and part C. Not that it's probably very relevant.

I'm also wondering how helpful the new/upcoming AMD BIOS in CPU stuff may be to this effort. Not so much on the Intel side, of course...
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


demetrioussharpe

  • Full Member
  • ***
  • Posts: 206
    • View Profile
Re: OS2 & eCS kernel
« Reply #44 on: 2011.05.11, 22:48:28 »
If I switch to survival mode the goal should be:
- A complete OS2KRNL file replacement that works exactly as the current kernel at the Warp 4.52. Under a OSI open source license that will open the doors to keep improving/bug fixing the kernel. No third party copyrights, no strange patents that will limit us in the future. 64bits, new architecture, should not be a must. Something that you can change the OS2KRNL file and system keeps running like nothing happened. But that is only my way to see it.  (since I'm complete dumb on how OS/2 internal works)

Sure, sound easy, but as it is not an easy goal to reach.

Like I told before, from the bounty perspective, if Demetrius want to work on this kind of project, you can offer the goal that can be reached and we can always talk to the sponsors to re-define the bounty.
Demetrius, even if you are not interested on working on this bounty I appreciate your analysis on this forum. I think your discussion is very useful for anybody interested on this topic.

I think the goals of this bounty should be well defined, but if we put a to extensive "santa's list" on it, will scare any developer to work on it against the money we can raise.

Ok, in the coming days, I'll write up a rough spec sheet to cover as many of the bases as possible & post them. If that's acceptable, then I'll work towards that spec sheet in an attempt to claim the bounty afterwards. If it's not suitable for the bounty, then I'll most likely still work towards that goal, however, I'll have to look at it from a business prespective & possibly turn it into a product. I may attempt to write it from scratch, but if there's something that's close enough, then I might use that as a starting point. Keep in mind, features that aren't feasible right now could be revisited without the necessity of a new bounty.
The difference between what COULD be achieved & what IS achieved
is directly relational to what you COULD be doing & what you ARE doing!