Supporting thread for the OS2 & eCS kernel (http://www.os2world.com/content/view/346/71/) bounty. The original and old bounty thread can be found here (http://www.os2world.com/cgi-bin/ultraboard/UltraBoard.cgi?action=Read&BID=83&TID=28&SID=).
Are there any updated information available on the Bounty that was posted for a "New" OS/2 & eCS Kernel?
Thanks.
SAB.
SAB,
This bounty post
OS2 & eCS kernel (http://www.os2world.com/content/view/346/71/) describes the general goal of the bounty.
This thread is to discuss
specifically the less general goals that should be part of the successful completion of the bounty (ie: not drivers, not PM, not WPS, etc - except as related to how they interact with the kernel).
Thus, based off the general goals in the bounty, here is where the discussion should take place as to what needs to be satisfied to complete/satisfy the bounty.
So, for instance (to start off this thread):
- Do we want a platform independent setup, similar in that aspect to the Mach kernel? (or is our goal currently Intel only?)
- How do we wish to handle 32bit compatibility, and should we be handling that in the kernel? Should we have a virtualized 32 bit API? Or should we use something that remaps the APIs in a similar fashion that Odin does?
Robert
I am contributing to this bounty in partnership with Craig Colby.
I hope kernel authors will claim this bounty when a suitable kernel is ready.
I envision a kernel able to access modern computer features that can still offer a 32-bit OS/2 environment, but with specific extensions for 64-bit and beyond.
Does this kernel have to be written from scratch or can this be a kernel that's been modified to replace the OS/2 kernel?
Hi demetrioussharpe
What do you mean by "modified" ?
According to the bounty rules it has to be something that can be legally distributed and that can have an Open Source approved license. The bounty sponsors can also give some conditions on this forum.
Martin
Well, there're so many opensource kernels out there. In theory, any kernel would work, as long as you have a booting system, the kernel, & doscall(1).dll; all done in LX format. I have an associate who was trying to modify FreeBSD into an OS/2 replacement. Later, he changed the initial base system to DragonflyBSD. I think that he eventually gave up. It's very hard to find some of the necessary info to complete such a task. But, with a lot of effort, this could work. I'd try to tackle this myself, but I'm busy working on GL/2.
There's something called osfree at
http://www.osfree.org
who are trying to create a new os/2 clone using the L4 microkernel if you're interested. There's lots of activity by OS/2 standards :) . The last SVN repository change was a few days ago on the 19th of Nov. They are at version 0.05 and trying to get the file system API sorted out.
If I recall correctly, their target is OS/2 PPC - the product that never made it to the masses. I'm more concerned about Merlin & Aurora. I think one of the BSD kernels would be a perfect fit.
what do you mean by "their target is OS/2 PPC". They are actually working on making a OS/2 clone that runs on x86. Check the project. It is worthy to keep an eye on it. OSfreeldr had made nice progress booting from USB and CD-Rom.
Yeh, I keep a constant watch on that project. They're using the software architecture of OS/2 PowerPC. It's a good design, but I'm not ready to see the Merlin/Aurora architecture go away yet. It's far too easy to get microkernel OS's wrong & too easy to get monolithic kernels right. IBM threw money & developers at a fundamentally sound architecture, but in the end, it just didn't work out. I'm not downing osfree's work, I think that they should keep going. But, even if they succeed, there's still the allure of being able to replace a kernel & a hand full of dll files, verses having to replace your whole OS/2 setup. So, regardless of other projects, is it acceptable for someone to plug in a replacement kernel & a few replacement dll's to close this bounty or at least to consider it a success?
Hi demetrioussharpe,
First I am not a programmer but I've read enough to be dangerous :) . It sounds impossible what you are trying to do: replacing the OS/2 kernel with something else without a complete rewrite of the other subsystems. For example, Reactos. And Reactos has been in alpha for so many years and is not even complete yet!
Perhaps I am misinterpreting what you are saying. A longer explanation might be useful for lay people like us.
Quote from: cytan on 2010.11.24, 04:06:20
Hi demetrioussharpe,
First I am not a programmer but I've read enough to be dangerous :) . It sounds impossible what you are trying to do: replacing the OS/2 kernel with something else without a complete rewrite of the other subsystems. For example, Reactos. And Reactos has been in alpha for so many years and is not even complete yet!
Perhaps I am misinterpreting what you are saying. A longer explanation might be useful for lay people like us.
Ok, here's the basics of it:
All programs, whether they're apps or system services, communicate with the kernel through the support DLLs. For OS/2, all system calls will eventually find their way to DLLs like doscall.dll or doscall1.dll. These kinds of DLL are the entrypoint. They serve to isolate the app from the kernel. As long as these DLLs behave as normal, the kernel can be changed & none of the apps will notice a difference. So, you change the kernel & you create the support DLLs for the apps to communicate with the kernel through.
Quote from: demetrioussharpe on 2010.11.24, 06:17:30
Quote from: cytan on 2010.11.24, 04:06:20
Hi demetrioussharpe,
First I am not a programmer but I've read enough to be dangerous :) . It sounds impossible what you are trying to do: replacing the OS/2 kernel with something else without a complete rewrite of the other subsystems. For example, Reactos. And Reactos has been in alpha for so many years and is not even complete yet!
Perhaps I am misinterpreting what you are saying. A longer explanation might be useful for lay people like us.
Ok, here's the basics of it:
All programs, whether they're apps or system services, communicate with the kernel through the support DLLs. For OS/2, all system calls will eventually find their way to DLLs like doscall.dll or doscall1.dll. These kinds of DLL are the entrypoint. They serve to isolate the app from the kernel. As long as these DLLs behave as normal, the kernel can be changed & none of the apps will notice a difference. So, you change the kernel & you create the support DLLs for the apps to communicate with the kernel through.
Demetrious, I'm glad you are around and helping to develop new things for this great OS. You seem to know your stuff and I'm in no position to question your intentions. Beware of comments and opinions other people post here.
I still remember a good programmer that created a great multimedia extension and after many critics and misinformed remarks he decided to leave, never to return to this day. His classes are still one of the best examples of why OS/2 is still a superior system.
Anyway, I hope you continue your work on OpenGL and reach your goal. We need all the help we can have I personally I don't think we are in a position to say this is important and this is not. If we want something specific to be done then we should put money into it, otherwise I see no point in the argument
Thanks and good coding
Quote from: lpino on 2010.11.24, 14:02:15
Quote from: demetrioussharpe on 2010.11.24, 06:17:30
Quote from: cytan on 2010.11.24, 04:06:20
Hi demetrioussharpe,
First I am not a programmer but I've read enough to be dangerous :) . It sounds impossible what you are trying to do: replacing the OS/2 kernel with something else without a complete rewrite of the other subsystems. For example, Reactos. And Reactos has been in alpha for so many years and is not even complete yet!
Perhaps I am misinterpreting what you are saying. A longer explanation might be useful for lay people like us.
Ok, here's the basics of it:
All programs, whether they're apps or system services, communicate with the kernel through the support DLLs. For OS/2, all system calls will eventually find their way to DLLs like doscall.dll or doscall1.dll. These kinds of DLL are the entrypoint. They serve to isolate the app from the kernel. As long as these DLLs behave as normal, the kernel can be changed & none of the apps will notice a difference. So, you change the kernel & you create the support DLLs for the apps to communicate with the kernel through.
Demetrious, I'm glad you are around and helping to develop new things for this great OS. You seem to know your stuff and I'm in no position to question your intentions. Beware of comments and opinions other people post here.
I still remember a good programmer that created a great multimedia extension and after many critics and misinformed remarks he decided to leave, never to return to this day. His classes are still one of the best examples of why OS/2 is still a superior system.
Anyway, I hope you continue your work on OpenGL and reach your goal. We need all the help we can have I personally I don't think we are in a position to say this is important and this is not. If we want something specific to be done then we should put money into it, otherwise I see no point in the argument
Thanks and good coding
Hi guys,
I hope I don't come across as a doubting thomas or overly critical about what demetrious is proposing. My intent was just to get a clarification of what he had in mind, nothing more.
No worries. After almost 14yrs in the Army, my skin's a bit tougher than most. It's also the reason I tend to jump into things & try to just get them done. I'm hoping to help spark a new trend of OS/2 innovation.
Quote from: RobertM on 2008.04.29, 22:01:44
SAB,
This bounty post OS2 & eCS kernel (http://www.os2world.com/content/view/346/71/) describes the general goal of the bounty.
This thread is to discuss specifically the less general goals that should be part of the successful completion of the bounty (ie: not drivers, not PM, not WPS, etc - except as related to how they interact with the kernel).
Thus, based off the general goals in the bounty, here is where the discussion should take place as to what needs to be satisfied to complete/satisfy the bounty.
So, for instance (to start off this thread):
- Do we want a platform independent setup, similar in that aspect to the Mach kernel? (or is our goal currently Intel only?)
- How do we wish to handle 32bit compatibility, and should we be handling that in the kernel? Should we have a virtualized 32 bit API? Or should we use something that remaps the APIs in a similar fashion that Odin does?
Robert
Hello all,
I'm back again & I'm wondering if these were the only goals that were outlined for a new kernel or if there were more goals listed. What all is necessary to claim this bounty? Does it have to be able to work with the software from the OS/2 or eCS install CDs or can it be a complete rewrite? Can it be another OS that's been repurposed into an OS/2 clone? What exactly is expected here?
As one of the contributors, I'd like to see this happen. Let me throw out some ideas..
1. Fairly OS/2 compatible. It should support most of the old device drivers. You can toss out support for IBM PS/2, 386/486 processors, maybe more, but it should be able to run most current OS/2 device drivers. It should either work with or replace DOSCALL1.DLL which is the OS/2 Control Program API.
2. Parsing and using Config.Sys is not a requirement, but many programs expect it.
3. Open source would be a big plus. And using someone else's kernel is no problem as long as it can run OS/2 device drivers and programs. The idea is to replace the current closed-source IBM kernel with something that can be fixed or enhanced.
4. It should go beyond what the current hardware limitations of OS/2, like the 2 TB limit on devices, the 4 GB limit on RAM, and 16-bit limits on system queues, semaphores, pipes and so forth. It should do this in a way that allows a well-written OS/2 program to take full advantage of new hardware.
5. It should support a robust trap recovery and debugging interface. It should make it possible to have a hard kill.
Well, I am no longer sure what were goals, or future plans or wants. Thus, some things on this list may overlap or solely fit into one of those categories. This list is probably not as extensive as it could be, based on the feedback and posts throughout this forum and elsewhere:
* 64bit support
This of course is highly problematic, because the kernel itself has not just 16bit entry points, but also contains a lot of 16bit code. In addition to that, most (or all) the non KEE drivers are 16bit. Since, in 64 bit mode, today's CPUs only support 64 bit and 32 bit instructions, the 16 bit code would need to be rewritten - and all 16 bit code that calls it would either need to be rewritten or run through a virtualization module, or a module that could do "on the fly conversion" to 32 bit codes (ie: perhaps in a similar fashion to how Odin detects the executable type and "translates" the Win32 calls into their OS/2 counterparts, while dealing with any weird mismatches in the capabilities between the calls/APIs).
* Maintaining a suitable threading model
This was deemed, from a technical standpoint, as well as from an experience (with Linux) standpoint as an aspect that would make simply dropping in a Linux kernel and a personality module an unweildly solution, since Linux does not support the type of multithreading that OS/2 does.
* Dealing with the few things that DO directly access the kernel
I don't think there are many - but the most notable one is one that is still used by a lot of big businesses, namely HPFS386 which (if memory serves) is (a) a Ring 0 driver, (b) one that has direct access to the hardware, and (c) one that has direct access to the kernel. IIRC, it's features were somewhat extensively used for certain Lotus Domino, Lotus Domino GoWebserver and DB/2 releases to allow direct device to device data transfer and direct hardware access/transfer.
* Full support for the 16 bit and 32 bit APIs in the kernel
Which brings us back to virtualizing and/or replacing them. It also brings us to what to do with the thunking layer.
* A method of dealing with the 16 bit device drivers (again, something we're stuck with as a problem for a 64 bit kernel)
There aren't enough 32 bit drivers out there - and from what I understand, there are still various 16 bit callbacks that even they use, simply due to OS/2's driver structure and kernel structure.
* Possible migration of pseudo-64 bit code to true 64 bit code
Such as the 64 bit data structures used by JFS and a few other replacement subsystems written for MCP/ACP. This of course is not really a priority, since they work as they are.
* Mach style kernel
At least at some point, that way as additional CPU design changes are made, the kernel can be easily swapped with one that supports the new architecture (if only the architecture specific kernels were as easy to make as dropping one in).
* Increasing the memory management and thread management capabilities
Regardless of a 64 bit kernel, increasing the thread management capabilities to something more akin to today's newer hardware is something various of us need - for instance, to allow high availability servers without exhausting the thread pool. Similar applies to processes. Sadly, these figures are hardcoded, as are their data structures and the underlying mechanisms that support them. So, while newer hardware has sufficient memory to allocate the management data structures for a lot more threads and processes, and CPUs are more than fast enough to handle the expanded thread set/data structures, this would require a lot of rewriting of the thread and process schedulers and their data structures.
This would possibly also require tweaks or changes to Aurora's/Merlin's new memory management scheme to ensure that memory pool exhaustion did not occur when dealing with the larger data structure set for threads and processes. Currently though, on a system using memory from that arena for other purposes (disk cache, etc), roughly 2,000 threads (give or take) is the reliable limit some of us have been running into - even though the kernel is designed to handle 4095 threads.
There were also suggestions about changing the memory management architecture to handle PAE mode - but that seems buggy and a kludge under any implementation.
There was also discussion about OS/2's already existing ability to "access" more than 4GB by paging and virtualization - I think there's an article someplace on EDM/2 that mentions it. It's a feature not used, and apparently barely understood by those who mention it.
* Dealing with the "kernel helpers"
Which in reality, act as extensions to the kernel itself (such as DOSCALL1, which you already mentioned, and the various other system DLLs - oh, and OS2LDR, which is quite a bit more than just a boot loader, and continues to run in conjunction with the kernel to provide kernel services). Of course, for a 64 bit kernel, this means once again dealing with the 16 bit code.
* Expanding the kernel for other OS's (namely Linux)
Via either pluggable APIs, personality modules, abstraction layers, etc. This probably doesn't require any additional work on the kernel though - just as Odin doesn't require much in that area.
* Replacing/rewriting OS2LDR
...to be able to work in conjunction with the new kernel as well as with larger hard drives, without the need for patching or kludges.
* Some method of support for all 16 bit calls
...since they cannot be run natively on a CPU in 64 bit mode. A virtualization layer? Something that remaps the calls a'la Odin to 32 bit calls? Whatever it is, it's needed for the variety of businesses that run apps and services that are 16 bit or 32/16 bit hybrid.
There's probably a bunch more... and, as I said, a few of these were probably not intended as part of the original bounty.
One big thing I've realized, though it's not mentioned much, is that consideration must be given for the large companies with big OS/2 and/or growing eCS installations - many of which have been running the same custom software since time immemorial. That's where the issues with HPFS386 and 16 bit (or hybrid 32/16 bit) apps comes in - and the importance of dealing with them - not to mention who knows how many other apps there may be out there that may be hybrids.
Ah... Aurora kernel can handle 64TB of virtualized address space, with 4GB allocated as protected mode memory (sans what's mapped to the various system buses, etc). Supposedly.
This was supposed to allow up to xGB of memory per process, with up to xGB being the active memory space being used at any given time. xGB = the virtual address limit memory limitation setting in the Aurora kernel (ie: up to 3GB).
http://www.os2voice.org/VNL/past_issues/VNL0708H/feature_3.html
Ooops, one more important one (at least for the 32 bit kernel). Cleaning up memory (ie: dealing with fragmentation that causes issues with memory allocation when there are no big enough contiguous blocks available).
All of this (and many of the other ones on my earlier list) only apply to a 64 bit kernel, as I guess you've probably already noted
* Re-implementing a suitable swapping mechanism
With the advent of a 64 bit kernel - and thus more memory being accessible, numerous other Linux ports or native apps can become available that will be able to take advantage of the expanded memory space.
While I have very very little experience with how Linux does this, Windows is absolutely pathetic at swapping - often swapping in use code to disk while plenty of memory is available, while OS/2 is very very decent at swapping the right code to disk, and only swapping active code/data to disk in the event of physical memory exhaustion.
(As one example) While memory is usually cheap, even in a 64 bit system, at say 16GB of RAM (after that, memory starts getting expensive per module for the bigger modules) it would be very easy to exhaust physical memory using something like a Blender rendering port.
* Creating compatibility arenas as needed for certain apps
Either as expanded or separate arenas, or arenas within the high memory arena, or as virtual arenas that can be allocated/loaded/activated as needed when a context switch is done to "activate" the code using that arena (in a similar fashion to how the kernel currently does it when switching between processes - but taking into account the larger amount of memory, instead of limiting such things to the current arena sizes).
* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.
* Revising/rewriting address allocation for bus/bus devices/APIC mappings
This one is pretty self explanatory - needs to work in a 64 bit memory implementation, thus, will probably need to be moved, as it should no longer be "mapped downwards" from the top of the 4GB 32 bit mapping space (otherwise, there will be a nice memory hole in the 64 bit memory space and some nice kludges needed to address that).
It also needs to take into account an increased mapping range - it needs to be expanded to something more reasonable, or needs to be open ended (due to the far larger available addressing space). If I understand this situation properly, currently, eCS and Warp users are already running into problems with this, though often they don't understand why... problems such as 512MB video cards registering only 128MB in OS/2 due to frame buffer mappings in the system arena - as well as the hole in the actual addressable space created by various motherboards to allocate the APIC mappings in the top 512MB.
Let's be clear, is this bounty for a 32-bit kernel or a 64-bit kernel? I'm asking because there's going to be a huge difference in development. However, both kernels would imply that a new device driver architecture, which isn't necessarily a bad thing.
Also, keep in mind that we'll loose OpenWatcom as a development environment for building a 64-bit kernel. The OW team hasn't produced the code for building 64-bit targets yet.
Hi Demetrius.
The description of the bounty is "Description: For future use of OS2 and eCS there is a need for a new 64-bit kernel and what ever the future technology might offer. "
But we are always open for discussion.
If a developer say offers to do something and will be according to the bounty rules (open source/ no copyright issues). We can always go back to the sponsors and discuss if we can adjust the requirements.
Long post, but this is important, so please be patient with me.First & foremost, I want it to be understood that I intend to work on this project regardless of it you guys award me the bounty or not. I know that this is a huge project. I also understand that I'm just one man, so this is something that'll be a long term process; not something that could be coded within 3-6 months.
Secondly, I have a preference for the OpenWatcom development platform. Though they aren't ready for compiling 64-bit targets, I'm sure that they'll get there sooner or later. Every other major platform has it's own staple development platform & I believe that the future of OS/2 & OW are intertwined.
Thirdly, be aware that a new kernel must bring new things with it. Some of these things we're ready for, others, we're not so prepared for. Undoubtedly, a new kernel would imply that there's a chance for a fresh look on device drivers & filesystem drivers. To be honest, there's not much to be gained from reusing the current OS/2 drivers & filesystems. The device driver interface is crippled & mentally retarded for today's purposes & the filesystem API isn't too much better. There are plenty of opportunity for things to be improved with new APIs for both of these subsystems; and to be honest, it really doesn't matter if we roll our own or if we 'borrow' the interfaces from other systems, just as long as we ditch the interfaces that we're currently using. Besides, a move to a 64-bit kernel would demand such a move anyway. Also, there's the issue of the host binary format. Mainly, there aren't many people who understand how to load LX binaries. And even after we jump that hurdle, there's still the matter of creating a new format for 64-bit OS/2 binaries. Since this wasn't done before IBM killed the commercial OS/2 product, we're on our own here.
Now, with that being said, I'll move on & attempt to address each of the points that have been touched on throughout the life of this forum topic. If I miss something, please don't think that it was on purpose. For many of these points, there may be a bit of overlap, so my replies to them may be similar or even the same; please bare with me! ;)
Here goes:
QuoteDo we want a platform independent setup, similar in that aspect to the Mach kernel? (or is our goal currently Intel only?)
To be honest, we have a better chance of survival if the kernel is structured to sit on top of a architecture dependent module. It doen't matter if it's called a HAL or an ARCH layer, just as long as all of the device specific code is contained in this layer & all of the platform independent code sits on top of it as a separate layer.
QuoteHow do we wish to handle 32bit compatibility, and should we be handling that in the kernel? Should we have a virtualized 32 bit API? Or should we use something that remaps the APIs in a similar fashion that Odin does?
The way I see it, the majority of 32-bit (& also 16-bit) support code can be contained within userspace emulation modules (the way dosbox does it). If we're talking about a 64-bit kernel, the we could use the approach that dosbox uses for a 32-bit layer & inside that 32-bit environment we could have a 16-bit emulation layer. Afterall, there really won't be much of a reason for 16-bit & 64-bit code to talk to each other. I'm really doubtful of the need for 32-bit & 16-bit code to talk to each other either, other than the fact that much of OS/2's kernel & driver systems were also 16-bit. If OS/2 had been fully 32-bit, there really wouldn't have been a real need for any 16/32-bit resource sharing or any of that thunking nonsense.
CONTINUED IN NEXT POST
QuoteFairly OS/2 compatible. It should support most of the old device drivers. You can toss out support for IBM PS/2, 386/486 processors, maybe more, but it should be able to run most current OS/2 device drivers. It should either work with or replace DOSCALL1.DLL which is the OS/2 Control Program API.
Again, this is a non-issue. New kernel, new drivers that require less jumping through hoops to write/port. With a 64-bit kernel, DOSCALL.DLL & DOSCALL1.DLL would end up being part of the 32-bit emulation layer, but nothing would really change for the current apps.
QuoteParsing and using Config.Sys is not a requirement, but many programs expect it.
This needs to be emulated by the use of a configuration pseudo-driver. Using the *nix philosophy that everything is a file, this pseudo-driver could be written to & read from like a regular file, but should be backed by a registry system. This would not be a perfect solution for current OS/2 apps, but I'd expect newer apps to use the registry API for access to configuration data. However, this is also the opportunity to bake multiuser support into the system from the ground up & thoroughly enforce security & protection policies from the start. This means that we don't make the Windows mistake that everyone's account defaults to having Administrator rights & privileges.
QuoteOpen source would be a big plus. And using someone else's kernel is no problem as long as it can run OS/2 device drivers and programs. The idea is to replace the current closed-source IBM kernel with something that can be fixed or enhanced.
As stated before, new kernel, new drivers. However, if my work becomes used for this bounty, then there's no reason not to open source it. Still, what use is an open sourced OS if there're precious few left to work on it. Even the handful of developers left haven't stepped up to work on this task other than the OSFree guys. I hope they're progressing nicely, but who really knows.
If my work is not acceptable for this bounty, then perhaps I'll end up making a product out of it. Regardless of the outcome, I see it as a win/win situation for everyone involved.
QuoteIt should go beyond what the current hardware limitations of OS/2, like the 2 TB limit on devices, the 4 GB limit on RAM, and 16-bit limits on system queues, semaphores, pipes and so forth. It should do this in a way that allows a well-written OS/2 program to take full advantage of new hardware.
Undoubtedly, this should be standard in a replacement kernel, regardless of whether it's 32- or 64-bits.
QuoteIt should support a robust trap recovery and debugging interface. It should make it possible to have a hard kill.
I agree. This is something that really should be built in from the very beginning to allow effective debugging while attempting to bring the kernel up.
Quote* 64bit support
This of course is highly problematic, because the kernel itself has not just 16bit entry points, but also contains a lot of 16bit code. In addition to that, most (or all) the non KEE drivers are 16bit. Since, in 64 bit mode, today's CPUs only support 64 bit and 32 bit instructions, the 16 bit code would need to be rewritten - and all 16 bit code that calls it would either need to be rewritten or run through a virtualization module, or a module that could do "on the fly conversion" to 32 bit codes (ie: perhaps in a similar fashion to how Odin detects the executable type and "translates" the Win32 calls into their OS/2 counterparts, while dealing with any weird mismatches in the capabilities between the calls/APIs).
This should not be much of an issue, considering that proper DLL replacements appropriately implement the expected APIs. To drive home the point: It really doesn't matter who built engine that is under the hood of your car, as long as it functions the way you expect it to. The DLLs should take care of making sure that all requests are transformed into the right size for the underlying kernel.
CONTINUED IN NEXT POST
Quote* Maintaining a suitable threading model
This was deemed, from a technical standpoint, as well as from an experience (with Linux) standpoint as an aspect that would make simply dropping in a Linux kernel and a personality module an unweildly solution, since Linux does not support the type of multithreading that OS/2 does.
To be honest, I've only seen one OS that had a wonderful threading model & that was the BeOS. With that being said, there are 2 ways forward from my point of view.
1). Use all available documentation about OS/2's threading model to recreate IBM's implementation as closely as possible.
2). Try to create something that's suitable & work to improve it.
Quote* Dealing with the few things that DO directly access the kernel
I don't think there are many - but the most notable one is one that is still used by a lot of big businesses, namely HPFS386 which (if memory serves) is (a) a Ring 0 driver, (b) one that has direct access to the hardware, and (c) one that has direct access to the kernel. IIRC, it's features were somewhat extensively used for certain Lotus Domino, Lotus Domino GoWebserver and DB/2 releases to allow direct device to device data transfer and direct hardware access/transfer.
I'm sure that this hurdle could be toppled with a robust VFS layer & an updated, reimplementation of HPFS386 that's built from the specs of the original.
Quote* Full support for the 16 bit and 32 bit APIs in the kernel
Which brings us back to virtualizing and/or replacing them. It also brings us to what to do with the thunking layer.
This really doesn't belong in the kernel. This belongs in userspace. There's no need for 16-bit APIs in the kernel when there's no 16-bit code in the kernel. If anything, the DLL that services 16-bit code should take care of converting it to the bit size of the underlying kernel.
Quote* A method of dealing with the 16 bit device drivers (again, something we're stuck with as a problem for a 64 bit kernel)
There aren't enough 32 bit drivers out there - and from what I understand, there are still various 16 bit callbacks that even they use, simply due to OS/2's driver structure and kernel structure.
New kernel, new drivers. No 16-bit code inside.
Quote* Possible migration of pseudo-64 bit code to true 64 bit code
Such as the 64 bit data structures used by JFS and a few other replacement subsystems written for MCP/ACP. This of course is not really a priority, since they work as they are.
JFS would need to be ported to the new kernel anyway, so there really wouldn't be much migration other than the basic porting work. A VFS layer should make this a bit easier.
Quote* Mach style kernel
At least at some point, that way as additional CPU design changes are made, the kernel can be easily swapped with one that supports the new architecture (if only the architecture specific kernels were as easy to make as dropping one in).
See my reply above about the HAL.
CONTINUED IN NEXT POST
Quote* Increasing the memory management and thread management capabilities
Regardless of a 64 bit kernel, increasing the thread management capabilities to something more akin to today's newer hardware is something various of us need - for instance, to allow high availability servers without exhausting the thread pool. Similar applies to processes. Sadly, these figures are hardcoded, as are their data structures and the underlying mechanisms that support them. So, while newer hardware has sufficient memory to allocate the management data structures for a lot more threads and processes, and CPUs are more than fast enough to handle the expanded thread set/data structures, this would require a lot of rewriting of the thread and process schedulers and their data structures.
I agree. These are the kinds of things that really should be dynamic. If OS/2 was still being developed today, I'm sure that these things would have been addressed already. However, this is a great time to address these issues.
QuoteThis would possibly also require tweaks or changes to Aurora's/Merlin's new memory management scheme to ensure that memory pool exhaustion did not occur when dealing with the larger data structure set for threads and processes. Currently though, on a system using memory from that arena for other purposes (disk cache, etc), roughly 2,000 threads (give or take) is the reliable limit some of us have been running into - even though the kernel is designed to handle 4095 threads.
Again, I agree.
QuoteThere were also suggestions about changing the memory management architecture to handle PAE mode - but that seems buggy and a kludge under any implementation.
This is the kind of thing that should be inside of the HAL. It should also be configurable, so that the PAE mode can be enabled or disabled based on the host architecture.
QuoteThere was also discussion about OS/2's already existing ability to "access" more than 4GB by paging and virtualization - I think there's an article someplace on EDM/2 that mentions it. It's a feature not used, and apparently barely understood by those who mention it.
This should be seemless & invisible, without needing to resort to tricks to get it to work correctly.
Quote* Dealing with the "kernel helpers"
Which in reality, act as extensions to the kernel itself (such as DOSCALL1, which you already mentioned, and the various other system DLLs - oh, and OS2LDR, which is quite a bit more than just a boot loader, and continues to run in conjunction with the kernel to provide kernel services). Of course, for a 64 bit kernel, this means once again dealing with the 16 bit code.
I agree with the kernel extensions, however, I don't agree with the way they're used in OS/2. I'm not really a huge fan of the kernel helpers, because they seem to imply portions of the code that were implemented based on a time schedule & rushed, rather than attempting to find an elegant solution. Yet & still, it's something that needs to be taken in account.
Quote* Expanding the kernel for other OS's (namely Linux)
Via either pluggable APIs, personality modules, abstraction layers, etc. This probably doesn't require any additional work on the kernel though - just as Odin doesn't require much in that area.
Solvable with subsystem DLLs.
Quote* Replacing/rewriting OS2LDR
...to be able to work in conjunction with the new kernel as well as with larger hard drives, without the need for patching or kludges.
I think that the current bootloader to kernel interface should be scrapped & replaced anyway. This makes room for something that's more maintainable & flexible for future advances.
CONTINUED IN NEXT POST
Quote* Some method of support for all 16 bit calls
...since they cannot be run natively on a CPU in 64 bit mode. A virtualization layer? Something that remaps the calls a'la Odin to 32 bit calls? Whatever it is, it's needed for the variety of businesses that run apps and services that are 16 bit or 32/16 bit hybrid.
Subsystem DLLs.
QuoteOne big thing I've realized, though it's not mentioned much, is that consideration must be given for the large companies with big OS/2 and/or growing eCS installations - many of which have been running the same custom software since time immemorial. That's where the issues with HPFS386 and 16 bit (or hybrid 32/16 bit) apps comes in - and the importance of dealing with them - not to mention who knows how many other apps there may be out there that may be hybrids.
True, however, I'm sure that these large companies have also ran into these same problems that are listed on this thread, so I'd imagine that they'd be in the market for something more modern (& still compatible with their current OS/2 investments).
QuoteCleaning up memory (ie: dealing with fragmentation that causes issues with memory allocation when there are no big enough contiguous blocks available).
This shouldn't be as much of an issue on a more modern kernel.
Quote* Re-implementing a suitable swapping mechanism
With the advent of a 64 bit kernel - and thus more memory being accessible, numerous other Linux ports or native apps can become available that will be able to take advantage of the expanded memory space.
Swapping's something that's not always easy to achieve depending on the role that's being filled by the OS. Desktop OSs usually have different requirements than server OSs. This implies that all policies effecting this mechanism need to be dynamically changeable.
QuoteWhile I have very very little experience with how Linux does this, Windows is absolutely pathetic at swapping - often swapping in use code to disk while plenty of memory is available, while OS/2 is very very decent at swapping the right code to disk, and only swapping active code/data to disk in the event of physical memory exhaustion.
Sounds like there needs to be a better aging policy implementation.
QuoteWhile memory is usually cheap, even in a 64 bit system, at say 16GB of RAM (after that, memory starts getting expensive per module for the bigger modules) it would be very easy to exhaust physical memory using something like a Blender rendering port.
I don't think this will be much of an issue for the replacement kernel.
CONTINUED IN NEXT POST
Quote* Creating compatibility arenas as needed for certain apps
Either as expanded or separate arenas, or arenas within the high memory arena, or as virtual arenas that can be allocated/loaded/activated as needed when a context switch is done to "activate" the code using that arena (in a similar fashion to how the kernel currently does it when switching between processes - but taking into account the larger amount of memory, instead of limiting such things to the current arena sizes).
I think this can be solved by the subsystem DLLs.
Quote* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.
Mainly, I don't think this will be much of an issue if the apps are given much more than just 512MB of memory. Even on a 32-bit kernel, I think that a 2GB/2GB, 3GB/1GB, or even 3.5GB/.5GB app/kernel split would be much better. I think that the arena idea came from the fact that compatibility with DOS programs & DLLs and resource sharing between 16- & 32-bit codebases were goals. I really don't see a reason to continue the arena paradigm, but if someone else has some enlightenment for me in this area, I'd love to hear it.
Quote* Revising/rewriting address allocation for bus/bus devices/APIC mappings
This one is pretty self explanatory - needs to work in a 64 bit memory implementation, thus, will probably need to be moved, as it should no longer be "mapped downwards" from the top of the 4GB 32 bit mapping space (otherwise, there will be a nice memory hole in the 64 bit memory space and some nice kludges needed to address that).
Keep in mind that the addresses for certain devices & buses are defined by certain specs. For instance, PCI configuration spaces are defined by the PCI specs. The addresses of device registers are usually stored in the BAR registers of that device. In many aspects, our hands are tied when it comes to certain mappings. However, you're correct, this might be cover-able with mappings. We won't really know what's possible until the code is being written.
QuoteIt also needs to take into account an increased mapping range - it needs to be expanded to something more reasonable, or needs to be open ended (due to the far larger available addressing space). If I understand this situation properly, currently, eCS and Warp users are already running into problems with this, though often they don't understand why... problems such as 512MB video cards registering only 128MB in OS/2 due to frame buffer mappings in the system arena - as well as the hole in the actual addressable space created by various motherboards to allocate the APIC mappings in the top 512MB.
It seems, to me, that all of this really is problem of the kernel not being able to properly plug onboard device memory into it's overall virtual memory map. I think that this can be addressed appropriately.
Now, with all of that out of the way, I'm sure that it's obvious that a 32-bit kernel is a goal that could be reached far sooner than a 64-bit one. A 64-bit one is a great goal to have, but it's something that I would consider not reachable until 5-10 years from now. Especially when you consider that we'd might as well move to an EFI boot system for a 64-bit system, as it really would be a waste of time implementing a system that boots from bios for a 64-bit OS/2 replacement that probably wouldn't be ready until far after most bios based motherboards are sitting at the dump. Also, remember that a 64-bit kernel implies the imminent development 64-bit replacements for the rest of the OS.
Your thoughts, please.
Quote from: demetrioussharpe on 2011.04.25, 04:41:25
Quote* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.
Mainly, I don't think this will be much of an issue if the apps are given much more than just 512MB of memory. Even on a 32-bit kernel, I think that a 2GB/2GB, 3GB/1GB, or even 3.5GB/.5GB app/kernel split would be much better. I think that the arena idea came from the fact that compatibility with DOS programs & DLLs and resource sharing between 16- & 32-bit codebases were goals. I really don't see a reason to continue the arena paradigm, but if someone else has some enlightenment for me in this area, I'd love to hear it.
I am NOT sure what mechanisms are used, but I do know REXX does various resource sharing. That would be VERY dangerous to break, because of (a) how the OS uses REXX for certain things, (b) how various WPS add-ons use REXX and (c) because of numerous apps with REXX hooks or that utilize REXX code.
That aside, with a 64 bit memory range, using the 32 bit limitations for anything but compatibility would kinda defeat one of the purposes. As a for instance, the JFS and HPFS386 caches are limited by the sizes of those arenas. That's highly detrimental on things like database and web servers - on other operating systems, I can drop much larger amounts of memory to those daemons. As a for instance, considering the size of the Star Trek Phase 2 forum (and it's amount of traffic), it would be very nice if I could use higher memory amounts for stuff like the temp tables (that get created on both complex requests or on searches). Sadly, the amount of memory I can allocate is limited due to the arena size limitations created by the 32 bit addressable space. Especially since memory is also being allocated to disk caches, (database) table caches, thread management (in my case, just shy of 2000) and so on.
Quote from: demetrioussharpe on 2011.04.25, 04:40:42
QuoteOne big thing I've realized, though it's not mentioned much, is that consideration must be given for the large companies with big OS/2 and/or growing eCS installations - many of which have been running the same custom software since time immemorial. That's where the issues with HPFS386 and 16 bit (or hybrid 32/16 bit) apps comes in - and the importance of dealing with them - not to mention who knows how many other apps there may be out there that may be hybrids.
True, however, I'm sure that these large companies have also ran into these same problems that are listed on this thread, so I'd imagine that they'd be in the market for something more modern (& still compatible with their current OS/2 investments).
Which brings us back to an updated eCS... there's a certain large company dealing with such things right now, and the solution is to port the smaller generic apps to Linux, and keep the OS/2 investments in the others.
Quote from: demetrioussharpe on 2011.04.25, 04:40:42
Quote* Re-implementing a suitable swapping mechanism
With the advent of a 64 bit kernel - and thus more memory being accessible, numerous other Linux ports or native apps can become available that will be able to take advantage of the expanded memory space.
Swapping's something that's not always easy to achieve depending on the role that's being filled by the OS. Desktop OSs usually have different requirements than server OSs. This implies that all policies effecting this mechanism need to be dynamically changeable.
Hence the problem. Server OS or desktop OS; Warp does a far better job at this than Windows - whatever mechanism it uses.
Quote from: demetrioussharpe on 2011.04.25, 04:40:42
QuoteWhile I have very very little experience with how Linux does this, Windows is absolutely pathetic at swapping - often swapping in use code to disk while plenty of memory is available, while OS/2 is very very decent at swapping the right code to disk, and only swapping active code/data to disk in the event of physical memory exhaustion.
Sounds like there needs to be a better aging policy implementation.
Got me... I wouldn't want to touch the Windows code that handles it with a ten foot pole. As it is, they've written workarounds and kludges to try to alleviate those issues, instead of trying to fix them.
Quote from: demetrioussharpe on 2011.04.25, 04:40:42
CONTINUED IN NEXT POST
Quote from: RobertM on 2011.04.25, 06:25:01
Quote from: demetrioussharpe on 2011.04.25, 04:41:25
Quote* Reworking the shared arena
To take into account 64 bit code, larger memory availability and addressing capabilities, and so on - thus removing the limitations currently imposed on the shared arena. This is really part and parcel to the point above this.
Mainly, I don't think this will be much of an issue if the apps are given much more than just 512MB of memory. Even on a 32-bit kernel, I think that a 2GB/2GB, 3GB/1GB, or even 3.5GB/.5GB app/kernel split would be much better. I think that the arena idea came from the fact that compatibility with DOS programs & DLLs and resource sharing between 16- & 32-bit codebases were goals. I really don't see a reason to continue the arena paradigm, but if someone else has some enlightenment for me in this area, I'd love to hear it.
I am NOT sure what mechanisms are used, but I do know REXX does various resource sharing. That would be VERY dangerous to break, because of (a) how the OS uses REXX for certain things, (b) how various WPS add-ons use REXX and (c) because of numerous apps with REXX hooks or that utilize REXX code.
That aside, with a 64 bit memory range, using the 32 bit limitations for anything but compatibility would kinda defeat one of the purposes. As a for instance, the JFS and HPFS386 caches are limited by the sizes of those arenas. That's highly detrimental on things like database and web servers - on other operating systems, I can drop much larger amounts of memory to those daemons. As a for instance, considering the size of the Star Trek Phase 2 forum (and it's amount of traffic), it would be very nice if I could use higher memory amounts for stuff like the temp tables (that get created on both complex requests or on searches). Sadly, the amount of memory I can allocate is limited due to the arena size limitations created by the 32 bit addressable space. Especially since memory is also being allocated to disk caches, (database) table caches, thread management (in my case, just shy of 2000) and so on.
I'm sure that REXX could stand to me ported & extended to 64-bit, since we wouldn't want to confine it to living in a 32-bit compatibility box.
Quote from: demetrioussharpe on 2011.04.25, 06:50:34
I'm sure that REXX could stand to me ported & extended to 64-bit, since we wouldn't want to confine it to living in a 32-bit compatibility box.
That, especially considering it's way of assigning memory/variable objects, would be wonderful. Not to mention the benefits of OREXX being compiled with a real compiler that doesn't impact performance or cause massive memory leaks.
I guess that means I need to procure an AMD64 box. I think I wanna persue this, whether I'm accepted as the claimer of this bounty or not.
Quote from: RobertM on 2011.04.25, 06:53:23
Quote from: demetrioussharpe on 2011.04.25, 06:50:34
I'm sure that REXX could stand to me ported & extended to 64-bit, since we wouldn't want to confine it to living in a 32-bit compatibility box.
That, especially considering it's way of assigning memory/variable objects, would be wonderful. Not to mention the benefits of OREXX being compiled with a real compiler that doesn't impact performance or cause massive memory leaks.
Ooops, side note on this. Though REXX has been released from IBM's clutches, it does not include the OS/2 specific code or APIs, and will also lack various OS hooks.
Quote from: RobertM on 2011.04.25, 07:50:35
Ooops, side note on this. Though REXX has been released from IBM's clutches, it does not include the OS/2 specific code or APIs, and will also lack various OS hooks.
I'm sure that new bits can be written to weave REXX into the OS/2 replacement. Besides, there also has to be a new presentation manager & workplace shell. So, these parts would've had to have been rewritten anyway.
Is there anything else that needs to be addressed? Is it fair to work on a 32-bit kernel, with an eye to the future for a 64-bit one? Afterall, OW isn't ready for 64-bit targets yet & I wouldn't want to base our future on the GCC toolchain.
Is there anything else that needs to be discussed before I start working on a rough write up?
Quote from: demetrioussharpe on 2011.05.08, 00:49:01
Is there anything else that needs to be discussed before I start working on a rough write up?
I'd suggest talking to Roderick Klein to see if he can give you some direction, ideas or such. rwklein on the forums.
Quote from: RobertM on 2011.05.08, 01:42:43
Quote from: demetrioussharpe on 2011.05.08, 00:49:01
Is there anything else that needs to be discussed before I start working on a rough write up?
I'd suggest talking to Roderick Klein to see if he can give you some direction, ideas or such. rwklein on the forums.
I'm in constant contact with him. Just trying to get as much info as possible before development actually starts. I'm trying to find a middle ground between what's wanted, what's possible, & what's impractical.
If I switch to survival mode the goal should be:
- A complete OS2KRNL file replacement that works exactly as the current kernel at the Warp 4.52. Under a OSI open source license that will open the doors to keep improving/bug fixing the kernel. No third party copyrights, no strange patents that will limit us in the future. 64bits, new architecture, should not be a must. Something that you can change the OS2KRNL file and system keeps running like nothing happened. But that is only my way to see it. (since I'm complete dumb on how OS/2 internal works)
Sure, sound easy, but as it is not an easy goal to reach.
Like I told before, from the bounty perspective, if Demetrius want to work on this kind of project, you can offer the goal that can be reached and we can always talk to the sponsors to re-define the bounty.
Demetrius, even if you are not interested on working on this bounty I appreciate your analysis on this forum. I think your discussion is very useful for anybody interested on this topic.
I think the goals of this bounty should be well defined, but if we put a to extensive "santa's list" on it, will scare any developer to work on it against the money we can raise.
Martin,
Love your idea for it. It is a great starting point. Alas, OS2KRNL replacement will also probably need an OS2LDR replacement. Demetrious has his work cut out for him.
Another interesting thing, IIRC, OS2KRNL is currently part assembly and part C. Not that it's probably very relevant.
I'm also wondering how helpful the new/upcoming AMD BIOS in CPU stuff may be to this effort. Not so much on the Intel side, of course...
Quote from: miturbide on 2011.05.11, 22:23:09
If I switch to survival mode the goal should be:
- A complete OS2KRNL file replacement that works exactly as the current kernel at the Warp 4.52. Under a OSI open source license that will open the doors to keep improving/bug fixing the kernel. No third party copyrights, no strange patents that will limit us in the future. 64bits, new architecture, should not be a must. Something that you can change the OS2KRNL file and system keeps running like nothing happened. But that is only my way to see it. (since I'm complete dumb on how OS/2 internal works)
Sure, sound easy, but as it is not an easy goal to reach.
Like I told before, from the bounty perspective, if Demetrius want to work on this kind of project, you can offer the goal that can be reached and we can always talk to the sponsors to re-define the bounty.
Demetrius, even if you are not interested on working on this bounty I appreciate your analysis on this forum. I think your discussion is very useful for anybody interested on this topic.
I think the goals of this bounty should be well defined, but if we put a to extensive "santa's list" on it, will scare any developer to work on it against the money we can raise.
Ok, in the coming days, I'll write up a rough spec sheet to cover as many of the bases as possible & post them. If that's acceptable, then I'll work towards that spec sheet in an attempt to claim the bounty afterwards. If it's not suitable for the bounty, then I'll most likely still work towards that goal, however, I'll have to look at it from a business prespective & possibly turn it into a product. I may attempt to write it from scratch, but if there's something that's close enough, then I might use that as a starting point. Keep in mind, features that aren't feasible right now could be revisited without the necessity of a new bounty.
Quote from: RobertM on 2011.05.11, 23:14:09
Martin,
Love your idea for it. It is a great starting point. Alas, OS2KRNL replacement will also probably need an OS2LDR replacement. Demetrious has his work cut out for him.
Another interesting thing, IIRC, OS2KRNL is currently part assembly and part C. Not that it's probably very relevant.
I'm also wondering how helpful the new/upcoming AMD BIOS in CPU stuff may be to this effort. Not so much on the Intel side, of course...
I kinda assumed that a new kernel would need a new bootloader & a new userland in order to function. The bootloader can actually be outsourced from other projects. The userland has to be written specifically for the kernel, so I assumed that it was an implied task.
Quote from: demetrioussharpe on 2011.05.11, 23:50:31
Quote from: RobertM on 2011.05.11, 23:14:09
Martin,
Love your idea for it. It is a great starting point. Alas, OS2KRNL replacement will also probably need an OS2LDR replacement. Demetrious has his work cut out for him.
Another interesting thing, IIRC, OS2KRNL is currently part assembly and part C. Not that it's probably very relevant.
I'm also wondering how helpful the new/upcoming AMD BIOS in CPU stuff may be to this effort. Not so much on the Intel side, of course...
I kinda assumed that a new kernel would need a new bootloader & a new userland in order to function. The bootloader can actually be outsourced from other projects. The userland has to be written specifically for the kernel, so I assumed that it was an implied task.
LoL, I knew you already thought of that (or was pretty darn sure you'd already considered all of that). Was clarifying for others who may not have considered such.
Here's something I was just thinking of, due to another thread, but how many hand-offs/reliances (if any) are there in PMMERGE?
Quote from: RobertM on 2011.05.11, 23:56:22
Quote from: demetrioussharpe on 2011.05.11, 23:50:31
Quote from: RobertM on 2011.05.11, 23:14:09
Martin,
Love your idea for it. It is a great starting point. Alas, OS2KRNL replacement will also probably need an OS2LDR replacement. Demetrious has his work cut out for him.
Another interesting thing, IIRC, OS2KRNL is currently part assembly and part C. Not that it's probably very relevant.
I'm also wondering how helpful the new/upcoming AMD BIOS in CPU stuff may be to this effort. Not so much on the Intel side, of course...
I kinda assumed that a new kernel would need a new bootloader & a new userland in order to function. The bootloader can actually be outsourced from other projects. The userland has to be written specifically for the kernel, so I assumed that it was an implied task.
LoL, I knew you already thought of that (or was pretty darn sure you'd already considered all of that). Was clarifying for others who may not have considered such.
Here's something I was just thinking of, due to another thread, but how many hand-offs/reliances (if any) are there in PMMERGE?
I'd imagine that the skeleton DLLs that replaced the original DLLs that combined into PMMERGE.DLL all rely on it. But that's going to have to eventually be reimplemented also.
Quote from: RobertM on 2011.05.11, 23:56:22
Quote from: demetrioussharpe on 2011.05.11, 23:50:31
Quote from: RobertM on 2011.05.11, 23:14:09
Martin,
Love your idea for it. It is a great starting point. Alas, OS2KRNL replacement will also probably need an OS2LDR replacement. Demetrious has his work cut out for him.
Another interesting thing, IIRC, OS2KRNL is currently part assembly and part C. Not that it's probably very relevant.
I'm also wondering how helpful the new/upcoming AMD BIOS in CPU stuff may be to this effort. Not so much on the Intel side, of course...
I kinda assumed that a new kernel would need a new bootloader & a new userland in order to function. The bootloader can actually be outsourced from other projects. The userland has to be written specifically for the kernel, so I assumed that it was an implied task.
LoL, I knew you already thought of that (or was pretty darn sure you'd already considered all of that). Was clarifying for others who may not have considered such.
Here's something I was just thinking of, due to another thread, but how many hand-offs/reliances (if any) are there in PMMERGE?
Or are you asking about how many reliances that PMMERGE has on other parts of the system?
Quote from: demetrioussharpe on 2011.05.12, 05:25:24
Or are you asking about how many reliances that PMMERGE has on other parts of the system?
Nah, you've got it covered...
On to a couple other things not mentioned, but needed.
(1) code that handles swapper.dat needs to be rewritten to take advantage of JFS (ie: greater than 2GB)
Below here are things reported elsewhere, but that I have no knowledge of, so I cannot confirm their validity.
(2) It appears (or so people at EDM/2 and elsewhere have reported) that the OS/2 kernel manages far more than 4GB of in-use RAM. (this is also why the swapper.dat code is problematic).
Inotherwords, while certain arenas are both (a) fixed in size and (b) for all intents and purposes, limited by whatever that size is; other arenas apparently are not.
This is apparently due to something rarely talked about. OS/2 "mixes and matches" use of both the flat memory architecture (ie: limited to 4GB) and the paged architecture. By paging userspace memory needs, per the post here (http://www.mombu.com/programming/programming/t-swap-file-size-and-maximum-ram-memory-size-638065.html), success has been claimed for using up to 256GB on AMD CPUs. And per this OS/2 Voice article (http://www.os2voice.org/VNL/past_issues/VNL0708H/feature_3.html) on the kernel, it supposedly "may" do so up to the 64TB limit imposed by the x86 CPU architecture. They cite some use of that ability here (quoted from the article):
"In the old 16/32 bits model one could already run, say, 50 processes using 100 MiB of protected memory (total 5 GiB). And now we can run very big processes each using up to 1-3 gigabytes of protected mode memory concurrently."Of course, if this is true, the major limitation still resides in the maximum size of the swap file - and of course, OS/2's ability to properly utilize the paging architecture of the particular CPU.
Here's a page with a few links to others' work on figuring out the kernel, which you may have already found:
http://en.ecomstation.ru/projects/developer/?action=kernel
Quote from: RobertM on 2011.05.14, 20:47:48
Inotherwords, while certain arenas are both (a) fixed in size and (b) for all intents and purposes, limited by whatever that size is; other arenas apparently are not.
To be honest, I'm still not sure of the original purpose of splitting the memory address range into arenas...
Quote from: demetrioussharpe on 2011.05.14, 21:20:53
Quote from: RobertM on 2011.05.14, 20:47:48
Inotherwords, while certain arenas are both (a) fixed in size and (b) for all intents and purposes, limited by whatever that size is; other arenas apparently are not.
To be honest, I'm still not sure of the original purpose of splitting the memory address range into arenas...
(The major aspect): Protection. OS/2 uses Ring 0, 2, and 3 to enforce the various protection schemes that are part of the x86 architecture post 386.
Quote from: RobertM on 2011.05.14, 23:35:53
Quote from: demetrioussharpe on 2011.05.14, 21:20:53
Quote from: RobertM on 2011.05.14, 20:47:48
Inotherwords, while certain arenas are both (a) fixed in size and (b) for all intents and purposes, limited by whatever that size is; other arenas apparently are not.
To be honest, I'm still not sure of the original purpose of splitting the memory address range into arenas...
(The major aspect): Protection. OS/2 uses Ring 0, 2, and 3 to enforce the various protection schemes that are part of the x86 architecture post 386.
It's a rather odd scheme, it seems to be overcomplicated. All that was really needed was rings 0 & 3. Protection is provided between memory spaces on other OSes without this arena concept. So, is there something else that arenas bring to the table?
Quote from: demetrioussharpe on 2011.05.15, 16:49:45
Quote from: RobertM on 2011.05.14, 23:35:53
Quote from: demetrioussharpe on 2011.05.14, 21:20:53
Quote from: RobertM on 2011.05.14, 20:47:48
Inotherwords, while certain arenas are both (a) fixed in size and (b) for all intents and purposes, limited by whatever that size is; other arenas apparently are not.
To be honest, I'm still not sure of the original purpose of splitting the memory address range into arenas...
(The major aspect): Protection. OS/2 uses Ring 0, 2, and 3 to enforce the various protection schemes that are part of the x86 architecture post 386.
It's a rather odd scheme, it seems to be overcomplicated. All that was really needed was rings 0 & 3. Protection is provided between memory spaces on other OSes without this arena concept. So, is there something else that arenas bring to the table?
Tried, tested, stable, working, hardware enforced protection - as opposed to software based in such products as Windows non-working version. I have no idea how Linux's approach fairs.
I also believe it is somewhat related to how the paging system and flat virtual memory access system are used. Sadly, my knowledge on this is old, and with no first hand experience about the underlying code base... the only OS/2 code I've been privy to is in the print subsystem and portions of the GDI/GPI subsystems that are related.
Quote from: RobertM on 2011.05.15, 18:50:45
Quote from: demetrioussharpe on 2011.05.15, 16:49:45
Quote from: RobertM on 2011.05.14, 23:35:53
Quote from: demetrioussharpe on 2011.05.14, 21:20:53
Quote from: RobertM on 2011.05.14, 20:47:48
Inotherwords, while certain arenas are both (a) fixed in size and (b) for all intents and purposes, limited by whatever that size is; other arenas apparently are not.
To be honest, I'm still not sure of the original purpose of splitting the memory address range into arenas...
(The major aspect): Protection. OS/2 uses Ring 0, 2, and 3 to enforce the various protection schemes that are part of the x86 architecture post 386.
It's a rather odd scheme, it seems to be overcomplicated. All that was really needed was rings 0 & 3. Protection is provided between memory spaces on other OSes without this arena concept. So, is there something else that arenas bring to the table?
Tried, tested, stable, working, hardware enforced protection - as opposed to software based in such products as Windows non-working version. I have no idea how Linux's approach fairs.
Even still, many other OSes use only rings 0 & 3, because all that really matters is if the machine's running in user mode or kernel mode. All 4 modes offered by x86 aren't really necessary. In face, afaik, OS/2 only uses 3 out of 4 of the protection levels. I'm more than confident that it's using one level too many. It needs to get off the fence. Either the process is in the kernel or it's outside of the kernel. And if it's outside of the kernel, there's no real reason for it to be running under any level other than ring 3.
Quote from: RobertM on 2011.05.15, 21:33:37
I also believe it is somewhat related to how the paging system and flat virtual memory access system are used. Sadly, my knowledge on this is old, and with no first hand experience about the underlying code base... the only OS/2 code I've been privy to is in the print subsystem and portions of the GDI/GPI subsystems that are related.
I understand taking advantage of the platform that you're running on. However, it just feels as if they'd gotten kinda carried away & tied themselves too tightly to x86. Perhaps they should've just stuck to the basics in some aspects.
Just a heads up. If I understand correctly, 16-bit code won't execute on a 64-bit machine in long 64-bit mode. I'm not sure yet about 32-bit code. But, with that being the case, there's going to be a huge push to create a standard for 64-bit functions, structures, & general data. In short, 64-bit OS/2 will need to be standardized. After soliciting advice from others in the OS/2 community, I was advised to try using elf in 64-bit mode for OS/2-64. Does anyone have an issue with this? Keep in mind, this is a long term project that entails creating the whole software stack from development tools to OS to drivers to APIs to applications. In essence, a brand new operating system & ecosystem. Is there anyone else willing to contribute?
Correct, 32 bit code should execute fine. 16 bit code will not execute. In my laboriously overly verbose way below, that's what I was trying to mention when I was asking about intercepting any 16 bit calls and executing them as (at least) 32bit - hence the "Odin-like" suggestion.
Sorry it's been awhile. I haven't been able to work on any kernel work because of a combination of preparing to move & medical issues. I'm scaling back most of my development efforts & sadly, this is one of the projects that's under the ax right now. Perhaps I'll return to this project once I'm doing better, health-wise.
I'm sorry to learn about your medical problems.
I hope that they are short-lived and leave without a trace.
Quote from: Ben on 2011.07.17, 22:33:35
I'm sorry to learn about your medical problems.
I hope that they are short-lived and leave without a trace.
Thank you. Unfortunately, I now have to learn how to live as a diabetic & how to drop my triglyceride levels. Apparently, I can't escape genetics...
Hi Dee,
wishing you all the best from the other side of the pond.
Keep on rockin', dude.
Cheers,
Thomas