Author Topic: Closed source parts of eComStation  (Read 18815 times)

Neil Waldhauer

  • Sr. Member
  • ****
  • Thank You
  • -Given: 6
  • -Receive: 15
  • Posts: 295
    • View Profile
    • Blonde Guy
Closed source parts of eComStation
« on: December 27, 2013, 07:20:12 pm »
I'm wondering how much work there is left to leave the IBM parts of eComStation behind and going to an unencumbered product?

OS/2 Kernel
MPTS
MMOS2
Presentation Manager
WorkPlace Shell
GRADD Graphics subsystem

I'm sure I forgot a few, and there are lots of little bits that haven't ever been replaced (look inside the OS2 directory tree, but those are the big subsystems.

File and Print is largely replaced by SAMBA and CUPS.


Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 244
  • -Receive: 46
  • Posts: 1642
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: Closed source parts of eComStation
« Reply #1 on: December 27, 2013, 08:17:17 pm »
Hi Neil.

I tried to talk a little bit about this on the earlier posts of my blog: http://openwarp.blogspot.com

Since I'm not very sharp on this subject I tried to understand the components from "Top to down". So I think that the three components that are almost the "face" of OS/2 are:
- Workplace Shell Classes - which are based in SOM
- SOM - which depends on PM
- PM - which is the GUI library.

I think that any efforts to start with any of this three components are good... no matter how little the effort is, like replacing some PM's DLL or making a single  WPS Class.

The stuff in the backend I don't understand it well yet, I will really like to create a more detailed graphic about it. Like understanding the relation between the Kernel, Loader, Drivers and the "Control Program", "GPI" , "REXX library", etc. I'm still now sure what the "Control Program" is, or where does the "GPI" stuff goes.

I tried to create a very high level graphic of the main OS/2 architectural components.

Martin Iturbide
OS2World NewsMaster
... just share the dream.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 244
  • -Receive: 46
  • Posts: 1642
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: Closed source parts of eComStation
« Reply #2 on: December 27, 2013, 09:15:28 pm »
Here it is the ODG (openoffice) file for the architecture graphics.

Everybody is welcome to go wild and create their own versions.
Martin Iturbide
OS2World NewsMaster
... just share the dream.

ivan

  • Hero Member
  • *****
  • Thank You
  • -Given: 0
  • -Receive: 24
  • Posts: 632
    • View Profile
Re: Closed source parts of eComStation
« Reply #3 on: December 27, 2013, 11:43:53 pm »
Neil, if you can get SAMBA and/or CUPS to print over the network to a NAS based print server that requires a username and password to log in I would love to know the necessary incantations and what must be sacrificed to do so.

Roderick Klein

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 7
  • Posts: 158
    • View Profile
Re: Closed source parts of eComStation
« Reply #4 on: December 28, 2013, 12:48:58 am »
I'm wondering how much work there is left to leave the IBM parts of eComStation behind and going to an unencumbered product?

OS/2 Kernel
MPTS
MMOS2
Presentation Manager
WorkPlace Shell
GRADD Graphics subsystem

I'm sure I forgot a few, and there are lots of little bits that haven't ever been replaced (look inside the OS2 directory tree, but those are the big subsystems.

File and Print is largely replaced by SAMBA and CUPS.


MOre work then you want to wish for. Its realy the question if you need certain of eCS in a new OS.
For example GRADD. If you do start from scratch you could just as well desugn something new or use something from Linux.

As to reimplementing the OS2KRNL. WHat features would you like to have of the kernel ?
You would need to set much more fin gradined targets. Looking how far the voyager priject came of Netlabs.
Its certainly not going to move forward fast without a big chunk of money.

Roderick

Paul Smedley

  • Sr. Member
  • ****
  • Thank You
  • -Given: 11
  • -Receive: 46
  • Posts: 354
    • View Profile
Re: Closed source parts of eComStation
« Reply #5 on: December 28, 2013, 12:52:27 am »
Neil, if you can get SAMBA and/or CUPS to print over the network to a NAS based print server that requires a username and password to log in I would love to know the necessary incantations and what must be sacrificed to do so.

To be honest - I thought this was possible with cups - I would expect specifying the printer queue as smb://username:Password@ip_address in CUPS would do this (but I could be wrong)

ivan

  • Hero Member
  • *****
  • Thank You
  • -Given: 0
  • -Receive: 24
  • Posts: 632
    • View Profile
Re: Closed source parts of eComStation
« Reply #6 on: December 28, 2013, 02:47:27 pm »
Thanks Paul.  I will give that a try on Monday when I go into work although I seem to remember trying something like that before and getting no response.

Is there any thing special I should be looking for?  Since all the printers are HP CLG 3500 I have your hplip installed with CUPS on my test machine.  Now all I have to do is work out what to do.

Boris

  • Sr. Member
  • ****
  • Thank You
  • -Given: 11
  • -Receive: 16
  • Posts: 310
    • View Profile
Re: Closed source parts of eComStation
« Reply #7 on: January 14, 2014, 02:14:30 am »
As to reimplementing the OS2KRNL. WHat features would you like to have of the kernel ?

Roderick

For sure, the very huge and very important part that really needs to be reimplemented is the stuff that do memory management.
Memmanagement is known to be very poor (two arenas, continous troubles with shrmem) on OS/2. Also the 4 GiB for all the processes limit surely should be overcamed.
With the latest development of Phoenix team -- PAE ramdrive, you can use up to 64 GiB of memory on OS/2 (with any kernel, any version of OS/2 from Warp 3 to eCS *nextVersion), but even with swapfile placed on the ramdrive, the limit of 4 GiB/all processes still remains.

This will not remove the limit of 4 GiB per ONE process, but for sure will be a great move forward for OS/2, because you, of course, can just "write another videodriver", but the 4 GiB of RAM limitation for all the system is now a fundamental barrier, which SHOULD be overcamed as soon as possible.



Another big trouble, a bottleneck of Operating System/2, is the graphics subsystem and some parts of WPS.
While the most annoying defect of WPS -- instability -- really needs attention, the Presentation Manager and the graphics subsystem itself inhibits the development of eComStation very much.

PM and the graphics subsystem is a huge bunch of code, and its reverse-engineering and (preferably) opensourcing should became the next long-term goal of the whole eComStation community in general and Mensys in particular.
While WPS improvement (making it usable as a light filemanager -- copy/move dialog replacement, file open container (opensource it!), etc) may be performed relatively easy, in view of XWP, which is a very good start (or even the half of a distance), PM and the graphics subsystem need a lot more work, and should became the primary target.
Its defects are: inability of on-the-fly resolution change, no support of multidisplay configurations (one stretched desktop (!) on many monitors with different resolutions (!!) connected to a number (!!!) of different (!!!!) videocards ran by different videodrivers (!!!!!) -- this is fully possible on Microsoft Windows and impossible on OS/2), and no support of transparency (=>non-square windows, shadows).
Consider, that Windows, GNU/Linux, Mac OS X do support all these features.



Also, if I started to whine here, I am also gonna note a few problems, no one of which are also encountered on other OSes, and which are easy to fix.

First. USBCOM driver. It just is not suitable for anything. USBCOM devices should be plug and playable (no need to preconfigure the amount of usbcom devices in the config.sys), three or five more chips support should be added.
Ubuntu 12 supports USBCOM hardware very well -- I've tryied three devices which didn't work with OS/2, on Ubuntu -- worked well out-of-the-box, no additional drivers needed.
The problem may be easily fixed (look at the usbserial module sources of the linux kernel).
There is no attention paid to this problem.

Second. USBMSD with SCSI-Transparent command set support should be implemented. Chris Wohlgemut once did this, but his driver does not work for my device, the driver's source code is not opensourced.
Windows XP with no service packs support these hardware w/o any additional drivers needed, out-of-the-box.
The problem may be easily fixed (look at the usbmsd module sources of the linux kernel).
There is no attention paid to this problem.



I suppose, that's all :)
LIABILITY DISCLAIMER: this is how I understand and what I know, I may be highly inaccurate, or even completely wrong! There are no claims, promises, or guarantees about the accuracy, completeness, or adequacy of the contents of my posts. Think on your own!

Fr4nk

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 2
  • Posts: 32
    • View Profile
    • subsys.de
Re: Closed source parts of eComStation
« Reply #8 on: January 14, 2014, 10:10:12 am »
Quote
the 4 GiB of RAM limitation for all the system is now a fundamental barrier

Could you please name one OS/2 application or one use case where we need more than 4 GB RAM?

Of course it would be nice to have but there is no priority for this from a point of a user. I understand that people buying windows machines and then wonder if they eCS work on it. But for me as an real user (I'm working 8h each day with eCS) there are about 100 things more important than >4GB RAM. As the fairy said: don't waste wishes if you have only three. ;)

I agree to get things solved like shared memory problem or USB-issues.
« Last Edit: January 14, 2014, 10:12:47 am by Fr4nk »

Michael Holzapfel

  • Newbie
  • *
  • Thank You
  • -Given: 2
  • -Receive: 4
  • Posts: 47
    • View Profile
Re: Closed source parts of eComStation
« Reply #9 on: January 14, 2014, 12:42:45 pm »
Could you please name one OS/2 application or one use case where we need more than 4 GB RAM?
Well, darktable and rawtherapee would be my candidates. Using eCS on daily base becomes more and more difficult to meet the requirements of digital photography.

Furthermore storing large video files (4K) for NAS-services like mediatomb is a real pain with the 2TB fize limitation for file systems.

The foregoing discussion increasingly resembles that of the transition from 16 to 32 bit operating systems. For a long time Microsoft had denied the need of 32 bit systems. We as eCS-users don't even have the possibility to use tools like dos-extenders to utilize all the available memory on our hardware.

BR
Michael

Boris

  • Sr. Member
  • ****
  • Thank You
  • -Given: 11
  • -Receive: 16
  • Posts: 310
    • View Profile
Re: Closed source parts of eComStation
« Reply #10 on: January 14, 2014, 04:39:51 pm »
Could you please name one OS/2 application or one use case where we need more than 4 GB RAM?
I run virtual machines and console java applications on a server with two CPUs and 4 GiB of RAM running OS/2.
Firefox could easily take more than 4 GiB of RAM (if not the crashes).
I would like to run virtual machines on my main PC (eComStation 2.1) and don't look on the memory indicator.

Quote
Of course it would be nice to have but there is no priority for this from a point of a user. I understand that people buying windows machines and then wonder if they eCS work on it. But for me as an real user (I'm working 8h each day with eCS) there are about 100 things more important than >4GB RAM. As the fairy said: don't waste wishes if you have only three. ;)
As far as you can see, I am a real OS/2 user. There are about four machines at my home that run OS/2. I am writing you from my laptop that runs eCS 2.1. I think I use OS/2 for about six (or more on weekand/holidays) hours per day.
I have participated in OS/4 kernel testing with success.
Also I've ported two small Qt utilities to OS/2.


For the *ucks sake, the "OS/2 does not need that" tactic IS A BAD TACTIC.
That's why we don't have skype, for example.

4 GiB per all processes is a fundamental barrier not because it is the most annoying defect, (maybe you need to reread me, huh?) it's because in 2020 you can write another video or audio driver, port another videoplayer or write USB 5.0 host controller driver, but the 4 GiB limitation still remains, and that's the limtation that prevents OS/2 from running on current, big servers. Symmetric multiprocessing support is good enough. Multitasking is good enough too. But nobody will even think to run OS/2 on a server with 64 GiB of RAM.

Don't wait 'til the 4 GiB limit will be the most annoying one, because it needs a lot of work, and we need to start as soon as possible.
If we are rewriting the memory manager, it will be too silly not to remove the 4 GiB/all processes limit.
LIABILITY DISCLAIMER: this is how I understand and what I know, I may be highly inaccurate, or even completely wrong! There are no claims, promises, or guarantees about the accuracy, completeness, or adequacy of the contents of my posts. Think on your own!

Boris

  • Sr. Member
  • ****
  • Thank You
  • -Given: 11
  • -Receive: 16
  • Posts: 310
    • View Profile
Re: Closed source parts of eComStation
« Reply #11 on: January 14, 2014, 04:45:28 pm »
We as eCS-users don't even have the possibility to use tools like dos-extenders to utilize all the available memory on our hardware.

Actually we do have the possibility.
The PAE ramdrive acts just like a dos-extender.
The hd4disk.add driver first turns off normal paging, then sets PAE paging, reads/writes from the upper memory, then turns off PAE and restores the normal paging.

For example, you can use the latest Windows XP 32-bit with no PAE support to access all the memory by putting the swapper file on a ramdrive, that supports PAE. And that really do work.

While on OS/2 it won't work, because of very poor memory manager.
LIABILITY DISCLAIMER: this is how I understand and what I know, I may be highly inaccurate, or even completely wrong! There are no claims, promises, or guarantees about the accuracy, completeness, or adequacy of the contents of my posts. Think on your own!

Fr4nk

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 2
  • Posts: 32
    • View Profile
    • subsys.de
Re: Closed source parts of eComStation
« Reply #12 on: January 14, 2014, 09:37:13 pm »
Boris,

no reason to get angry. I don't talk about tactic. We have very small capacities of developers. So I'm talking about priorities.

To say os2 don't needs above 4GB RAM is no tactic - it is a fact. Because we don't have applications that use such an amount of RAM. Maybe it can be a problem in future. But today there are loads of things more important and that drops me back to high priorities.

And when you talk about running loads of virtual machines with more RAM: its a bad tactic to design a system, to perfectly run other systems - while the own identity is going down (and btw other systems do the visualization even better right now). So at first priority you need unique software and special use cases. If you want to sell a OS like eCS for 270 bucks  you need a unique reason. Something special. One of the good points of eCS or OS/2 is that it runs very smooth one small hardware. Its a very powerful system with very small system requirements.

So when we talk about "fundamental barriers" the first point should be to find a reason why to use this system and not another one.
Then the price is a "fundamental barrier" for new users. Then we come to installation and communication. E.g. it should be no problem to share data with USB sticks. It should be no need to fire up dfsee to make an USB stick running (if youre lucky enough). ACPI and basic drivers. And then we come to lack of standard software, which is a real fundamental barrier. We don't have real working video player nor a full featured bitmap editor nor a really good and stable office suite. To use your words: "ffs" Mensys doesn't even have reference system.

And, and, and ... and then after maybe we really need more than 4 GB.

Atm eCS lives because some people have been arranged with the situation, using old software and workarounds, or use the system in a very basic way.

@Michael
When you talk about bitmap and video manipulation - stop me if I'm wrong but we don't have these applications  for OS2. I would be fine when I'm wrong. We don't even have decent working version of gimp. 4 GB limit might be a small part of the whole problem. When we get 100 GB RAM we don't solve this situation.

Boris

  • Sr. Member
  • ****
  • Thank You
  • -Given: 11
  • -Receive: 16
  • Posts: 310
    • View Profile
Re: Closed source parts of eComStation
« Reply #13 on: January 14, 2014, 10:19:03 pm »
Boris,

no reason to get angry. I don't talk about tactic. We have very small capacities of developers. So I'm talking about priorities.

To say os2 don't needs above 4GB RAM is no tactic - it is a fact. Because we don't have applications that use such an amount of RAM. Maybe it can be a problem in future. But today there are loads of things more important and that drops me back to high priorities.

And when you talk about running loads of virtual machines with more RAM: its a bad tactic to design a system, to perfectly run other systems - while the own identity is going down (and btw other systems do the visualization even better right now). So at first priority you need unique software and special use cases. If you want to sell a OS like eCS for 270 bucks  you need a unique reason. Something special. One of the good points of eCS or OS/2 is that it runs very smooth one small hardware. Its a very powerful system with very small system requirements.

So when we talk about "fundamental barriers" the first point should be to find a reason why to use this system and not another one.
Then the price is a "fundamental barrier" for new users. Then we come to installation and communication. E.g. it should be no problem to share data with USB sticks. It should be no need to fire up dfsee to make an USB stick running (if youre lucky enough). ACPI and basic drivers. And then we come to lack of standard software, which is a real fundamental barrier. We don't have real working video player nor a full featured bitmap editor nor a really good and stable office suite. To use your words: "ffs" Mensys doesn't even have reference system.

And, and, and ... and then after maybe we really need more than 4 GB.

Atm eCS lives because some people have been arranged with the situation, using old software and workarounds, or use the system in a very basic way.

@Michael
When you talk about bitmap and video manipulation - stop me if I'm wrong but we don't have these applications  for OS2. I would be fine when I'm wrong. We don't even have decent working version of gimp. 4 GB limit might be a small part of the whole problem. When we get 100 GB RAM we don't solve this situation.

Stop for a minute. And follow the thread.
Roderick:
Quote
As to reimplementing the OS2KRNL. WHat features would you like to have of the kernel ?

Roderick
Then me, talking about the 4 GiB limit. So, that's not offtopic. For example, talking about GIMP in the what-do-you-need-in-os2-kernel thread, would be offtopic.

While we have enough developers to solve the 4 GiB problem, we need to do this.

Because you can edit bitmaps in a virtual machine on OS/2, but you can't use virtual machine to allocate more than 4 GiB memory.
You can now transfer data from USB sticks, but you can't use more than 4 GiB of address space.
That's what I call a fundamental barrier.

You can rearrange priorities, and more, fixing a fundamental problem may have a lower priority than fixing the USB widget to write DLAT *: at a usb stick connect, but, sooner or later, you will end up with not enough virtual memory and perfectly working USB sticks, and no way to fix that, because there'll be no developers that are able to rewrite memory management.
LIABILITY DISCLAIMER: this is how I understand and what I know, I may be highly inaccurate, or even completely wrong! There are no claims, promises, or guarantees about the accuracy, completeness, or adequacy of the contents of my posts. Think on your own!

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 38
  • Posts: 928
    • View Profile
Re: Closed source parts of eComStation
« Reply #14 on: January 15, 2014, 05:38:42 am »
Quote
While we have enough developers to solve the 4 GiB problem, we need to do this.

Well, I think you are being  bit optimistic. We don't even have enough developers to get a Wireless NIC working (although that seems to be on the short list).

Personally, I now have 4 systems with 4 GB of memory (one is not currently used with eCS). I could load up every program that I have installed, and I wouldn't fill up all of that memory (unless I made some large virtual memory systems, in VBox). At the moment, the 4 GB (actually something closer to 3.3 GB) addressing limit is not a limiting factor. More limiting is the 500 meg memory space for each program (which includes shared memory space).

Very recently, the QSINIT project has demonstrated that using memory above the 32 bit addressing limit is possible. That is currently restricted to using it as a RAMDISK, but it has been demonstrated that it does work. More serious work is required to be able to use that memory more productively, and it is likely that programs would need to be modified to use it. Note that this is NOT the ideal solution, but, at the moment, it is the ONLY game in town.

Quote
That's what I call a fundamental barrier.

It is, BUT, there are many other barriers, that are much more limiting, and require a lot less resources to fix. If the manpower is redirected to fixing the 4 GB barrier, the other, more important, problems will not be addressed. If they are not addressed, there will be no point in having more than 4 GB of usable memory. Then, there is the question about which programs you will need, that will actually use that much memory, and where are they going to come from?

I don't mind people asking for a way to get to the next solar system, as long as they don't insist that we stop all other development to accomplish it. We have enough problems, without shooting ourselves in the foot. In due time, eCS will either die, or it will continue to try to keep up with the rest of the world. As I have said before, the only way that eCS is likely to accomplish any goals, beyond that, is to accumulate enough money to do the job. That means that all of you who seem to think that Mensys owes you free goodies, without buying into the project, had better wake up, and contribute to the project, or, somebody needs to win a HUGE lottery, and donate a good portion of it to the project (through Mensys, or in other ways). You can wish all you want, but insisting that we need to be able to do something that is clearly not possible (in today's world), is just going to kill eCS (OS/2). faster.

Now, if Mensys can just get eCS 2.2 out the door, so they can get to work on Multimac support for wireless devices, it will be the first step to getting some of this done. Meanwhile, nobody has stepped up to use the wired Multimac templates, to produce more support for wired NICs. It has always been the plan that Mensys would develop a few NIC drivers, which they did (Intel, Realtek, and NVIDIA), and then let others follow the lead, and make more (Broadcom, Ralink, etc.). Mensys did their part, now we, the OS/2 community, need to step up and do our part. Sitting back, and waiting for someone else to do it, is not an option, and Mensys can't afford to do much more than what they already do.

The ball is in your court...