Author Topic: PMSHELL  (Read 17309 times)

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
PMSHELL
« on: July 29, 2018, 08:50:51 pm »
Did some of you get storage issue ?
I saw that as soon my shared memory gors below 10Mb, I have display freeze and fullscreen mode fails.
I check free memory using SL.

I tried to kill PMSHELL used by WPS which results into a WPS restart and 70MB freed memory into share storage.
How to configure to have shared memory freed as soon it is no more used ?
Thanks     

Doug Bissett

  • Hero Member
  • *****
  • Posts: 1593
  • Karma: +4/-2
    • View Profile
Re: PMSHELL
« Reply #1 on: July 29, 2018, 10:54:07 pm »
I wouldn't call it a "storage" issue. It is a shortage of lower shared memory space. I find that it is only a matter of a few minutes, until my system crashes, when lower shared memory drops below 100 MB (it is probably already dead if it shows 10 MB). To avoid the problem, I add the line"
Code: [Select]
VIRTUALADDRESSLIMIT=2560to CONFIG.SYS. Some systems will be able to use 3072, but, apparently, some other systems can't use that. It only increases the available upper shared memory anyway.

Doing that does not actually change the problem, without taking further steps to mark some things to actually use upper shared memory space (there are some fixes in the ArcaOS kernel for upper shared memory support too). The easiest thing to set to use upper shared memory space, is Apache Open Office. In the Extras folder, there is a tool to set it to use upper shared memory (some experimenting may be required). If you do that, you may find that there will be a LIBC problem (usually not obvious), if you close Open Office more than once. To avoid that, use the Quickstart thing, to prevent it from actually closing.

You should also realize, that shared memory may fragment. If the largest fragment is 5 MB, and some program needs 5.5 MB, you are dead in the water. If the program is properly written, to check if it actually got the requested memory, it will probably just crash the program. If it doesn't check (apparently, most programs don't), and tries to use it anyway, it usually causes what you describe.

Roderick Klein

  • Hero Member
  • *****
  • Posts: 655
  • Karma: +14/-0
    • View Profile
Re: PMSHELL
« Reply #2 on: July 29, 2018, 11:32:39 pm »
I wouldn't call it a "storage" issue. It is a shortage of lower shared memory space. I find that it is only a matter of a few minutes, until my system crashes, when lower shared memory drops below 100 MB (it is probably already dead if it shows 10 MB). To avoid the problem, I add the line"
Code: [Select]
VIRTUALADDRESSLIMIT=2560to CONFIG.SYS. Some systems will be able to use 3072, but, apparently, some other systems can't use that. It only increases the available upper shared memory anyway.

Doing that does not actually change the problem, without taking further steps to mark some things to actually use upper shared memory space (there are some fixes in the ArcaOS kernel for upper shared memory support too). The easiest thing to set to use upper shared memory space, is Apache Open Office. In the Extras folder, there is a tool to set it to use upper shared memory (some experimenting may be required). If you do that, you may find that there will be a LIBC problem (usually not obvious), if you close Open Office more than once. To avoid that, use the Quickstart thing, to prevent it from actually closing.

You should also realize, that shared memory may fragment. If the largest fragment is 5 MB, and some program needs 5.5 MB, you are dead in the water. If the program is properly written, to check if it actually got the requested memory, it will probably just crash the program. If it doesn't check (apparently, most programs don't), and tries to use it anyway, it usually causes what you describe.

If I did not understand you correctly Doug. You typed:
"without taking further steps to mark some things to actually use upper shared memory space".
However GCC applications already store data (such as browser cache) into the high area.

For Remy:

The real killer is the size of the large DLL's that get loaded this can fixed by using a tool to mark the code segments to load high. Do this for Firefox DLL's, THunderbird AND Open Office.  Do not mark the datasegments high of the DLL.
Also make a backup of your DLL's before doing this.  There is a tool from Yuri called high512 that can mark the DLL's high.

You are most likely not the first and the last OS/2 user to expereince these type of lockups. It might be usefull to report this in a ticket to AN ?

Roderick

David McKenna

  • Hero Member
  • *****
  • Posts: 742
  • Karma: +24/-0
    • View Profile
Re: PMSHELL
« Reply #3 on: July 30, 2018, 01:19:22 pm »
 I wonder if it would be possible in theory to create a 'memory defragmentation' daemon that monitors memory and warns when fragments get below a specified size (or even better, defrags when needed)?

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: PMSHELL
« Reply #4 on: July 30, 2018, 02:38:59 pm »
I would say no. Of course it's possible to monitor memory usage (above512 and others) and fragmentation (interpret Theseus memory information and calculate the gaps) but what is the value when you should be warned about to 'low' memory?

From the kernels POV memory is not to 'low' until an application request memory and the kernel can not fulfill this request. Not earlier. Moreover even if you think some memory is low (as above512 tells you) you do not know if there is some large block already freed by some application somewhere. The kernel knows this can be given to another application if requested  (interpret Theseus and you may know too). But it depends on the value of memory the application requests. You don't know beforehand how much memory f.i. a java script wants to allocate before this specific one runs and makes the request to the OS.

When it comes to defragmentation it's simply not possible AFAIK. The kernel would need to to tell some application to free some specific memory. But neither the kernel nor the applications have implemented such algorithm.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: PMSHELL
« Reply #5 on: July 30, 2018, 04:16:54 pm »
I wouldn't call it a "storage" issue. It is a shortage of lower shared memory space. I find that it is only a matter of a few minutes, until my system crashes, when lower shared memory drops below 100 MB (it is probably already dead if it shows 10 MB). To avoid the problem, I add the line"
Code: [Select]
VIRTUALADDRESSLIMIT=2560to CONFIG.SYS. Some systems will be able to use 3072, but, apparently, some other systems can't use that. It only increases the available upper shared memory anyway.

Doing that does not actually change the problem, without taking further steps to mark some things to actually use upper shared memory space (there are some fixes in the ArcaOS kernel for upper shared memory support too). The easiest thing to set to use upper shared memory space, is Apache Open Office. In the Extras folder, there is a tool to set it to use upper shared memory (some experimenting may be required). If you do that, you may find that there will be a LIBC problem (usually not obvious), if you close Open Office more than once. To avoid that, use the Quickstart thing, to prevent it from actually closing.

You should also realize, that shared memory may fragment. If the largest fragment is 5 MB, and some program needs 5.5 MB, you are dead in the water. If the program is properly written, to check if it actually got the requested memory, it will probably just crash the program. If it doesn't check (apparently, most programs don't), and tries to use it anyway, it usually causes what you describe.

It is already set to VIRTUALADDRESSLIMIT=2560 but this is usefull for high memory.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: PMSHELL
« Reply #6 on: July 30, 2018, 04:19:43 pm »
I wouldn't call it a "storage" issue. It is a shortage of lower shared memory space. I find that it is only a matter of a few minutes, until my system crashes, when lower shared memory drops below 100 MB (it is probably already dead if it shows 10 MB). To avoid the problem, I add the line"
Code: [Select]
VIRTUALADDRESSLIMIT=2560to CONFIG.SYS. Some systems will be able to use 3072, but, apparently, some other systems can't use that. It only increases the available upper shared memory anyway.

Doing that does not actually change the problem, without taking further steps to mark some things to actually use upper shared memory space (there are some fixes in the ArcaOS kernel for upper shared memory support too). The easiest thing to set to use upper shared memory space, is Apache Open Office. In the Extras folder, there is a tool to set it to use upper shared memory (some experimenting may be required). If you do that, you may find that there will be a LIBC problem (usually not obvious), if you close Open Office more than once. To avoid that, use the Quickstart thing, to prevent it from actually closing.

You should also realize, that shared memory may fragment. If the largest fragment is 5 MB, and some program needs 5.5 MB, you are dead in the water. If the program is properly written, to check if it actually got the requested memory, it will probably just crash the program. If it doesn't check (apparently, most programs don't), and tries to use it anyway, it usually causes what you describe.

If I did not understand you correctly Doug. You typed:
"without taking further steps to mark some things to actually use upper shared memory space".
However GCC applications already store data (such as browser cache) into the high area.

For Remy:

The real killer is the size of the large DLL's that get loaded this can fixed by using a tool to mark the code segments to load high. Do this for Firefox DLL's, THunderbird AND Open Office.  Do not mark the datasegments high of the DLL.
Also make a backup of your DLL's before doing this.  There is a tool from Yuri called high512 that can mark the DLL's high.

You are most likely not the first and the last OS/2 user to expereince these type of lockups. It might be usefull to report this in a ticket to AN ?

Roderick

Arent't there set to load high by default?
How to know all dll's which aren't set to be loaded into high storage area?
Thanks

For now, I'm unable to boot ArcaOS after having got a TRAP0003 and lost of many JFS link entries... (made unrecoverable by JFS)
Trying to recover my system

I think it is no more possible to recover...
264 file(s) where scratched during the trap and a lot of uncoverable files... (including, exe, dll and drivers)  :( :o
« Last Edit: July 30, 2018, 05:14:30 pm by Remy »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: PMSHELL
« Reply #7 on: July 30, 2018, 07:07:19 pm »
Hi Rémy. About high memory support in OS/2.
Understand by default OS/2 ver 2-4 only supported using the 1st GB of address space due to needing to be compatible with OS/2 ver 1.x along with the 16 bit code still in OS/2.
While IBM added support for high memory in Warp Server and then the desktop at ver 4.5 (fp#13), it was never finished. The kernel has bugs with it that have been slowly patched in the most recent kernels so it is best to not mark DLLs to load high before shipping them but rather leave it to the user to mark them.
Only some DLLs can be marked high and continue to function, do not mark system DLLs and mileage can vary depending on your setup. For Mozilla, I mostly recommend marking xul.dll to load code high though most all the DLLs can be marked. It is possible to use Theseus to see DLLs that are loaded high and I guess you could scan DLLs to see if the load high bit has been set.
About VIRTUALADDRESSLIMIT. while the CPU can access 4GBs of address, generally the top of memory is given over to hardware, PCI devices, video memory and such are mapped into the top of memory. How much is used is going to vary depending on your hardware, though generally about half a GB. Once again IBM never finished support so there i some experimentation involved in deciding the value to use. I've had hardware that worked fine with the maximum setting of 3072, this current hardware, I just reduced it to 2560, which fixed a few weird issues I was having and made the system more stable, previously it was locking up regularly. Once again need to experiment, best if the numbers are even when expressed in HEX, B00H, C00H, A00H etc.

I'm sure I'm leaving stuff out and this is just my understanding.

Doug Bissett

  • Hero Member
  • *****
  • Posts: 1593
  • Karma: +4/-2
    • View Profile
Re: PMSHELL
« Reply #8 on: July 30, 2018, 07:23:47 pm »
Quote
However GCC applications already store data (such as browser cache) into the high area.

Only when they are programmed to do so. It looks, to me, like very few actually do that by default. Of course, you do need the ArcaOS kernel patches to be able to use it successfully, so it would be silly for programs to automatically use high shared memory since not all users have upgraded.

I wonder if it would be possible in theory to create a 'memory defragmentation' daemon that monitors memory and warns when fragments get below a specified size (or even better, defrags when needed)?

Of course, it is "possible". The problem would be to find somebody who can do it. I doubt if anybody knows enough about how it works, to be able to do anything about it.

Quote
but what is the value when you should be warned about to 'low' memory?

I use Above512 with a REXX script, that displays, and logs, the results every second. I have seen lower shared memory completely disappear from 100 MB (crash, before it logged 0 MB), and I have seen it run successfully with 1 MB left. If there was some way to intercept the request, and be sure that it would be successful, before returning to the program, a warning could be issued, and the user could be allowed to close other programs, then tell the interceptor to check again, to see if it would be successful, and return to the program if it is okay (the programs should do that themselves, but it seems that most do not, and that may not even be the programmers fault).

Quote
Arent't there set to load high by default?

Not from what I have seen (but it has been a while since I looked). You can't do that, without having a separate set of DLLs for those systems than can use upper shared memory, and for those that can't use upper shared memory.

Quote
How to know all dll's which aren't set to be loaded into high storage area?

HIGHMEM.EXE (it is included in Open Office) will tell you. It can also set DLLs to load data and/or code high, or reset them to load low. Be warned that simply setting all DLLs to load high will probably not work well, and some programs may work with individual DLLs loaded high, while others won't. I find that it is best to leave the common DLLs (in RPM/YUM) alone, and only play with the program specific DLLs (such as XUL.DLL in mozilla).

Some experimenting is required, because each user has a different set of programs that they use, in different ways.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: PMSHELL
« Reply #9 on: July 30, 2018, 07:52:13 pm »
Quote
However GCC applications already store data (such as browser cache) into the high area.

Only when they are programmed to do so. It looks, to me, like very few actually do that by default. Of course, you do need the ArcaOS kernel patches to be able to use it successfully, so it would be silly for programs to automatically use high shared memory since not all users have upgraded.

You're confusing loading the DLL's high with allocating memory in the high arena, which seems to work with older kernels. We've been doing this with the Mozilla apps for years, both with the OS/2 API and by building with -Zmem-high. You are right that the builder does need to set the -Zmem-high (and include os2safe.h before os2.h) flag to use high memory. I think the latest libcx might automatically try to use high memory now.
Quote
I wonder if it would be possible in theory to create a 'memory defragmentation' daemon that monitors memory and warns when fragments get below a specified size (or even better, defrags when needed)?

Of course, it is "possible". The problem would be to find somebody who can do it. I doubt if anybody knows enough about how it works, to be able to do anything about it.

Actually I wouldn't be so sure its possible, even with all the kernel etc sources. It's just the way C works and defragging memory is hard with C. Other languages such as JAVA I believe can do it in their own space. Even when implemented, there is overhead. The old Apple II used to grind to a halt now and again as the Basic interpreter did garbage collection, though eventually it was optimized to be much better.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: PMSHELL
« Reply #10 on: August 02, 2018, 03:10:09 am »
...The real killer is the size of the large DLL's that get loaded this can fixed by using a tool to mark the code segments to load high. Do this for Firefox DLL's, THunderbird AND Open Office.  Do not mark the datasegments high of the DLL...

Help me out with this particular concept from the statement above: so what problem is caused by attempting to load BOTH code and data segments?

I always thought that if the application is written in such a way as to support this mode of operation, it would be the best possible scenario to load both to upper memory.

Is that a wrong assumption?

I just checked my OpenOffice413 install, all DLLs are marked as BOTH. What that means on my machine (VIRTUALADDRESSLIMIT=3072) is that even when my total RAM drops to 1.5Gig (PU Monitor) and shmemon shows only 80.5M left I am still able to open up any document OO I wish. Granted, I did not try any other highmem combos, so I can not tell if there is a difference there or not?

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: PMSHELL
« Reply #11 on: August 02, 2018, 06:52:18 am »
...The real killer is the size of the large DLL's that get loaded this can fixed by using a tool to mark the code segments to load high. Do this for Firefox DLL's, THunderbird AND Open Office.  Do not mark the datasegments high of the DLL...

Help me out with this particular concept from the statement above: so what problem is caused by attempting to load BOTH code and data segments?

Older kernels (and perhaps the current one though it has fixes for this) had problems when loading both high. With Mozilla, I've found sometimes it works fine and sometimes introduces instability, especially when closing the app. Best to experiment but I usually advice to load the code high as it is safer.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: PMSHELL
« Reply #12 on: August 02, 2018, 11:02:04 am »
Whenever you have data in a data segment that is marked to load high you will get in deep shit trouble if that data needs to be passed for access by a 16-bit DLL (either directly or indirectly via intermediate DLLs).
That's because the standard way to thunk a 32-bit address to a 16-bit address will miserably fail for data in the high memory area but thunking is necessary so that 16-bit code can access data in a 32-bit data segment.

It should be obvious that it is almost impossible to know if data will eventually be accessed by 16-bit code and therefore it is a bad idea in general to mark data segments to load high.

Marking code segments to load in high memory is less risky. Only if some 16-bit code needs the address for a piece of code loaded in high memory this will miserably fail. But this happens not very often, compared to the "data case".

And then there is dynamically allocated (data) memory, that is, (data) memory allocated by an application during the runtime of the application. Here you can only rely on the programmer to make the correct decisions as you cannot influence this as a user.
It is the programmer's job to investigate if that memory is going to be used by 16-bit code or not. If yes, there are also means to generate a memory alias in low memory so that the address can be properly thunked for use by 16-bit code. But again, it's the programmers' job to do it right.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: PMSHELL
« Reply #13 on: August 02, 2018, 06:41:19 pm »
Has it something to do with PMSHELL ?

When I open bigicons from ArcaOS and did a fast slide down and up, following by a close, 30MB are used in the shared storage. After folder close, only 1Mo is freed.... Do it again and a new 30MB are used in shared strage and 1MB released after close...
I did test with other applications and the amount of taken shared storage is release at end of session...
Strange...
If some of you could test to check if this happens for others too...

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: PMSHELL
« Reply #14 on: August 03, 2018, 09:16:23 am »
That much sounds like a memory leak either in some WPS class (which is likely, given its lousy stability) or even in the memory subsystem (which is rather unlikely, compared to option 1).

But this memory leakage problem is not related to the question of low/high memory arena. It is likely a problem of dynamically allocated memory that is not properly freed when no longer in use. Since every WPS class remains loaded for as long as PMSHELL is executing, that memory will be leaked until you shut down the system or do a WPS restart.

As a test: if you observe this leakage, just do a WPS restart (via XWorkplace/eWorkplace) and see if that brings back the lost memory.