OS2 World Community Forum
OS/2, eCS & ArcaOS - Technical => Programming => Topic started by: Dariusz Piatkowski on September 26, 2023, 10:08:32 pm
-
I'm making some adjustments to the PUMonitor utility to show the largest available shared memory area (low shared memory I believe) alongside the current 'available memory' estimate.
Specifically, the PUMonitor utility relies on win32k.h definitions and library calls:
/* $Id: win32k.h,v 1.11 2001/03/15 20:03:10 bird Exp $
*
* Top level make file for the Win32k library.
* Contains library and 32-bit IOCtl definition.
*
* Copyright (c) 2000 knut st. osmundsen (knut.stange.osmundsen@pmsc.no)
...
That library has a K32SYSTEMMEMINFO record which appears to already collect the shared memory information:
...
/* Virtual Memory manager info. */
ULONG ulAddressLimit; /* VM: Current user virtual address limit - use this for high arena check. (VirtualAddressLimit / 0x20000000) */
ULONG ulVMArenaPrivMax; /* VM: Current highest address in the private arena. (vmRecalcShrBound()) */
ULONG ulVMArenaSharedMin; /* VM: Current lowest address in the shared arena. (ahvmShr) */
ULONG ulVMArenaSharedMax; /* VM: Current highest address in the shared arena. (ahvmShr) */
ULONG ulVMArenaSystemMin; /* VM: Current lowest address in the system arena. (ahvmhSys) */
ULONG ulVMArenaSystemMax; /* VM: Current highest address in the system arena. (ahvmhSys) */
ULONG ulVMArenaHighPrivMax; /* VM: Current highest address in the high private arena - aurora/smp only. (vmRecalcShrBound) */
ULONG ulVMArenaHighSharedMin; /* VM: Current lowest address in the high shared arena - aurora/smp only. (ahvmhShr) */
ULONG ulVMArenaHighSharedMax; /* VM: Current highest address in the high shared arena - aurora/smp only. (ahvmhShr) */
...
Just looking at that single header file I see the following functions being called out:
/*******************************************************************************
* External Functions *
*******************************************************************************/
/* Win32k APIs */
APIRET APIENTRY libWin32kInit(void);
APIRET APIENTRY libWin32kTerm(void);
BOOL APIENTRY libWin32kInstalled(void);
APIRET APIENTRY libWin32kQueryOptionsStatus(PK32OPTIONS pOptions, PK32STATUS pStatus);
APIRET APIENTRY libWin32kSetOptions(PK32OPTIONS pOptions);
/* "Extra OS2 APIs" */
APIRET APIENTRY DosAllocMemEx(PPVOID ppv, ULONG cb, ULONG flag);
APIRET APIENTRY W32kQueryOTEs(HMODULE hMTE, PQOTEBUFFER pQOte, ULONG cbQOte);
APIRET APIENTRY W32kQuerySystemMemInfo(PK32SYSTEMMEMINFO pMemInfo);
APIRET APIENTRY W32kProcessReadWrite(PID pid, ULONG cb, PVOID pvSource, PVOID pvTarget, BOOL fRead);
APIRET APIENTRY W32kHandleSystemEvent(ULONG ulEvent, HEV hev, BOOL fHandle);
/* Helper function */
USHORT APIENTRY libHelperGetCS(void);
...
In particular PUMonitor uses the W32kQuerySystemMemInfo function to retrieve the existing physical available memory information (cbPhysAvail), and it would seem the following would provide what I'm looking for:
ULONG ulVMArenaSharedMin; /* VM: Current lowest address in the shared arena. (ahvmShr) */
ULONG ulVMArenaSharedMax; /* VM: Current highest address in the shared arena. (ahvmShr) */
However...I haven't been able to find any references and DEV explanations for these, so I'm curious if anyone has used these?
The DosQuerySysInfo is a known way to get this, but obviously I'd rather not re-invent the wheel if I can use the existing approach.
As always, appreciate the feedback!
-
Wouldn't using win2k.h introduce a dependency in win2k.sys, which is now depreciated IIRC.
-
Hi Dave,
Wouldn't using win2k.h introduce a dependency in win2k.sys, which is now depreciated IIRC.
Somehow this must not be the case: I've had PUMonitor running on my machine for quite a few years now...I did actually make a few small changes to deal with my multi-core box back in 2020 and I do NOT run the win32k.sys driver on my machine.
Having said that, I do have the Odin runtime installed, so if anything I suspect that the LIBs included with the PUMonitor source tree are in fact making use of the ODIN DLLs...the only problem here is that 'DLL Tree' doesn't actually show anything like that.
-
Wouldn't using win2k.h introduce a dependency in win2k.sys, which is now depreciated IIRC.
Somehow this must not be the case: I've had PUMonitor running on my machine for quite a few years now. [...] and I do NOT run the win32k.sys driver on my machine.
XCenter's "Sentinel Memory Watcher" also attempts to use win32k. If it fails to open the driver, then it falls back to conventional means to generate it numbers. This 'PUMonitor' (which I'd never heard of before) almost certainly does the same. Without a device driver (or access to a backdoor into the kernel), the kernel data the util wants to use is simply not accessible to a Ring-3 app.
-
Dariusz, thanks for starting this thread. It motivated me to resurrect a long-stalled project: a memory meter that shows linear address usage the way OS/2 sees it.
All the mem monitors I've seen divide memory into Private and Shared. However, that "Shared" memory is actually two separate pieces: actual Shared memory, and the *Unassigned* memory that lies between private and shared. This address space can be assigned to either as needed. Because existing monitors lump the two together, they will claim that your *shared* memory just decreased if any of that Unassigned memory gets reassigned for use as Private. This scenario happens constantly with the browsers and gives a mistaken impression of how they use all that memory.
Attached is 'memry.exe', a text-mode memory monitor that addresses the issue. It's a proof-of-concept that I'll eventually reimplement as an XCenter widget. Here's what it's output looks like:
| Private | Unassigned | Shared [Used Free] |
| 150 | 207 | 155 [ 155 0 ] |
There are two display modes:
* static - enter the 'memry' command on the top line of an OS/2 window. The data line will update silently every 2.5 seconds.
* scrolling - enter the 'memry' command on the last line of an OS/2 window. The data line will update every 2.5 seconds and scroll down one line, giving you a listing of recent memory usage.
Edit: see the next posting for a newer version
-
Attached is an update to 'memry' that now displays both high and low memory usage. Start it before opening your browser, then watch its numbers change when you open it. Here's my before and after with Dooble.
-
It may be appropriate to point out my memory-reporting tool:
https://github.com/altsan/os2-mem
which also breaks down memory regions when the /v switch is used, although it lacks the ability to identify "unassigned" like Rich's.
One feature it does have, on ArcaOS systems (and possibly also on systems with QSINIT) is the ability to report the full amount of physical RAM, and any PAE-exclusive region available for use as a memory disk, even if OS/2 can't access it as system RAM.
(It's installed by default as mem.exe on all ArcaOS systems.)
-
Hey Rich!
Attached is an update to 'memry' that now displays both high and low memory usage. Start it before opening your browser, then watch its numbers change when you open it. Here's my before and after with Dooble.
This is pretty neat stuff. I have been dreaming of building a GUI "browser" that basically visually maps the current memory utilization, goal being: make it human digestable!!!
So while tools like Theseus for example will expose a whole bunch of stuff, I often times wish that I could just see a quick snapshot that "lays it out" on the screen. Now, not knowing enough about this subject that may very well be a naive way of looking at it, but worth a try.
Do you mind sharing your source? I am very interested in the APIs you are using there, especially in light of the addition I'm trying to make to PUMonitor (to show the SHARED memory use).
In fact, running your monitor and comparing it's output to the likes of 'Free Shared Mem' util, or Alex's 'mem', I am not quite seeing the same numbers.
For example:
1) mem output
Total physical memory: 8,191 MB
Accessible to system: 3,199 MB
Additional (PAE) memory: 4,992 MB
Resident memory: 1,184 MB
Available virtual memory: 567 MB
Available process memory:
Private low memory: 192 MB
Private high memory: 1,344 MB
Shared low memory: 84 MB
Shared high memory: 564 MB
2) Free Shared Mem
88,408,064 bytes
3) Rich's memry
[Lo Priv | Unused | Shared Used Free] [Hi Priv | Unused | Shared Used Free]
109 | 84 | 318 291 27 1338 | 565 | 145 145 0
So focusing on SHARED memory:
mem shows: 84 MB
memry shows: Lo Priv Unused as 84
Are these two talking about the same memory area?...because your util Rich is certainly showing different numbers under the "Shared" label and neighter the Lo nor Hi match.
-
Gents!
Wouldn't using win2k.h introduce a dependency in win2k.sys, which is now depreciated IIRC.
Somehow this must not be the case: I've had PUMonitor running on my machine for quite a few years now. [...] and I do NOT run the win32k.sys driver on my machine.
XCenter's "Sentinel Memory Watcher" also attempts to use win32k. If it fails to open the driver, then it falls back to conventional means to generate it numbers. This 'PUMonitor' (which I'd never heard of before) almost certainly does the same. Without a device driver (or access to a backdoor into the kernel), the kernel data the util wants to use is simply not accessible to a Ring-3 app.
Rich is spot on here, as was Dave initially as well.
I went back to the code and sure enough, what I missed was a call to check the availability of the win32k stuff (libWin32kInstalled) and when not present the usage of Dos16MemAvail API instead.
Now, having said that, while I can find some on-line references to Dos16MemAvail, I cannot for the life of me find actual documentation for this. Is that some kind of a macro maybe?
Instead I find a lot of references to DosQuerySysInfo, where I could use:
QSV_TOTAVAILMEM - max num of bytes that can be allocated by a process in the system
QSV_MAXPRMEM - max num of bytes that can be allocated by a process in the PRIVATE arena
QSV_MAXSHMEM - max num of bytes that can be allocated by a process in the SHARED arena
So it would seem that perhaps I need to replace that single Dos16MemAvail call with a DosQuerySysInfo call using the three params listed above to get all the info I need?
Rich,
Any chance that's what you're using in your memry util?
-
running your monitor and comparing it's output to the likes of 'Free Shared Mem' util, or Alex's 'mem', I am not quite seeing the same numbers.
First off, my apologies. A dumb arithmetic error gave you 512mb more high memory than you actually have. The attached 'memry07.zip' fixes the error.
This is pretty neat stuff. I have been dreaming of building a GUI "browser" that basically visually maps the current memory utilization, goal being: make it human digestable!!!
That's been my goal as well which is why I've spent the last few days working on my "memap" XCenter widget. Below are some screenshots: the widget at its default size (nice and small), the widget expanded for greater visibility, and the widget showing its details window. Since you may want to monitor the details as you open and close stuff, I've made the details window movable so you move it elsewhere to keep it from getting covered. Note that this is a work-in-progress and subject to change (in particular, the details window).
BTW... the scaling for high and low memory differs. For low memory (bottom line) each box represents 64k. For high mem (top line) each box is total_high_mem / 8. Because I have 2gb of high mem, each box is 256mb; if you have 1gb, each box would be 128k.
Edit: added a screenshot of an alternate layout for the Details window. Which one is better?
-
I went back to the code and sure enough, what I missed was a call to check the availability of the win32k stuff (libWin32kInstalled) and when not present the usage of Dos16MemAvail API instead.
Now, having said that, while I can find some on-line references to Dos16MemAvail, I cannot for the life of me find actual documentation for this. Is that some kind of a macro maybe?
Some background at https://svn.netlabs.org/repos/xworkplace/branches/branch-1-0/001/xwphelp2/xcenter/xc_mem.html , which says:
"Internally, this widget uses the Dos16MemAvail API, an old 16-bit API that is no longer documented by IBM.
The documentation for OS/2 1.3 says that this reports "the size of the largest block of free memory". This is obviously not true any more, since all 32-bit OS/2 versions (since 2.0) no longer allocate memory in blocks, but with 4 KB page granularity instead.
From my experience, this API now does indeed show the amount of free physical memory in the system, that is, the amount of RAM that is currently not used by OS/2, either because it has never been used or swapped out or released again by a process. (From my testing, this API returns the same values as Theseus does for the free physical RAM.) This free RAM can be used by applications immediately without &os2; having to make room by swapping out other memory pages."
More info can be found at http://www.edm2.com/index.php/DosMemAvail which matches the information in the OS/2 1.3 "Control Program Programming Reference".
-
running your monitor and comparing it's output to the likes of 'Free Shared Mem' util, or Alex's 'mem', I am not quite seeing the same numbers.
First off, my apologies. A dumb arithmetic error gave you 512mb more high memory than you actually have. The attached 'memry07.zip' fixes the error.
This is pretty neat stuff. I have been dreaming of building a GUI "browser" that basically visually maps the current memory utilization, goal being: make it human digestable!!!
That's been my goal as well which is why I've spent the last few days working on my "memap" XCenter widget. Below are some screenshots: the widget at its default size (nice and small), the widget expanded for greater visibility, and the widget showing its details window. Since you may want to monitor the details as you open and close stuff, I've made the details window movable so you move it elsewhere to keep it from getting covered. Note that this is a work-in-progress and subject to change (in particular, the details window).
BTW... the scaling for high and low memory differs. For low memory (bottom line) each box represents 64k. For high mem (top line) each box is total_high_mem / 8. Because I have 2gb of high mem, each box is 256mb; if you have 1gb, each box would be 128k.
Edit: added a screenshot of an alternate layout for the Details window. Which one is better?
From a graphical point of view I would think this would be more obvious:
Private: Low High
Used: xxx Bytes xxx Bytes
Unused: xxx Bytes xxx Bytes
Shared:
Unused: xxx Bytes xxx Bytes
Used: xxx Bytes xxx Bytes
because that is effectively how the memory layout looks like.
-
Here are screenshots of the final(?) product.
'mmap-v085.png' shows the widget and its details window which you can toggle on/off by clicking the widget.
'mmap-v085-props.png' shows the Properties dialog which offers "live" updates (i.e. drop a color, the graph changes immediately). It also demonstrates that you can move the window elsewhere if needed - doing so will display an 'X' button to make closing it easier.
This will probably show up in the XWP that follows the soon-to-be-released v1.0.16.
-
Tom, everyone...
...while I can find some on-line references to Dos16MemAvail, I cannot for the life of me find actual documentation for this. Is that some kind of a macro maybe?
Some background at https://svn.netlabs.org/repos/xworkplace/branches/branch-1-0/001/xwphelp2/xcenter/xc_mem.html , which says:
"Internally, this widget uses the Dos16MemAvail API, an old 16-bit API that is no longer documented by IBM.
The documentation for OS/2 1.3 says that this reports "the size of the largest block of free memory". This is obviously not true any more, since all 32-bit OS/2 versions (since 2.0) no longer allocate memory in blocks, but with 4 KB page granularity instead.
From my experience, this API now does indeed show the amount of free physical memory in the system, that is, the amount of RAM that is currently not used by OS/2, either because it has never been used or swapped out or released again by a process. (From my testing, this API returns the same values as Theseus does for the free physical RAM.) This free RAM can be used by applications immediately without &os2; having to make room by swapping out other memory pages."...
Alright...so I've got the makefile fully rebuilt now to use NMAKE32 (just b/c that's what I wanted to use, and I needed a good lesson in makefile functionality), and so off to the "races" I went modifying the code to show the available shared memory.
However...while that seems to work just fine using DosQuerySysInfo and QSV_MAXSHMEM (the results between PUMon and other utils all match), I cannot for the life of me get a matching result when it comes to what Dos16MemAvail API produces.
Here is an example:
Dos16MemAvail shows me 544.8M
...and in the meantime the Theseus snapshot gives:
17. QSV_TOTPHYSMEM = -940527616 (3275820K -> 3199.043M).
18. QSV_TOTRESMEM = 1241985024 (1212876K -> 1184.449M).
19. QSV_TOTAVAILMEM = 634712064 (619836K -> 605.309M).
20. QSV_MAXPRMEM = 142082048 (138752K -> 135.500M).
21. QSV_MAXSHMEM = 74907648 (73152K -> 71.438M).
27. QSV_MAXHPRMEM = 1409286144 (1376256K -> 1344.000M).
28. QSV_MAXHSHMEM = 429481984 (419416K -> 409.586M).
...none of which match the 544.8M that Dos16MemAvail shows.
Now you might say "..oh c'mon man, that's an obsolete API, we have better stuff, like DosQuerySysInfo...", sure, yet strangely enough 'Theseus=>Ram Usage by Process' still returns the correct 544.8M result that the old Dos16MemAvail API does.
What da???!!!
It's bad enough just trying to keep up with the logic flow of PUMon (although it does some nice & sophisticated work with the display cell definition logic), but I need some help in deciphering the above difference.
For what it's worth: I do NOT mind tossing in the gloves and settling in on using one of the most appropriate QSV values, but which one???
1) QSV_TOTAVAILMEM - seems like the best overall fit since it accounts for the virtual memory pool room that can be used to serve ALL processes
2) QSV_MAXPRMEM - seems like more appropriate fit since that really defines the most room a single process actually has left in the low private arena
In other words, while #1 is an overall SYSTEM-WIDE picture, it is really #2 that will dictate whether any particular process you wish to run can successfully allocate all the memory it needs (I believe this is ultimately due to memory segmentation that eventually happens).
Heck...I need to make the matching RC changes anyways (so these are all user selectable), so maybe that's a moot point...???
Thanks for all the feedback!
-
1) QSV_TOTAVAILMEM
Apples...
2) QSV_MAXPRMEM
Oranges...
You are conflating two separate concepts: 'physical/virtual memory' and 'linear address space'. Here's a simplistic analogy.
You have a store with inventory (RAM) and shelving to put it on (address space). Periodically, you'll face one of two problems:
A) you want to add more shelf space but you have nothing to put on it because your entire inventory is already on the other shelves. You could put empty boxes on the shelves (i.e. swap space) to make it look like you have more inventory than you really do but eventually you'll run out of that too. In the real world, you're out of 'physical/virtual memory'.
B) you have plenty of inventory but can't put it out because the shelves where it should go are already full. In the real world, you're out of 'linear address space'.
These days 'A' isn't the problem it used to be - most of us have far more than we can use. And for many of us, we can't even find a use for all the physical RAM we _could_ use, so we never run out and *never swap*. Those of us who know their needs are modest (don't build QT6, don't keep 40-50 browser tabs open, etc.) have no reason to give physical memory a second thought.
In contrast, 'B' (linear address space) will always be an issue because OS/2 will always be limited to 512mb of low/16-bit compatible memory. It isn't as bad as it used to be with the advent of high memory (i.e linear address space whose addresses start at 512mb). Still, it's worth monitoring - which is why I wrote a little widget to do so (shameless plug).
With the possible exception of PUMon when it uses 'Win32K.sys' (which I've never seen), every monitor that attempts to show available private and shared memory suffers from the same problem: GIGO. The values returned for QSV_MAXPRMEM and QSV_MAXSHMEM are misleading and substantially useless because the same stretch of memory is included in both. To illustrate this, I've copied the screen shot of my 'memmap' widget from an earlier posting. Take a look at it.
QSV_MAXPRMEM includes both the areas in red and those in gray. QSV_MAXSHMEM includes the areas in gray and in green. What good is that? I labelled the area in gray as "Free" due to space constraints but it's more correctly labelled "Unassigned". It's available for use as either private or shared memory, whichever claims it first.
When I took the screenshot, Dooble had been open and had claimed 46mb of the unassigned memory for private use (in addition to the 64mb of private memory every app is guaranteed). Even after closing Dooble, that 46mb would remain assigned to "private" until the system was forced to reclaim it. If you were to look at a typical mem monitor, you'd get the mistaken impression that Dooble had allocated but not freed 46mb of shared memory. But that wouldn't be correct because that memory was _never_ shared, it was "unassigned".
FYI... "real" shared memory starts at the lowest address that's actually being used. If that allocation is freed, it goes back into the unassigned pool. Any memory above that which gets freed is stuck behind the dividing line and will remain shared memory - whence the figure for "Shared - unused".
FYI2... unassigned memory claimed for private use, then freed, is not reclaimed until the system has run out of shared and unassigned mem. It will then survey all running apps to see how much private memory they're actually using. If <= 64mb, it will return the excess to the pool, then use what it needs to fill the request for shared mem that triggered the review.
Correction: high mem starts at 512mb, not 1gb as originally written
-
Hi Rich,
Thanks for this detailed explanation on a complex subject - very informative. Was wondering if your widget made it into xwps 1.0.16? I'd like to give it a try...
Regards,
-
Was wondering if your widget made it into xwps 1.0.16?
No, it wasn't ready so it will probably show up in v1.0.17 which I hope to have out sooner rather than later.
-
OK, thanks... looking forward to it.
Regards,