Author Topic: Memory 'footprint' API?  (Read 4549 times)

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Memory 'footprint' API?
« on: April 29, 2020, 07:12:26 pm »
In short, how do I take a snapshot of what's currently loaded?

I'd like to cover both the low and high areas.

What I am not looking for is a summary level stuff. Instead I want to see DLL/object level stuff, where it is sitting and so forth.

I suppose the closest I can describe it to is what the Theseus => Linear Memory Map => Show Process Contexts provides, except that I want to struture a memory map which shows each process currently running and which DLLs it links to and where these DLLs reside in memory.

Is there a starting point in terms of OS/2 APIs I should look at?

ivan

  • Hero Member
  • *****
  • Posts: 1558
  • Karma: +17/-0
    • View Profile
Re: Memory 'footprint' API?
« Reply #1 on: April 29, 2020, 07:30:23 pm »
Hi Dariusz,

I don't know if the attached will give what you want but they might give you ideas.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: Memory 'footprint' API?
« Reply #2 on: April 29, 2020, 11:00:14 pm »
Hi Ivan,

Actually, I got that little util, pretty handy, although at some point in time it started crashing on several of the tabs, so things like : Modules, Mod Tree, Sharedmem cause a SYS3175 now.

More importantly, I would like to develop the code needed to provide the memory snapshot. Part of this is driven by my desire to better understand the Shared Memory situation. I see the darn thing dropping slowly, at times I see applications which will consume large chunks that never appear to be given back. This is a known issue, which I wish to understand better.

ivan

  • Hero Member
  • *****
  • Posts: 1558
  • Karma: +17/-0
    • View Profile
Re: Memory 'footprint' API?
« Reply #3 on: April 29, 2020, 11:34:15 pm »
Quote
Actually, I got that little util, pretty handy, although at some point in time it started crashing on several of the tabs, so things like : Modules, Mod Tree, Sharedmem cause a SYS3175 now.

Very strange, it works well on my Ryzen boxes.  It sounds as if you have something strange going on with your computer - it is almost as if you have a suspect memory stick, either speed or a chip going bad (I have seen memory problems when the memory speed isn't optimum for the processor and chipset).

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4788
  • Karma: +99/-1
    • View Profile
Re: Memory 'footprint' API?
« Reply #4 on: April 29, 2020, 11:59:53 pm »
In short, how do I take a snapshot of what's currently loaded?

I'd like to cover both the low and high areas.

What I am not looking for is a summary level stuff. Instead I want to see DLL/object level stuff, where it is sitting and so forth.

I suppose the closest I can describe it to is what the Theseus => Linear Memory Map => Show Process Contexts provides, except that I want to struture a memory map which shows each process currently running and which DLLs it links to and where these DLLs reside in memory.

Is there a starting point in terms of OS/2 APIs I should look at?

Theseus is the best we have. For shared DLLs, high light a process, doesn't matter much which, and under process, click on Shared Object Summary, which will tell you which and where DLLs are loaded. Bottom of the list above address 1FFF0000 is the high memory arena. There's also the private memory summary and various other stuff under process such as General Process Information which will tell you what files a program has opened, where the DLLs a process is using is listed.
If you look under help, there's the various programming interfaces, C and Rexx if you want to write a better program using the Theseus Data.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: Memory 'footprint' API?
« Reply #5 on: April 30, 2020, 12:24:38 am »
Hi Ivan,

Actually, I got that little util, pretty handy, although at some point in time it started crashing on several of the tabs, so things like : Modules, Mod Tree, Sharedmem cause a SYS3175 now.

More importantly, I would like to develop the code needed to provide the memory snapshot. Part of this is driven by my desire to better understand the Shared Memory situation. I see the darn thing dropping slowly, at times I see applications which will consume large chunks that never appear to be given back. This is a known issue, which I wish to understand better.

Just use theseus4, so I did following dave's suggestion
Then look on the shared memory and specially the high shr storage

How to use theseus4, very easy as it was suggested to me by Dave.

> Open Theseus, highlight a program in the list on the right, doesn't
> matter too much which one, perhaps THESEUS4. Then click Process-->Shared
> Object Summary which will bring up another window. Scroll down to close
> to the bottom where the addresses (first column) are over 1FFF0000. You
> can then see which DLLs are loaded high, how much memory they're using
> etc.

To capture the problem, start/stop the program which decreases the high shr storage each time.
Once close, run theseus and save the output in a file.
Then, start the program and do not stop it, refresh theseus and save the output in a new file. Then, compare both output.

The problem happens because the program loads module into high shr storage and at close time, it is freed (and it is) but at new run, the freed space isn't taken again but new address range is taken (the previous freed address range are no more usable - the pb).

It only occurs with high shr storage as I could see. To bypass this problem, modules have to be permanently loaded but as soon you decide to free them, next load will do the same as does the program at each start.
e.g.  ffmpeg 4.0.2 with DLLs flag to go above 512MB line may consume about 21MB in shr at each run. With the dlls loaded permanently, 21MB are used one time (as it is planned) and no more.
With my ffmpegHigh tool, you run it ones and don't close the prompt window used to run the tool to have the benefit of permanent loaded DLLs. With the existing OS/2 problem, each time you have to load DLLs which where freed, it takes again the dlls size into the memory (storage decreases than).

Here is an example of the storage with some modules loaded into high shr:

8DA60000  00108000  00000000  00000000  00000000  X264150  #0001 (shared code)
8DB68000  00008000                                 Free
8DB70000  001F2000  00000000  00000000  00000000  LIBVPX5  #0001 (shared code)
8DD62000  0000E000                                 Free
8DD70000  000A3000  00000000  00000000  00000000  SDL2     #0002 (shared code)
8DE13000  0000D000                                 Free
8DE20000  00019000  00000000  00000000  00000000  POSTPR55 #0001 (shared code)
8DE39000  00007000                                 Free
8DE40000  00004000  00000000  00000000  00000000  AVDEVI58 #0001 (shared code)
8DE44000  0000C000                                 Free
8DE50000  0001A000  00000000  00000000  00000000  SWRESA3  #0001 (shared code)
8DE6A000  00006000                                 Free
8DE70000  0007E000  00000000  00000000  00000000  SWSCAL5  #0001 (shared code)
8DEEE000  00002000                                 Free
8DEF0000  00216000  00000000  00000000  00000000  AVFILT7  #0001 (shared code)
8E106000  0000A000                                 Free
8E110000  00BD2000  00000000  00000000  00000000  AVCODE58 #0001 (shared code)
  Object Allocated Committed   Present   Swapped
 address    memory    memory    memory    memory  Description
8ECE2000  0000E000                                 Free
8ECF0000  00219000  00000000  00000000  00000000  AVFORM58 #0001 (shared code)
8EF09000  00007000                                 Free
8EF10000  00070000  00000000  00000000  00000000  AVUTIL56 #0001 (shared code)

At higher address the storage module size is seen and flagged as free but no more used :

8EF80000  00108000  00000000  00000000  00000000
8F088000  00008000                                 Free
8F090000  001F2000  00000000  00000000  00000000
8F282000  0000E000                                 Free
8F290000  000A3000  00000000  00000000  00000000
8F333000  0000D000                                 Free
8F340000  00019000  00000000  00000000  00000000
8F359000  00007000                                 Free
8F360000  00004000  00000000  00000000  00000000
8F364000  0000C000                                 Free
8F370000  0001A000  00000000  00000000  00000000
8F38A000  00006000                                 Free
8F390000  0007E000  00000000  00000000  00000000
8F40E000  00002000                                 Free
8F410000  00216000  00000000  00000000  00000000
8F626000  0000A000                                 Free
8F630000  00BD2000  00000000  00000000  00000000
  Object Allocated Committed   Present   Swapped
 address    memory    memory    memory    memory  Description
90202000  0000E000                                 Free
90210000  00219000  00000000  00000000  00000000
90429000  00007000                                 Free
90430000  00070000  00000000  00000000  00000000
   
I put 2 "size in storage" entries into bold for illustration.

This is why I wrote ffmpegHigh to not have these all free "holes" left in the storage resulting into a big storage decrease.
But, to have the DLLs permanent (the bypass to prevent "holes") the prompt window used to run ffmpegHigh must not be closed (just minimize it like it is done for ffturbo).

Hope this may help you.
Best regards

Updt: It looks I wrote at the same time as Dave :D
« Last Edit: April 30, 2020, 12:28:11 am by Remy »

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile