Virtual memory, a curious phenomenon

[roberto]

There is a strange result in virtual memory, depending on where the Swapper position is defined in the config.
In my case if I have the config.sys:
SWAPPATH=C:\OS2\SYSTEM 0 4096
after start the system:
[C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:     514 MB   <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

 But if I create a new unit in 512MB memory,with Ramdisk, unit L: \, HPFS type, and define in the config.sys as:
 SWAPPATH=L:\ 0 4096
 [C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

It seems that the 512MB is more important, than anything else, so that it occurs, that change.
On the other hand I do not see any change in the rest of the operation of the system, but it took a short time of evidence.

Saludos

[ivan]

I can never understand your setup.  I have ArcaOS v5.0.7 as a standard setup with c a couple of exceptions, the boot drive C: is hpfs and the swap file is on D:  I get

[C:\]mem /v

Total physical memory:      7,906 MB
Accessible to system:       3,311 MB
Additional (PAE) memory:    4,595 MB

Resident memory:              193 MB
Available virtual memory:   2,590 MB

Available process memory:
  Private low memory:         213 MB
  Private high memory:      2,240 MB
  Shared low memory:          149 MB
  Shared high memory:       2,160 MB

[C:\]

  SWAPPATH=D:\ 2048 2048

My test setup on my Ryzen third generation processor is very much the same.

[roberto]

Hello Ivan
You already have half of the test done.

Now you run this program:
C:\sys\bin\HIDISK.EXE
Define 512MB, HPFS, and Format US HPFS
Name unit X:
Change your config.sys
rem SWAPPATH=D:\ 2048 2048
SWAPPATH=X:\ 2048 2048
Restart the system
and
run mem /v
saludos

[Dariusz Piatkowski]

...
Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB
...

The 'Available virtual memory' you're seeing is what I have here on a 8G = 4G + 4G RAMDISK.

What I think is uber strange is that 'Shared high memory' reading!!! wwwwhhhhaaattt? isn't that an OS/2 dream for everyone...seriously, the LOW/SEGMENTED shared memory area is our weak point!!!

[Dave Yeo]

...
Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB
...

The 'Available virtual memory' you're seeing is what I have here on a 8G = 4G + 4G RAMDISK.

What I think is uber strange is that 'Shared high memory' reading!!! wwwwhhhhaaattt? isn't that an OS/2 dream for everyone...seriously, the LOW/SEGMENTED shared memory area is our weak point!!!

Hi Dariusz, there's something weird about your mem -v output if you have 8GB installed, your "Total physical memory" should show close to 8GB with "Additional (PAE) memory" showing about 4GB.
Here with 12 GB of ram installed, and a 8GB or so ram disk (JFS)

Code: [Select]

Total physical memory:     12,169 MB
Accessible to system:       3,241 MB
Additional (PAE) memory:    8,928 MB

Resident memory:              153 MB
Available virtual memory:   1,855 MB

Available process memory:
  Private low memory:         294 MB
  Private high memory:      2,240 MB
  Shared low memory:          177 MB
  Shared high memory:         980 MB

As you can see, my "Available virtual memory" does not reflect upper memory. I don't really understand virtual memory excepting it includes ram plus swap file space and it is changable depending on various things like when I built Qt5 and the system was swapping (yes I hit the swap file with 3.2GB of ram visible), it likely dropped to close to zero.
As for Shared high memory, that is where DLL's go when marked to load high, I have Dooble and SeaMonkey open right now so as you see, "Shared high memory" is down to 980 MB. Closing Dooble gives,

Code: [Select]

Total physical memory:     12,169 MB
Accessible to system:       3,241 MB
Additional (PAE) memory:    8,928 MB

Resident memory:              152 MB
Available virtual memory:   1,973 MB

Available process memory:
  Private low memory:         294 MB
  Private high memory:      2,240 MB
  Shared low memory:          193 MB
  Shared high memory:       1,175 MB

More virtual memory now as well as shared low/high memory.

[roberto]

Dariusz- The important thing for me is not to compare a computer with another. The important thing is to compare the results on the same computer.

Dave- After several days trying, the improvement is clear for me. The sensation is that the Swapper was not working well, and now if he is doing it correctly. What I cannot understand, is because it is so important that the amount of 512MB, of space in the Ramdisk, makes the swapper improve. I tried it with 64mb and it didn't work. I don't know what happens with other Ramdisk unit values.
I compare the two results after starting the same computer, and without having started any other application. Except for the applications that are started in the startup, but in both cases they are the same.
Saludos

[Roderick Klein]

There is a strange result in virtual memory, depending on where the Swapper position is defined in the config.
In my case if I have the config.sys:
SWAPPATH=C:\OS2\SYSTEM 0 4096
after start the system:
[C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:     514 MB   <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

 But if I create a new unit in 512MB memory,with Ramdisk, unit L: \, HPFS type, and define in the config.sys as:
 SWAPPATH=L:\ 0 4096
 [C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

It seems that the 512MB is more important, than anything else, so that it occurs, that change.
On the other hand I do not see any change in the rest of the operation of the system, but it took a short time of evidence.

Saludos

Sorry but the location of the swappath in the config.sys has no influence when the swapper gets set up.
This is most likely one of the kernel statements that gets processed very early by the kerne.
DEVICE= statement for example depend on the order in the config which driver is loaded.
I mean the location of the libpath line in the config.sys also does not alter that behaviour.

Roderick

[Rich Walsh]

I just did a lot of testing on this. I found that the value shown for "Available virtual memory" depends on the size of your swap partition and nothing else. If your swap partition is less than at least 4gb, then the value shown is likely to be useless.

Here are the numbers for my normal setup using  "SWAPPATH=N:\ 2048 2048", where N: is the 4.9gb boot drive.

Code: [Select]

Total physical memory:     16,171 MB
Accessible to system:       2,355 MB
Additional (PAE) memory:   13,816 MB

Resident memory:              130 MB
Available virtual memory:   1,215 MB

Available process memory:
  Private low memory:         379 MB
  Private high memory:      1,792 MB
  Shared low memory:          347 MB
  Shared high memory:       1,763 MB

Moving the swappath to a 1gb ramdisk _or_ harddrive gives me:

Code: [Select]

Available virtual memory:   3,157 MB

Increase the partition to 2gb and I get:

Code: [Select]

Available virtual memory:      67 MB

Move the swappath to a 50gb harddrive:

Code: [Select]

Available virtual memory:      2,179 MB

Turn off swapping (MEMMAN=NOSWAP,PROTECT):

Code: [Select]

Available virtual memory:      2,101 MB

The values that you see here are returned by a call to DosQuerySysInfo(QSV_TOTAVAILMEM). I don't think anyone has ever questioned its output before, but it looks like there is good reason to believe it's buggy (or at least _very_ counter-intuitive).

[roberto]

Hello
The virtual memory, from what I see is rotary, goes from 0 to 4GB and 4GB to 0, constantly.
Therefore, it begins with an amount or another can be the same in theory. But while towards the tests
I have seen how with certain values, the keyboard was blocked for several minutes.
The latter as it has been after restarting several times, I don't know if it can be a different problem.

My results are very similar to yours, 1GB gives me more free, 2GB gives me less free.

For a while I will use the 512MB, and see how it works for me.

Saludos

[Dariusz Piatkowski]

Dave, everyone...

Hi Dariusz, there's something weird about your mem -v output if you have 8GB installed, your "Total physical memory" should show close to 8GB with "Additional (PAE) memory" showing about 4GB.
...

Not sure what happened there, other than a few neurons clearly missfiring on my part...grrh...sorry, my post is of course blatantly incorrect. Thank you for pointing that out.

[roberto]

-Roderick
       It is not the same as the swapper put it in different places c:\ c:\os2 \  d: \ etc.
        To the size of the disc where you put it, is different.
        I think that has already been clear in this thread.

-Dariusz
       Do not apologize, the important thing is the interest and curiosity for things,
       keep it up.

Now I am going to tell you a real case, Arcaos 507 new, recently installed in a Lenovo laptop
Ideapad z710 with virtualaddreslimit = 1536 all default

As soon as I start the laptop ,this is memory, I couldn't even start the Seamonkey ¿¿???
[C:\]mem /v

Total physical memory:      8 124 MB
Accessible to system:       2 502 MB
Additional (PAE) memory:    5 622 MB

Resident memory:              134 MB
Available virtual memory:   2 322 MB

Available process memory:
  Private low memory:         257 MB
  Private high memory:        896 MB
  Shared low memory:          166 MB
  Shared high memory:         875 MB

[C:\]

AFTER  ramdisk 512mb and and modify the config.sys changing the swapper to that newly created partition

[C:\]mem /v

Total physical memory:      8 124 MB
Accessible to system:       2 502 MB
Additional (PAE) memory:    5 622 MB

Resident memory:              130 MB
Available virtual memory:   2 331 MB

Available process memory:
  Private low memory:         380 MB  -------257--------
  Private high memory:        896 MB
  Shared low memory:          320 MB  ------166---------
  Shared high memory:         875 MB

Here you can see Swapper's magic in 512MB, memory increases.
[C:\]

AFTER virtualaddress 3072  and restart

[C:\]mem /v

Total physical memory:      8 124 MB
Accessible to system:       2 502 MB
Additional (PAE) memory:    5 622 MB

Resident memory:              130 MB
Available virtual memory:   2 332 MB ----2331-------
Available process memory:
  Private low memory:         380 MB
  Private high memory:      2 240 MB ----896-------
  Shared low memory:          321 MB ----320------
  Shared high memory:       2 219 MB ----875------


If this would be on a computer, it could be just something curious, but at work I had long
With this modification, and it is clearly not the same, in all and seen something very similar.
With an important improvement of functions and performance.
As it is very simple to do it, anyone can try it and will be before and after

Saludos

[Remy]

Here is what I have using my settings  (I changed virtualaddresslimit until I could have max free available memory, other value provides less usable memory)

VIRTUALADDRESSLIMIT=2944
SWAPPATH=M:\ 131072 524288
M is my virtual DRIVE using all PAE memory                 

Total physical memory:      16262 MB
Accessible to system:        3470 MB
Additional (PAE) memory:    12792 MB

Resident memory:              215 MB
Available virtual memory:    2412 MB

Available process memory:
  Private low memory:         282 MB
  Private high memory:       2128 MB
  Shared low memory:          193 MB
  Shared high memory:        1329 MB


Note: this doesn't prevent me having mouse freeze from time to time but it more seems due WPS with not refreshed screen during the freeze time (suspect png despite I have all png updates installed supposed to correct this kiond of problems (reduces may I say) ! - opening a very bigfolder with png show a dramatic free shared storage decrease and sometimes, when not release at folder close, a WPS restart frees some memory but not all)

[roberto]

-Remy
     
If you have no more available memory, it's because you don't want to.
Or that I have not explained myself well,or....
You do not need to reinstall the entire system, just from a good position.
the procedure is the next:(I understand that I do not indicate the installation of your system must be by default.)
Try to do this and tell me how it run:
SWAPPATH=C:\ 2048 2048 or something similar
Virtualaddrreslimit = 1536
Restart the system
Create a Disk RAM with 512MB HPFS, and PUT the swappath in that newly created unit, swappath=M:\ 2048 2048
Restart Again
Now virtualaddresslimit = 3072
Restart and test.

Saludos
PD: The grass that smokes must be very good, my whiskey changed for your grass,( just kidding )

[Remy]

-Remy
     
If you have no more available memory, it's because you don't want to.
Or that I have not explained myself well,or....
You do not need to reinstall the entire system, just from a good position.
the procedure is the next:(I understand that I do not indicate the installation of your system must be by default.)
Try to do this and tell me how it run:
SWAPPATH=C:\ 2048 2048 or something similar
Virtualaddrreslimit = 1536
Restart the system
Create a Disk RAM with 512MB HPFS, and PUT the swappath in that newly created unit, swappath=M:\ 2048 2048
Restart Again
Now virtualaddresslimit = 3072
Restart and test.

Saludos
PD: The grass that smokes must be very good, my whiskey changed for your grass,( just kidding )

Hi Roberto,

I already tested several settings with all giving me different results like you found too.
The setting I posted is the one providing me the most usable storage and the most impacting parameter was
virtualaddresslimit (the value you suggest decreased usable storage and I found that setting it to 2944 was better than 3072).

Compare my results with virtualaddresslimit=2944 with yours virtualaddresslimit=3072 !
Any usual vitualaddresslimit 1536, 2048, 2560, 3072 .... produces worst results.   
                             
Total physical memory:      16262 MB
Accessible to system:        3470 MB         <<<<<   with 3072, you have 2502 ! (no change compared to 1536 !)

Resident memory:              215 MB           <<<<<   with 3072, you have 130 !  (no change compared to 1536 !) 
Available virtual memory:    2412 MB        <<<<<   with 3072, you have 2332 !  (no change compared to 1536 !)


I agree with you that results are strange like you described which made me search which value allow me to get the most usable storage. Give a try with virtualaddresslimit=2944 instead of 3072 and check mem output.
See mem results I have.

Is the mem output accurate ?

[Dave Yeo]

Is the mem output accurate ?

Not an expert but I assume that it accurately reports what the kernel reports. Which leads to the question of whether the kernel's report is accurate. Judging by the wildly different results, it doesn't seem to be, likely as when it was designed having 4GB's of ram installed was a fantasy and before Warp V4.5 (Warp V4+fp 13) the non-server OS/2 could only access 1GB of address space, half of which was given to the kernel. That is low memory.
We're dealing with different types of memory.
Accessible to the system is how much actual ram is accessible, 4GB minus PCI space and such including video memory. I couldn't even install ArcaOS to this box until I reduced the video ram (onboard graphics). Now it is about 3.4 GB's, I'm not in OS/2 right now so going by memory. My usual box, it is 3.2 GB.
Virtual memory is how much the CPU can access, you can have multiple processes using the same address range like with multiple DOS sessions. IIRC, the virtual limit, in theory, is 2TB. As it allocated in 4KB chunks, there's selectors and such to keep track of it and they'll run out. The Kernel also has to keep track and uses 32 bit variables, sometimes signed, so 2-4GB limit before perhaps wrapping or overflowing and crashing. Virtual memory can live in the swap file as well as ram. Our swap file is limited to 2GB, I've had the system die due to a swap file full condition when it grew past 2GB.
In theory, you could start a bunch of DOS sessions, each with access and committing 1GB of memory and once ram is full, it'll swap. In practice, those 32bit variables limit things.
I think VIRTUALADDRESSLIMIT is how much address space a (32 bit) process can use, so the higher the better, at least until it interferes with kernel (device drivers etc) and PCI space.  My feeling is that IBM didn't quite finish things so we have to manually set it.
Then there is low and high memory, 16 bit programs are limited to 1GB of virtual memory, mapped to the lower 1GB, half of which is claimed by the kernel so 512MB of low memory, shared and private with the shared having a bunch of DLL's loaded before you even launch a program. So basically shared is DLL space.
I think each process gets its own private address space as well but I'm not sure.
Anyways, that's my understanding, much of it might be wrong, maybe someone can correct me but the bottom line is the most important part of the mem output is shared memory, especially low shared memory as that is what we usually run out of first.
It is possible to run out of address space as well. Linking xul.dll (Firefox) for a while would fail without virtualaddresslimit set to 3072, and with 1.5 Gb of real ram, I had to be careful not to run anything else as the swap file would overflow, now some of the Qt DLLs and source code seem to have the same problem and with 3.2GB's of ram accessible to the system, building Qt5, I saw the swap file grow by a few hundred MB's, not sure how big as I wasn't paying attention and it likely shrank before I noticed. Also have to limit the make jobs that are running, 2 instead of 5 on this 4 core system.