Virtual memory, a curious phenomenon

[Remy]

Is the mem output accurate ?

I think VIRTUALADDRESSLIMIT is how much address space a (32 bit) process can use, so the higher the better, at least until it interferes with kernel (device drivers etc) and PCI space.  My feeling is that IBM didn't quite finish things so we have to manually set it.
Then there is low and high memory, 16 bit programs are limited to 1GB of virtual memory, mapped to the lower 1GB, half of which is claimed by the kernel so 512MB of low memory, shared and private with the shared having a bunch of DLL's loaded before you even launch a program. So basically shared is DLL space.
I think each process gets its own private address space as well but I'm not sure.

It looks to be like this, OS/2 was build using ideas out of IBM MVS.

[roberto]

Hi Remy

In my work computer with 2944 virtualaddressl ... A improved, it is modern.
In my personal laptop with 2944, it is worse, it is old but very well built. But comment, in this equipment, I use the screen pre-actualice the video in intermediate memory,
And it consumes me 8Mb. and with 3072 the screen was corrupted, and left the 3072 without pre -actualizing the screen,
or I get down to 3064 and preactive the panatalla. (This better option)

I have two questions, how did you calculate the value 2944?
In the swappath that you had previous 52428/131072 = 4 exact, you had same system to calculate it,
Was it for proof and error?

I understand that programers have to adjust many things, including the low memory, which is very annoying,
When you tell you that you lack memory, and you have a lot unused, of another type.

I guess we will have to start playing with the SWAPPATH.
Saludos

[Dave Yeo]

Hi Roberto,
If you use hex for virtualaddresslimit, you see the proper values end in 00, which is a page boundary.
3072=C00h
2816=B00h
2560=A00h
2304=900h
2048=800h
1792=700h
1536=600h

it is likely important to have a page boundary such as above.

[roberto]

I really don't know what Page Boundary is, but I know what works for me and what doesn't. It seems to me that with 3064 it is going well, but it is soon to say anything, I need more use to verify it.
What I am starting to think is that SwaPPath is a frequency and not an amount, which generates a Swapper file.
I would like to try this:

Code: [Select]

swappath=L:\ 2000 506000

saludos

[roberto]

I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

[Dave Yeo]

I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

I guess it depends on how things are implemented, 2944=B80h, still a fairly round number, remembering that 80h is half of 100h.
Personally, I'd stick with the the recommended values. Could use Theseus to look closer.
Really this is just playing with numbers, the important thing is how much memory can actually be allocated and committed, and when it matters.

[Remy]

I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

I guess it depends on how things are implemented, 2944=B80h, still a fairly round number, remembering that 80h is half of 100h.
Personally, I'd stick with the the recommended values. Could use Theseus to look closer.
Really this is just playing with numbers, the important thing is how much memory can actually be allocated and committed, and when it matters.

Well, just refresh me, a page is 4K, right ?
2944 is a multiple of 4 ( 4 *  736 ) or ( 64 * 46 )
When I use any of suggest values, my usable storage is always dropping to a low value, I did approach using 4k pages and this value was the one providing me with the maximum available storage.

Hi Roberto,
If you use hex for virtualaddresslimit, you see the proper values end in 00, which is a page boundary.
3072=C00h
2816=B00h
2560=A00h
2304=900h
2048=800h
1792=700h
1536=600h

it is likely important to have a page boundary such as above.

Why do you use a MiB converted to hex (MiB value) to determine page boudary ?
I though a page is 4KiB, is it ? 
1536 is MiB or 1572864 KiB  ( 180000H ) - number of 4k pages in hex value
2944 -> 3014656 KiB ( 2E0000H ) - number of 4k pages in hex value

OS/2 is able to use up to near 3.5GiB, could you get a value near this one using your settings ?
I tried several setting in respect of 4k pages and I could see that using 3072, the system rounded the storage to a lower value. I think (should be confirmed) that OS/2 is calculating the value to satisfy boundaries pages and despite the given value, it changes it to satisfy its architecture and the mem output is the value OS/2 determined to be correct according the given limit.

It is correct ?
I'm interested to get more details about how it works here.   
 

[Remy]

Hi Remy

In my work computer with 2944 virtualaddressl ... A improved, it is modern.
In my personal laptop with 2944, it is worse, it is old but very well built. But comment, in this equipment, I use the screen pre-actualice the video in intermediate memory,
And it consumes me 8Mb. and with 3072 the screen was corrupted, and left the 3072 without pre -actualizing the screen,
or I get down to 3064 and preactive the panatalla. (This better option)

I have two questions, how did you calculate the value 2944?
In the swappath that you had previous 52428/131072 = 4 exact, you had same system to calculate it,
Was it for proof and error?

I understand that programers have to adjust many things, including the low memory, which is very annoying,
When you tell you that you lack memory, and you have a lot unused, of another type.

I guess we will have to start playing with the SWAPPATH.
Saludos

SWAPPATH=M:\ 131072 524288

About my swappath values, I set numbers very high because I have a big PAE disk. a high allocated value reduces allocation processes into os/2 when it uses the swap file. The other value is the mini I decided to not go below which gives me time to have an action before "damage"
e.g. 512 * 1024 (1K binary) =>   524288 and  124 * 1024 = 131072   (I think that a good approach about the initial swappath value would be something near the maximum swappath size used - of course, this has more impact when using the swappath on a HD but reducing processes like increase swappath seems to me to be wellcome despite it is into memory)     

[Remy]

The lack of memory I have occurs below 512M and often when handling big and many png image files (WPS uses a lot too !) despite having the latest fixes for png supposed correcting this, the problem is reduced but not resolved. WPS is an other one consuming low storage and not always releasing it (WPS restart will get back parts of the storage)   
Other programs have they dlls loaded into the high storage and I have enough storage there.

[Dave Yeo]

I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

I guess it depends on how things are implemented, 2944=B80h, still a fairly round number, remembering that 80h is half of 100h.
Personally, I'd stick with the the recommended values. Could use Theseus to look closer.
Really this is just playing with numbers, the important thing is how much memory can actually be allocated and committed, and when it matters.

Well, just refresh me, a page is 4K, right ?
2944 is a multiple of 4 ( 4 *  736 ) or ( 64 * 46 )
When I use any of suggest values, my usable storage is always dropping to a low value, I did approach using 4k pages and this value was the one providing me with the maximum available storage.

Hi Roberto,
If you use hex for virtualaddresslimit, you see the proper values end in 00, which is a page boundary.
3072=C00h
2816=B00h
2560=A00h
2304=900h
2048=800h
1792=700h
1536=600h

it is likely important to have a page boundary such as above.

Why do you use a MiB converted to hex (MiB value) to determine page boudary ?
I though a page is 4KiB, is it ? 
1536 is MiB or 1572864 KiB  ( 180000H ) - number of 4k pages in hex value
2944 -> 3014656 KiB ( 2E0000H ) - number of 4k pages in hex value

My use of page was probably not the best, apologies. For the i386, a page is 4KiB, an i686+ can also have 1MiB pages, not that it happens with OS/2.
OS/2 2.x allocates in 4KiB chunks, Warp allocates 64KiB chunks, but might only commit 4KiB, so probably better to use 64KiB's in your calculations.
I used the page boundary converted to hex as computers work in binary which more accurately are represented in hex. For certain things such as fixup tables, round hex numbers are better. I (and most of us) don't know how things are programmed in the kernel when it comes to high memory, so I was guessing that it only works with some numbers such as I listed.
The documentation is lacking, looking now, I can only find mention of VIRTUALADDRESSLIMT in the programming addendum under DosQuerySysInfo(), which is the function that mem uses to get most of its info, it says,

Code: [Select]

30 QSV_VIRTUALADDRESSLIMIT

Size of the user's address space in megabytes (that is, the value of the rounded VIRTUALADDRESSLIMIT)
Not much to go on, besides what others say. I did test your setting of VIRTUALADDRESSLIMIT=2944,
Seems to work, From Theseus, General System Information,"QSV_VIRTUALADDRESSLIMIT  = b8000000"

OS/2 is able to use up to near 3.5GiB, could you get a value near this one using your settings ?
I tried several setting in respect of 4k pages and I could see that using 3072, the system rounded the storage to a lower value. I think (should be confirmed) that OS/2 is calculating the value to satisfy boundaries pages and despite the given value, it changes it to satisfy its architecture and the mem output is the value OS/2 determined to be correct according the given limit.

It is correct ?
I'm interested to get more details about how it works here.   

As I said, I'm not really sure if my understanding is correct. The documentation is clear that the values returned by DosQuerySysInfo() is only advisory,

Code: [Select]

19 QSV_TOTAVAILMEM

Maximum number of bytes of memory that can be allocated by all processes in the system. This number is advisory and is not guaranteed, since system conditions change constantly.

With similar for the others.
I'm really not sure how much some of these numbers such as available virtual memory matter as well as how accurate they are. Changing the size/location of the swap file shouldn't matter, yet it changes what the system reports, weird.
I guess what matters is getting out of memory errors and when it happens.
Building Qt5, I got failures of cpp.exe not being able to allocate enough memory, reboot fixed, I also got  popups about not enough resources to start make (lots of make processes run during the build), also fixed by a reboot, and I also saw my swap file grow, seems I used more memory then the 3.2 GiB accessible to my system.
Unluckily, a lot of the times, the system just locks up when memory gets low. Probably depends on the type of memory as things get funky when shared gets low. Builds do fail in similar ways, locks up and eventually have to reboot to get rid of the zombie process, reducing the number of jobs that make runs usually fixes that issue, as well as having VIRTUALADDRESSLIMIT set to 3072, which at one point wlink would complain and fail due to not enough spill memory (address range) without that setting.
Not an expert and I'm sure I've got some stuff wrong.

[Dave Yeo]

The lack of memory I have occurs below 512M and often when handling big and many png image files (WPS uses a lot too !) despite having the latest fixes for png supposed correcting this, the problem is reduced but not resolved. WPS is an other one consuming low storage and not always releasing it (WPS restart will get back parts of the storage)   
Other programs have they dlls loaded into the high storage and I have enough storage there.

The problem is that 16 bit functions have to be in low memory, and there's a few important ones. Here's a partial list, too lazy to fight with github right now so this is outdated but gives the idea, you can look at the installed os2safe.h. [urlhttp://trac.netlabs.org/libc/browser/branches/libc-0.6/src/emx/include/os2safe.h[/url]
I don't know why png uses so much but the WPS and PMSHELL do use these functions and have to be in low memory. For libc programs, there's forwarders to load high and call low.
Another big issue is memory fragmentation. After a while of committing and releasing memory, it can get fragmented and even though you have 200 MB of free low shared memory, the biggest chunk might be quite a bit smaller so trying to allocate lets say 50 MB when the biggest hole is 40 MB is going to fail.

[Remy]

Hi Dave!

I think it could be helpfull to read how it works into MVS (or just before MVS) because IBM used it's logics (how it should work) to build OS/2. In MVS, pages = 4K (of course, 4k and 64k are the most used values). Each init has an address space witha size which can be above the installed storage and up to the maximum addressable limit ... (very simplified view) 
About pagination, I think there is a expansion streshold and a contraction streshold and It would be interesting to find if these streshold can be tunned or are these streshold hardcoded by architecture !

 

[Remy]

Hi Remy

In my work computer with 2944 virtualaddressl ... A improved, it is modern.
In my personal laptop with 2944, it is worse, it is old but very well built. But comment, in this equipment, I use the screen pre-actualice the video in intermediate memory,
And it consumes me 8Mb. and with 3072 the screen was corrupted, and left the 3072 without pre -actualizing the screen,
or I get down to 3064 and preactive the panatalla. (This better option)

I have two questions, how did you calculate the value 2944?
In the swappath that you had previous 52428/131072 = 4 exact, you had same system to calculate it,
Was it for proof and error?

I understand that programers have to adjust many things, including the low memory, which is very annoying,
When you tell you that you lack memory, and you have a lot unused, of another type.

I guess we will have to start playing with the SWAPPATH.
Saludos

Hi roberto,

The value I use can be divided by 4 and 64 while yours only by 4 ! (I would not use)
May be you could try 3072, 3008, 2944, 2880 ...  (values I tried in the past) 

[roberto]

 REMY
Thanks for the virtualaddresslimit, proposed as possible.
I have only tried 3008, for my screen problem. And it doesn't work, the screen is corrupted.
In principle, the rest, and stay with those of a lifetime, proposed by Dave.

The Swappph, SwaPPath = m: \ 131072 524288, it seems that it works and is correct, but it is slow.
Generates this swapper.dat:
01/31/23 16:47 1,048,576 0 A --- Swapper.dat
Then it is not valid. That value for me means that an unpleasant value is introduced, and one is created by default.
I would wait created a next 500mb File

Now I will explain my frequency theory.
I think this is correct, but I don't see especially good results. Maybe the swapper is canceled partially.

It is a frequency, when you multiply a number * x frequency = n frequency
The OS/2 frequency constant is 31.25
The value has to be an entire number, that is, without decimals.
The swapper.dat if it is a file and has a size.
Examples below.

The minimum theoretical size would be 31.25*4 = 125, to have an entire number,
 the maximum depends on the space for the swapper.dat file

Therefore 125+125 = 250+125 = 375+125 = 500 .... will be valid values, which coincide with my test.

Some examples, the size of the swappath and the generated file:
swappath=L:\ 0 506000    ----31.25*16192-- = ---125*4048---
28/01/23 22:49    519.045.120      0 a---  SWAPPER.DAT
        3 file(s) 519.045.130 bytes used
                      9.307.136 bytes free

swappath=L:\ 0 3125      -----31.25*100---------
28/01/23 19:23      4.194.304      0 a---  SWAPPER.DAT

swappath=L:\ 0 6250       -----31.25*200---------
28/01/23 19:58      7.340.032      0 a---  SWAPPER.DAT

swappath=L:\ 0 15625    -----31.25*500-----------
28/01/23 20:12     16.777.216      0 a---  SWAPPER.DAT

swappath=L:\ 0 20625    -----31.25*660------------
28/01/23 20:46     22.020.096      0 a---  SWAPPER.DAT

swappath=L:\ 0 31125    ----31.25*996--------------
28/01/23 21:48     32.505.856      0 a---  SWAPPER.DAT

If you find a number that fulfills my 125xn theory that fails, my theory would be wrong.

I have been with 0 4096 for many months, with very satisfactory results, so I am with you that the 4K can be important.
Saludos

[Remy]

REMY
Thanks for the virtualaddresslimit, proposed as possible.
I have only tried 3008, for my screen problem. And it doesn't work, the screen is corrupted.
In principle, the rest, and stay with those of a lifetime, proposed by Dave.

The Swappph, SwaPPath = m: \ 131072 524288, it seems that it works and is correct, but it is slow.
Generates this swapper.dat:
01/31/23 16:47 1,048,576 0 A --- Swapper.dat
Then it is not valid. That value for me means that an unpleasant value is introduced, and one is created by default.
I would wait created a next 500mb File

Now I will explain my frequency theory.
I think this is correct, but I don't see especially good results. Maybe the swapper is canceled partially.

It is a frequency, when you multiply a number * x frequency = n frequency
The OS/2 frequency constant is 31.25
The value has to be an entire number, that is, without decimals.
The swapper.dat if it is a file and has a size.
Examples below.

The minimum theoretical size would be 31.25*4 = 125, to have an entire number,
 the maximum depends on the space for the swapper.dat file

Therefore 125+125 = 250+125 = 375+125 = 500 .... will be valid values, which coincide with my test.

Some examples, the size of the swappath and the generated file:
swappath=L:\ 0 506000    ----31.25*16192-- = ---125*4048---
28/01/23 22:49    519.045.120      0 a---  SWAPPER.DAT
        3 file(s) 519.045.130 bytes used
                      9.307.136 bytes free

swappath=L:\ 0 3125      -----31.25*100---------
28/01/23 19:23      4.194.304      0 a---  SWAPPER.DAT

swappath=L:\ 0 6250       -----31.25*200---------
28/01/23 19:58      7.340.032      0 a---  SWAPPER.DAT

swappath=L:\ 0 15625    -----31.25*500-----------
28/01/23 20:12     16.777.216      0 a---  SWAPPER.DAT

swappath=L:\ 0 20625    -----31.25*660------------
28/01/23 20:46     22.020.096      0 a---  SWAPPER.DAT

swappath=L:\ 0 31125    ----31.25*996--------------
28/01/23 21:48     32.505.856      0 a---  SWAPPER.DAT

If you find a number that fulfills my 125xn theory that fails, my theory would be wrong.

I have been with 0 4096 for many months, with very satisfactory results, so I am with you that the 4K can be important.
Saludos

About your screen issue, do you have a dedicated video card or is the full storage shared with the video (HW).
May be you sould check your bios and specifically the amount of MiB allocated for the video. OS/2 needs less storage (2D rendering) and 16M or if you have other OSes installed, 32MiB is good (I have it set to 32MiB and use win7 as well ArcaOs